It is mainly meant to serve as an example implementation, so that you could have a look at the code and see how to work with IndexedDB. But I figured that people are also interested in actually using it, as it abstracts away many of the tedious internals of IndexedDB (like transactions) – and it is perfectly fine to use IDBWrapper for all non-overly-complex scenarios.

So here’s a tutorial about how to work with IDBWrapper and add a little background info about IndexedDB internals every now and then, instead of writing Yet-Another-Super-Technical-IDB-Blargh. Part one will cover some info about what IndexedDB is, getting IDBWrapper to run and how to read and write data to a store. Part two will be about querying the store.

What is IndexedDB – why is it special?

IndexedDB really differs from other client-side storage engines. It’s an ObjectStore, whereas localStorage is a key-value store and Sqlite is a relational database. Think of it as a database like those SQL thingies, but with more freedom but without a query language. Uh, roughly, yeah. You store and retrieve JavaScript objects (unlike localStorage) and you don’t have fixed tables with predefined columns and types (unlike Sqlite).

Getting IDBWrapper

You can download/clone/fork IDBWrapper at GitHub. To include it in your page/project, you have two options: You can either include the IDBStore.js file via a script tag, or you can require it using an AMD loader like RequireJS.

Let’s go!

In this tutorial, we’ll create a people store where we store information about humans – a group of customers, for example.

Step one: open the store
A note on the properties we pass to the constructor:

• dbName, dbDescription, storeName: just choose meaningful names
• dbVersion: start with ’1.0′, and only change this when the structure of your db/store changes
• keyPath: the property you want as unique primary key
• autoIncrement: set this to true if you do not want to take care of the uniqueness of the primary key yourself
• onStoreReady: a function to be called when your store is ready to work with

IDBwrapper will open the database, check if a store with the given name exists and if not, it will create it using the above properties. You’ll then get an IDBStore instance back that you can use to work with the store. Once the store is created, it’s persistent. But, if you need to delete and re-create it (for example, if you happened to pick the wrong value for keyPath), you can delete the store using the deleteObjectStore method.

Inside IndexedDB: IDBWrapper works with one store; when you do a new IDBStore() it represents one store in an IndexedDB. If you want multiple stores inside of the same IndexedDB, you’d have to do a new IDBStore() for each store. This is an artificial limitation of IDBWrapper; if you work with IndexedDB directly, you can lock multiple stores when starting a transaction. This be useful when you want to update one store depending on the information found in another store, and you want to prevent this other store’s contents to change during your update action.

Step 2: storing a customer

Note how we don’t have an id property on the dude object, as we specified autoIncrement as true, so the database will take care of this. In the onsuccess callback we’ll get the id of the newly inserted record back. Also, see how we attached an array of emails to the record? In a relational database you’d set up a different table for email addresses – no need to do that in IndexedDB.

Inside IndexedDB: Make sure to wait until the store is ready before operating on it! The whole nature of IndexedDB is asynchronous. Whatever you do, you will have to work with callbacks. There is, however, also a draft for a specification of a synchronous API, but that is not implemented in any browser (and I don’t think this is going to happen anytime soon).

Step 3: reading the customer back from the store

We just pass the id we got after inserting to the get method, and there we have our record from the store. It’s exactly the same thing we stored earlier, with one exception: the object now has the id property which has been attached to it by the database.

Inside IndexedDB: What we pass as first parameter to the get method is the keyPath value. That means when we pass 1 to it, the database will look for an object that has 1 as value in it’s keyPath property, which we set to ‘id’ when we created the store.

Step 4: updating data

To update a record, you also use the put method: If there already is an object in the store with the same keyPath value (the id), it will simply overwrite it. Note that it will really, really overwrite the existing object, which means you cannot simply put a modified property of it, but have to put the whole object, including all properties.

Inside IndexedDB: keyPath values are also Type-sensitive. That means, if the id of our dude is 1 and of type Number, you have to pass numeric 1 to the get method to retrieve the dude, and for updates the value of the id property needs to be numeric 1. If you use “1″ as String, IndexedDB will think of it as a different value.

Step 5: getting all items

Nothing really to say about it. Just one note: IDBStore has default error handlers for every async method that prints potential errors to the console. So if you’re just playing around you don’t need to pass your own handler to the methods.

Step 6: deleting an item

Different browsers and versions return different values to the success callback; that’s why there is an extra check in the success callback.

Step 7: clearing the store

If you need to clear the store from all stored entries, you can use the clear method. Note that this won’t reset Chrome’s autoIncrement counter.

That’s it, thanks for listening!

Now you know everything to do basic IndexedDB data operations with IDBWrapper. Make sure to check out the examples and the documentation!

Part two of this tutorial about querying the store will follow soon.

Resources

• IDBWrapper on GitHub
• IDBWrapper Documentation
• IDBWrapper Examples

As of version 1.6, dojo comes with the new Dojo Object Store API. This is an awesome thing, as it greatly simplifies the work with data stores in Dojo. Everybody who had to do with the traditional dojo.data API felt it was overly complex and hard to use – this has finally changed now. There are also wrappers from and to the old and new APIs, so that you can do stuff like using your traditional data-aware widgets with a new Object Store. And the goodness doesn’t end here; but more on this later. If you haven’t done so yet, you might want to read the excellent post on the new Dojo Object Stores by Kris Zyp where he explains all the awesomeness he created.

Dojo also comes with two fresh implementations for the API, a non-persistent memory store and a JsonRest store that interacts with a server through RESTful HTTP requests. You can also observe changes made to objects in the store. What is missing is a store that uses a client-side persistent backend, which would be useful for a couple of reasons (e.g., as one commenter in that post asks, to use it as chache store for dojo.store.Cache). As Kris already mentions, it’s a piece of cake to create one, so let’s go ahead and do it!

Choosing the Backend

The post mentions two possible candidates: dojox.storage and StorageJS, both already having a similar API, so making them compliant to the Dojo Object Store API is fairly easy. For this tutorial I choose StorageJS – because it is very lightweight and still provides all we need (and, well, for a couple of other reasons that don’t belong in this post). So, what is the exact API we need to comply to? Check it out in the docs. See the first sentence in that paragraph? “Every method in the API is optional, it’s presence indicating support for that feature.” Wow, that’s nice!

Basic compliance

Let’s start with the methods get(), put(), add() and remove(). That will make our store pretty usable for most store scenarios.

First, create a ‘store’ object we can work with (StorageJS creates a global variable called storage where all it’s methods reside):

‘store’ now contains everything that ‘storage’ does, but we can leave the original ‘storage’ object alone and modify our ‘store’ object if needed. Time for the first method, let’s start with get(). StorageJS works with Strings as keys and values only, so we need to parse what we get from it.

Easy one, now move on to put(). The Object Store API allows user to do stuff like this: store.put({foo: 'bar'}, {id: 3}) as well as store.put({id: 3, foo: 'bar'}). Also, StorageJS only has the method set() instead of put() and add().

Easy, as well – you could write this as a one-liner. Now, add(). It’s close to put, except that it won’t overwrite data that already exists. So we just need to check if something with the given id already exists and throw an error if so.

What about remove()? Nothing to do there, it already has the desired signature and functionality.

That’s it, we’re done with basic compliance. Our ‘store’ object now implements the basic methods of the Dojo Object Store API and can be used by anything that can work with a Dojo Object Store – and it will keep it’s state persistent in the browser.

More compliance? Querying!

There’s one interesting thing in the API: the query() method. Uh, yeah, we want that! Lucky us, Dojo already provides a query engine as well as a method that makes sure there are iterative methods and a total property containing the number of hits available in our result (well, it also abstracts away the differences between sync and async results, but as StorageJS is purely synchronous, this is not important to us). First, define a queryEngine:

So, if we want our store to be able to run queries, all we need to do is to take the store’s data and hand it over to the query engine. We use StorageJS’ getAll() method to acquire the data. It will return an array of objects in the form { key: ‘theKey’, value: ‘theValue’}, but the queryEngine wants an array containing only the objects, so we need to do a conversion.

Now we can query our store for specific data, like so: var results = store.query({ prime: true }); This will give us all objects in the store that have a property called ‘prime’ and have that property set to true.

Hierarchie and Transactions

The API also mentions methods about data hierarchie and transactions, but I’m not going to cover these in this tutorial.

Convenience and Optimization

Currently, every time we run a query, we fetch all data again from our storage engine. So, if you don’t have a massive amount of data and want to query the store a lot, it might be wise to maintain a copy of all data in memory. There are other things you could optimize as well – or you could write your own queryEngine.

For your convenience,  here’s a wrapper that creates a Dojo Object Store like we did above and accepts a useMemory parameter during instantiation that denotes whether a copy of the data should be kept in memory or not. It uses the traditional require/provide instead of the AMD format. If you want to use it, make sure you use the right namespace in the declare() call! (Or, not recommended, copy it in the /dojo/store directory.)

If you want to fool around with it in the console first, you can go here (non-memory-using) or here (memory-using), check out the source and fire up the console and try things out.

Getting StorageJS

The above code uses the features base and getAll, so you could do a specific build of StorageJS with these features or just grab a full build of it for your desired storage engine (the storage-full-[engine].js files). If you don’t know about storage engines, all modern browser support localStorage, so this is the one what you might want.

IndexedDB is all the buzz right now, but it is pretty hard to find information about it’s implementations. In addition to that, the impls change on a weekly basis. So I figured it would be nice to summarize every now and then what has been happening around IDB in the last time.

And as Mozilla released a new Firefox 4 beta these days, and introduces changes to it’s IDB impl, let’s start the series with a brief summary of the most important of these changes.

Changes in 4.09b

• The indexedDB object has been renamed: moz_indexedDB is now mozIndexedDB.
• When opening a database, the event handed over to the onsuccess callback no more has a property called “result”, where a reference to the databasse used to be; it is now available at event.target.result (event.target contains the IDBRequest).
• This applies to many (all?) cases, where you’d expect the result of a request to live in event.result, where it now is in event.target.result.
• When creating an object store, the second argument, keyPath, now has to be passed as a property inside of an object: {keyPath: "keyPathName"}
• When creating an objectStore, the third argument, autoIncrement, is dropped and now has to be part of the second argument: {autoIncrement: true}
• Methods that read from an objectStore now only seem to work inside of a transaction; before, it was allowed to call methods like get directly.
• keyRange support seems to have increased. However, I’m not really interested (right now) in keyRanges and queries, so I did not play around with these.

Note that the changes for the createObjectStore arguments are against the spec – the 4.08b was compliant, the 4.09b is no more. But, on the other hand, it is now closer to the Chrome impl, which also expects an object as second argument (but has no support for autoIncrement).

What it does *not* have (yet)

The sync API. Alas. To me, the snychronous implementation is exactly what makes indexedDB that attractive – I’m not a fan of databases and I don’t like to start transactions, iterate over resultSets and the like… I want a store with easy access. Seriously, in most real life scenarios, an async approach won’t get you anywhere but displaying a waiting message to your user, because you can’t go on until you have the data from the store.

More info?

If you want to stay in touch with what happens around the indexedDB impl, the best place to go ist the Mozilla bugtracker. Um, yeah, reading browser source code can be no fun at times, but it gets you as close as you can get. Or, check what happens in the test files.

Need working code?

The *only* place to get reliable, working code is the test files in the repository. Seriously, there’s so much code out there claiming to work, but it simply doesn’t. And even if it did one day, it might break the other day, after changes have been made to the impl. And there will be *lots* of changes in the future.

Where are these test files?

In the repo. To see them online, go here: http://hg.mozilla.org/mozilla-central/file/ and then navigate to dom/indexedDB/test.

Something else?

Nope, not really. There don’t seem to be any *major* changes for 4.09b – I hope you still found this short summary useful.

I hope to continue this series about IDB and post about different IDB topics every now and then  – so if you want to know more about indexedDB in general, or something specific, just drop a line in the comments!

Further reading

• The spec: http://www.w3.org/TR/IndexedDB/
• The MDC article: https://developer.mozilla.org/en/IndexedDB

Please grab all phones you have and navigate to http://jensarps.de/tests/storage-tests/.

There are four tiny tests there. Please do them all and report your results for each phone. If you like, include your twitter handle so I can say thank you!

Your help is greatly appreciated!! Thanks a ton!!

Among other tests, the main thing I wanted to know is how long does it take to read or write 1000 entries from/to localStorage. The keys were of the form “key000″, the values “val000″, with increasing numbers. So the tests were using minimal data – only 6  bytes – but the results are still useful to get an idea of how it goes – and how the browsers perform compared to each other.

Desktop

All test were run on a Mac Mini. Opera and IE were tested on a virtual machine, so there might be an additional performance penalty based on virtualization. However, here are the numbers (average time needed to run the respective method 1000 times):

Safari 4:
- getItem: 1.4ms
- setItem: 1.7ms
- key: 0.8ms

Chrome:
- getItem: 240ms
- setIem: 280ms
- key: 230ms

Firefox:
- getItem: 77ms
- setItem: 2100ms
- key: 25ms

Opera:
- getItem: 19ms
- setItem: 20ms
- key: 18ms

IE8:
- getItem: 5ms
- setItem: 72ms
- key: 3.6ms

Mobile

As we can see, Safari is lightening fast. That made me want to run the tests on a mobile phone that has a modern Safari browser. I took an HTC Desire, wich comes with Android 2.1, and thus has a Safari capable of localStorage. Here are the results:

HTC Desire:
- getItem: 16.4ms
- setItem: 19.5ms
- key: 8.7ms

This is pretty amazing, considering that it is a phone. It still beats most of the browseres on the desktop.

If I find that test page again (uh, yeah, it’s lost somehow), I’ll run the tests again with newer browser releases and publish the link here, so you can run the tests yourself.

So here you go: dojox.storage.encrypted, a Blowfish encrypted storage. It sits on top of dojox.storage, and you get all the dojo storage manager goodness, mainly the automatic selection of the best storage provider available. It exposes the complete API that dojox.storage does. If an attacker gains access to the storage area, he can still nuke the storage, but the data found within will be useless.

I’m not completely happy with it’s architecture, and I didn’t have the time yet to fully test it with all providers, but it works pretty well with localStorage and the performance is not too bad. If you have a spare minute and are interested, feel free to to test it yourself (ah I know, you don’t have the time yourself, but I’d be thankful!)…

Usage

Just use it as you would use dojox.storage, except that you need to set your passphrase before you start working with the storage:

dojo.require("dojox.storage.encrypted");
var sto = dojox.storage.encrypted;
sto.setPassphrase("my super secret passphrase");
sto.put('key','value');
var value = sto.get('key');

It uses/requires dojox.encoding.crypto.Blowfish but that’s been in dojo for ages, so no worries.

To use it, just put the encrypted.js file in your dojox/storage directory.

Files

the code (encrypted.js)
the test page

The common way to persist data on the client side – application state, offline data, whatever –  still is to use cookies. But times have changed, and so have browsers, and there are better ways to do it today.

But why are cookies that bad? Well, here are the top three reasons:

1. Of all client side storage mechanisms, cookies have the worst limitation in size (4k if you want to stay IE-safe)
2. Cookies are sent to the server on every requests that matches the cookie domain – inlcuding XHR calls (aka. How to slow down your AJAX app)
3. Cookies perform bad, can be easily disabled, and, oh well, they are sooo 1995…

What else to use? There are several options, let’s start with the best:

localStorage

This is the storage mechanism proposed by the HTML5 draft. It offers you a fast, reliable key-value store with a whopping 5M (!!) of storage space. And here’s the kicker: It’s implemented in all current major browser releases. That is, FF, Safari, Opera, IE and Chrome/Chromium. Yes, you name it, all of them, the full list. To read more about localStorage, check the HTML5 draft. But here’s all you need to know in short:

Storage Methods

To test if the user agent supports localStorage, just test for the existence of window.localStorage. Working with the storage object is trivial:

Setting a value:

localStorage.setItem('key','value');

Though the draft says the user agent should write a “structured clone” of the value to store, don’t think that you can store objects – you can only store strings. To store objects, you need to convert them to JSON before.

To retrieve a value, there are two options. First, you can get a value by it’s key:

var value = localStorage.getItem('key');

The user agent will return the value for the key, or, if it doesn’t find a matching record, null. Also, you can retrieve a key by it’s index:

var value = localStorage.key(n);

The storage object has a length property, and n is a number between zero and length-1. You do not get the index of a value when inserting it, but you can use the key() method when iterating over all items in the store using the length attribute.

There is another method to retrieve a value, but as this not documented (at least I didn’t find anything about it), I do not recommend it:

var value = localStorage.key;

Deleting an item:

localStorage.removeItem('key');

Deleting the whole storage:

localStorage.clear();

This clears only the storage objects of the same storage area. That is, storage objects of the current domain.

So far, everything is pretty trivial, but let’s look at

Storage Exceptions

When accessing the storage object, a user agent may raise an SECURITY_ERR exception – but I have never managed to trigger one. When setting an item, a user agent may raise an QUOTA_EXCEEDED_ERR exception. This is not only for the case when storage space is at it’s limit, but may be raised for other reasons as well. Again, I have never managed to trigger it, but it may be wise to wrap a setItem call in a try/catch block.

Storage Event

Now things get tricky, and the different browsers have different (if any) implementations. Ok, first, the theory: methods that modify the storage object (setItem(),removeItem(),clear()) do not report if – or when – the changes are in effect. To get notified about this, you need to connect to the storage event:

window.addEventListener("storage", function(evt){
    // a change to the storage object has been attempted.
}, false);

The event should have the following attributes:

• key
• oldValue
• newValue
• url (the address of document object whose storage object was modified)
• storageArea (the affected storage object)

With this information, you can easily determine to what change the storage event belongs. But, alas, that event is not fully implemented everywhere. Here’s what I found out:

FF 3.6.3 lets you connect to the event, and fires it according to the draft. But it does not contain *any* of the fields mentioned above.

Safari 4.0.5 lets you connect to the event, and fires it according to the draft. It’s event contains all attributes except url.

on Chrome 5 & Chromium I couldn’t connect to the event at all or it wouldn’t let my execute code in my callback function – I have no idea if it’s even fired.

Latest WebKit nightly wouldn’t report anything to me, like Chrome.

Opera 10 lets you connect to the event, fires it accordingly, and the event contains all attributes (woot!)

IE 8 didn’t report anything back to me, too.

I used this simple test file to check for the event. Feel free to test other browsers and report what you found!

Other options

If the browser is too old to have localStorage, there are still other options to persist data: Firefox 2 had globalStorage, which behaves similar to localStorage. IE has userData beahvior since version 5.5. Safari has it’s built-in Sqlite database, but I don’t know since when. Plus, there’s also plugin-based storage via Flash or Gears. Only if they all fail, you should get back to cookies.

Of course, it would be a lot of work to write wrappers for all these storage mechanisms – but, hey, you don’t have to, others have already done it. There are several toolkits that do the work for you, and I will present three of them.

Dojo

The dojo toolkit has dojox.storage. It currently offers support for Flash, Gears and globalStorage. But with the files you can grab at my previous post on storage, you gain access to localStorage, userData behavior and cookies. Working with dojox.storage is pretty simple, and you don’t have to worry about anything: dojo’s storage manager will pick the best available provider. Here’s a simple example:

dojo.require("dojox.storage");
dojox.storage.put('key','value');
var value = dojox.storage.get('key');

That’s it. Simple as that. You do have more options, though; you can also specify a namespace for your keys – so you can simulate working with different “tables” in one store. You can also specify a callback function for the put() method, to get notified when the changes are in effect and if the change was successful.

Lawnchair

Lawnchair has been created by Brian Leroux from Nitobi. Working with it is dead simple, too. It supports localStorage, userData behavior, Gears, Webkit’s Sqlite and Cookies. It works more like a document store (you can save an object w/o specifying a key, for example), but you can also use it like a straight key/value store. In opposition to dojo’s storage system, Lawnchair works completely asnc, so you have to pass a callback function to a get() method to retrieve a value (being async is not a drawback, and it is neccessary to work with WebKit’s Sqlite). You can find some docs here and get it over at Github.

Persist.js

Persist.js has support for localStorage, globalStorage, Gears, Flash, userData behavior and Cookies. It also works completely async. You can get it at Github.

Uh, persistant local storage?

Ahm, yeah, in case you don’t know, that means a client-side storage, be it in a browser, or in an Air app. It’s not very popular, but that concept has been around for a long time. So why local? There are two major reasons for it: First, we need this for apps and tools that work offline – and apps and tools that work online but need an offline backup and sync later. Kris Zyp wrote a post about the JsonRestStore and OfflineRest back in 2008, he goes a little into detail there. Secondly, we need it for apps and tools that rely on persistence for other reasons, no matter if online or offline – like it happended to me. When I ran into Lawnchair, I had the idea to build a tool that was monitoring CSS selector usage on websites and apps (you know how you sometimes lose control about CSS selectors in webapps…). To achieve this, I needed to store data locally, and persistent.

What dojo offers

As I was using dojo’s excellent CSSRuleStore to find out what selectors where used, I wanted to switch from Lawnchair to a dojo solution. Dojo has dojox.storage (the above mentioned OfflineRest uses dojox.storage), which provides wrappers for the following backends: Gears, Flash, globalStorage and Adobe Air (globalStorage is Firefox’s early implementation of the early WhatWG storage draft). Hm. Not that much. Gears and Flash are 3rd party solutions, globalStorage is a Firefox-only-thingy (and old, though it’s still supported in current Versions) and Air is non-browser. That means: No IE without plugins. No Safari when Flash is not available.

What dojo does not offer (but should)

Guaranteed, reliable support for *any* browser. Without depending on 3rd party plugins. As of now, to have this, you’d need to use Lawnchair or persist,js, which is pretty complete. Speaking of it, let me quote from it’s readme (it’s a good read on local persistent storage in general):

The most notable attempt at addressing this problem [storing client-side persistent data] is probably Dojo Storage. Unfortunately, Dojo Storage does not support Internet Explorer without Flash, and it does not support Safari or other WebKit-based
browsers at all (at least, not without Flash). Also, Dojo Storage is
not standalone; it requires a several other Dojo components in order to
operate.

Whoa, even other JS tools mention the lack of support in dojo…

New providers

Attached to this post are wrappers for the following mechanisms:

• localStorage: this is the current spec, and it is supported by IE8, Firefox 3, Safari 4, Opera 10, Chrome 5.
• userData behavior: this is Microsofts client-side storage for IE < 8.
• cookie: worst case fallback solution…

With these three, there’s an almost 100% coverage of browser situations (see storage overview). There are still no-go scenarios, of course, such as IE7 with userData disabled and cookies disabled, but there’s no way to address them all.

Notes on BehaviorStorageProvider

This one uses the userData behavior, present in IE from version 5 up. It works with DOM elements that have a special behavior assigned. You can then use  set/get/removeAttribute() methods to store data. It has some limitations though – of course…

• Maximum size varies from 64k to 1M – depending on the zone the site is in. So, you can’t rely on more than 64k.
• UserData can be switched off in security settings.
• I couldn’t find a way to read out stored keys if you don’t know them. Maybe this is some security thingy or just a bug – but it means that some methods in the provider don’t work. (that is, getNamespaces, getKeys and clear). To work around this, we have to take some of the precious storage space and store the keys and namespaces seperately.

Usage

Copy the the providers into your /dojox/storage directory and add the needed dojo.requires to /dojox/storage/_common.js:

dojo.require("dojox.storage.LocalStorageProvider");
dojo.require("dojox.storage.GearsStorageProvider");
dojo.require("dojox.storage.BehaviorStorageProvider");
//>>excludeStart("offlineProfileExclude", kwArgs.dojoxStorageBuildOption == "offline");
dojo.require("dojox.storage.WhatWGStorageProvider");
dojo.require("dojox.storage.FlashStorageProvider");
//>>excludeEnd("offlineProfileExclude");
dojo.require("dojox.storage.CookieStorageProvider");

What’s left?

Well, it would be pretty cool to have a dojo.data API compliant store that uses dojox.storage as backend, so that you could write your apps using dojo.data calls and have a reliable offline backup. I started working on that, but then came Christmas, parents, vacation, new year and a lot of beer. But hey, it’s 2010 now, and there’s plenty of time left!

Storage Overview

Files

LocalStorageProvider.js
BehaviorStorageProvider.js
CookieStorageProvider.js
Storage Test Page