Please grab all phones you have and navigate to http://jensarps.de/tests/storage-tests/.

There are four tiny tests there. Please do them all and report your results for each phone. If you like, include your twitter handle so I can say thank you!

Your help is greatly appreciated!! Thanks a ton!!

Among other tests, the main thing I wanted to know is how long does it take to read or write 1000 entries from/to localStorage. The keys were of the form “key000″, the values “val000″, with increasing numbers. So the tests were using minimal data – only 6  bytes – but the results are still useful to get an idea of how it goes – and how the browsers perform compared to each other.

Desktop

All test were run on a Mac Mini. Opera and IE were tested on a virtual machine, so there might be an additional performance penalty based on virtualization. However, here are the numbers (average time needed to run the respective method 1000 times):

Safari 4:
- getItem: 1.4ms
- setItem: 1.7ms
- key: 0.8ms

Chrome:
- getItem: 240ms
- setIem: 280ms
- key: 230ms

Firefox:
- getItem: 77ms
- setItem: 2100ms
- key: 25ms

Opera:
- getItem: 19ms
- setItem: 20ms
- key: 18ms

IE8:
- getItem: 5ms
- setItem: 72ms
- key: 3.6ms

Mobile

As we can see, Safari is lightening fast. That made me want to run the tests on a mobile phone that has a modern Safari browser. I took an HTC Desire, wich comes with Android 2.1, and thus has a Safari capable of localStorage. Here are the results:

HTC Desire:
- getItem: 16.4ms
- setItem: 19.5ms
- key: 8.7ms

This is pretty amazing, considering that it is a phone. It still beats most of the browseres on the desktop.

If I find that test page again (uh, yeah, it’s lost somehow), I’ll run the tests again with newer browser releases and publish the link here, so you can run the tests yourself.

So here you go: dojox.storage.encrypted, a Blowfish encrypted storage. It sits on top of dojox.storage, and you get all the dojo storage manager goodness, mainly the automatic selection of the best storage provider available. It exposes the complete API that dojox.storage does. If an attacker gains access to the storage area, he can still nuke the storage, but the data found within will be useless.

I’m not completely happy with it’s architecture, and I didn’t have the time yet to fully test it with all providers, but it works pretty well with localStorage and the performance is not too bad. If you have a spare minute and are interested, feel free to to test it yourself (ah I know, you don’t have the time yourself, but I’d be thankful!)…

Usage

Just use it as you would use dojox.storage, except that you need to set your passphrase before you start working with the storage:

dojo.require("dojox.storage.encrypted");
var sto = dojox.storage.encrypted;
sto.setPassphrase("my super secret passphrase");
sto.put('key','value');
var value = sto.get('key');

It uses/requires dojox.encoding.crypto.Blowfish but that’s been in dojo for ages, so no worries.

To use it, just put the encrypted.js file in your dojox/storage directory.

Files

the code (encrypted.js)
the test page

The common way to persist data on the client side – application state, offline data, whatever –  still is to use cookies. But times have changed, and so have browsers, and there are better ways to do it today.

But why are cookies that bad? Well, here are the top three reasons:

1. Of all client side storage mechanisms, cookies have the worst limitation in size (4k if you want to stay IE-safe)
2. Cookies are sent to the server on every requests that matches the cookie domain – inlcuding XHR calls (aka. How to slow down your AJAX app)
3. Cookies perform bad, can be easily disabled, and, oh well, they are sooo 1995…

What else to use? There are several options, let’s start with the best:

localStorage

This is the storage mechanism proposed by the HTML5 draft. It offers you a fast, reliable key-value store with a whopping 5M (!!) of storage space. And here’s the kicker: It’s implemented in all current major browser releases. That is, FF, Safari, Opera, IE and Chrome/Chromium. Yes, you name it, all of them, the full list. To read more about localStorage, check the HTML5 draft. But here’s all you need to know in short:

Storage Methods

To test if the user agent supports localStorage, just test for the existence of window.localStorage. Working with the storage object is trivial:

Setting a value:

localStorage.setItem('key','value');

Though the draft says the user agent should write a “structured clone” of the value to store, don’t think that you can store objects – you can only store strings. To store objects, you need to convert them to JSON before.

To retrieve a value, there are two options. First, you can get a value by it’s key:

var value = localStorage.getItem('key');

The user agent will return the value for the key, or, if it doesn’t find a matching record, null. Also, you can retrieve a key by it’s index:

var value = localStorage.key(n);

The storage object has a length property, and n is a number between zero and length-1. You do not get the index of a value when inserting it, but you can use the key() method when iterating over all items in the store using the length attribute.

There is another method to retrieve a value, but as this not documented (at least I didn’t find anything about it), I do not recommend it:

var value = localStorage.key;

Deleting an item:

localStorage.removeItem('key');

Deleting the whole storage:

localStorage.clear();

This clears only the storage objects of the same storage area. That is, storage objects of the current domain.

So far, everything is pretty trivial, but let’s look at

Storage Exceptions

When accessing the storage object, a user agent may raise an SECURITY_ERR exception – but I have never managed to trigger one. When setting an item, a user agent may raise an QUOTA_EXCEEDED_ERR exception. This is not only for the case when storage space is at it’s limit, but may be raised for other reasons as well. Again, I have never managed to trigger it, but it may be wise to wrap a setItem call in a try/catch block.

Storage Event

Now things get tricky, and the different browsers have different (if any) implementations. Ok, first, the theory: methods that modify the storage object (setItem(),removeItem(),clear()) do not report if – or when – the changes are in effect. To get notified about this, you need to connect to the storage event:

window.addEventListener("storage", function(evt){
    // a change to the storage object has been attempted.
}, false);

The event should have the following attributes:

• key
• oldValue
• newValue
• url (the address of document object whose storage object was modified)
• storageArea (the affected storage object)

With this information, you can easily determine to what change the storage event belongs. But, alas, that event is not fully implemented everywhere. Here’s what I found out:

FF 3.6.3 lets you connect to the event, and fires it according to the draft. But it does not contain *any* of the fields mentioned above.

Safari 4.0.5 lets you connect to the event, and fires it according to the draft. It’s event contains all attributes except url.

on Chrome 5 & Chromium I couldn’t connect to the event at all or it wouldn’t let my execute code in my callback function – I have no idea if it’s even fired.

Latest WebKit nightly wouldn’t report anything to me, like Chrome.

Opera 10 lets you connect to the event, fires it accordingly, and the event contains all attributes (woot!)

IE 8 didn’t report anything back to me, too.

I used this simple test file to check for the event. Feel free to test other browsers and report what you found!

Other options

If the browser is too old to have localStorage, there are still other options to persist data: Firefox 2 had globalStorage, which behaves similar to localStorage. IE has userData beahvior since version 5.5. Safari has it’s built-in Sqlite database, but I don’t know since when. Plus, there’s also plugin-based storage via Flash or Gears. Only if they all fail, you should get back to cookies.

Of course, it would be a lot of work to write wrappers for all these storage mechanisms – but, hey, you don’t have to, others have already done it. There are several toolkits that do the work for you, and I will present three of them.

Dojo

The dojo toolkit has dojox.storage. It currently offers support for Flash, Gears and globalStorage. But with the files you can grab at my previous post on storage, you gain access to localStorage, userData behavior and cookies. Working with dojox.storage is pretty simple, and you don’t have to worry about anything: dojo’s storage manager will pick the best available provider. Here’s a simple example:

dojo.require("dojox.storage");
dojox.storage.put('key','value');
var value = dojox.storage.get('key');

That’s it. Simple as that. You do have more options, though; you can also specify a namespace for your keys – so you can simulate working with different “tables” in one store. You can also specify a callback function for the put() method, to get notified when the changes are in effect and if the change was successful.

Lawnchair

Lawnchair has been created by Brian Leroux from Nitobi. Working with it is dead simple, too. It supports localStorage, userData behavior, Gears, Webkit’s Sqlite and Cookies. It works more like a document store (you can save an object w/o specifying a key, for example), but you can also use it like a straight key/value store. In opposition to dojo’s storage system, Lawnchair works completely asnc, so you have to pass a callback function to a get() method to retrieve a value (being async is not a drawback, and it is neccessary to work with WebKit’s Sqlite). You can find some docs here and get it over at Github.

Persist.js

Persist.js has support for localStorage, globalStorage, Gears, Flash, userData behavior and Cookies. It also works completely async. You can get it at Github.

Uh, persistant local storage?

Ahm, yeah, in case you don’t know, that means a client-side storage, be it in a browser, or in an Air app. It’s not very popular, but that concept has been around for a long time. So why local? There are two major reasons for it: First, we need this for apps and tools that work offline – and apps and tools that work online but need an offline backup and sync later. Kris Zyp wrote a post about the JsonRestStore and OfflineRest back in 2008, he goes a little into detail there. Secondly, we need it for apps and tools that rely on persistence for other reasons, no matter if online or offline – like it happended to me. When I ran into Lawnchair, I had the idea to build a tool that was monitoring CSS selector usage on websites and apps (you know how you sometimes lose control about CSS selectors in webapps…). To achieve this, I needed to store data locally, and persistent.

What dojo offers

As I was using dojo’s excellent CSSRuleStore to find out what selectors where used, I wanted to switch from Lawnchair to a dojo solution. Dojo has dojox.storage (the above mentioned OfflineRest uses dojox.storage), which provides wrappers for the following backends: Gears, Flash, globalStorage and Adobe Air (globalStorage is Firefox’s early implementation of the early WhatWG storage draft). Hm. Not that much. Gears and Flash are 3rd party solutions, globalStorage is a Firefox-only-thingy (and old, though it’s still supported in current Versions) and Air is non-browser. That means: No IE without plugins. No Safari when Flash is not available.

What dojo does not offer (but should)

Guaranteed, reliable support for *any* browser. Without depending on 3rd party plugins. As of now, to have this, you’d need to use Lawnchair or persist,js, which is pretty complete. Speaking of it, let me quote from it’s readme (it’s a good read on local persistent storage in general):

The most notable attempt at addressing this problem [storing client-side persistent data] is probably Dojo Storage. Unfortunately, Dojo Storage does not support Internet Explorer without Flash, and it does not support Safari or other WebKit-based
browsers at all (at least, not without Flash). Also, Dojo Storage is
not standalone; it requires a several other Dojo components in order to
operate.

Whoa, even other JS tools mention the lack of support in dojo…

New providers

Attached to this post are wrappers for the following mechanisms:

• localStorage: this is the current spec, and it is supported by IE8, Firefox 3, Safari 4, Opera 10, Chrome 5.
• userData behavior: this is Microsofts client-side storage for IE < 8.
• cookie: worst case fallback solution…

With these three, there’s an almost 100% coverage of browser situations (see storage overview). There are still no-go scenarios, of course, such as IE7 with userData disabled and cookies disabled, but there’s no way to address them all.

Notes on BehaviorStorageProvider

This one uses the userData behavior, present in IE from version 5 up. It works with DOM elements that have a special behavior assigned. You can then use  set/get/removeAttribute() methods to store data. It has some limitations though – of course…

• Maximum size varies from 64k to 1M – depending on the zone the site is in. So, you can’t rely on more than 64k.
• UserData can be switched off in security settings.
• I couldn’t find a way to read out stored keys if you don’t know them. Maybe this is some security thingy or just a bug – but it means that some methods in the provider don’t work. (that is, getNamespaces, getKeys and clear). To work around this, we have to take some of the precious storage space and store the keys and namespaces seperately.

Usage

Copy the the providers into your /dojox/storage directory and add the needed dojo.requires to /dojox/storage/_common.js:

dojo.require("dojox.storage.LocalStorageProvider");
dojo.require("dojox.storage.GearsStorageProvider");
dojo.require("dojox.storage.BehaviorStorageProvider");
//>>excludeStart("offlineProfileExclude", kwArgs.dojoxStorageBuildOption == "offline");
dojo.require("dojox.storage.WhatWGStorageProvider");
dojo.require("dojox.storage.FlashStorageProvider");
//>>excludeEnd("offlineProfileExclude");
dojo.require("dojox.storage.CookieStorageProvider");

What’s left?

Well, it would be pretty cool to have a dojo.data API compliant store that uses dojox.storage as backend, so that you could write your apps using dojo.data calls and have a reliable offline backup. I started working on that, but then came Christmas, parents, vacation, new year and a lot of beer. But hey, it’s 2010 now, and there’s plenty of time left!

Storage Overview

Files

LocalStorageProvider.js
BehaviorStorageProvider.js
CookieStorageProvider.js
Storage Test Page