This is my personal blog. The views expressed on these pages are mine alone and not those of my employer.

Tuesday, April 25, 2006

Now in a Browser Near You: Offline Access and Permanent, Client-Side Storage, Thanks to Dojo.Storage

[Note: This blog post is out of date. For up to date information on Dojo Offline, Dojo Storage, and Moxie please see the official web page.]




I'm proud to announce the immediate availability of dojo.storage and a new web-based editor named Moxie.

Imagine if web applications could store megabytes of data on the client-side, in the browser, both persistently and securely. No server needed.

Imagine if web applications could work offline with the click of a button. Want to access your web based word processor when you are not on the network, with your private files stored privately, right on your own machine and not on some server? Now you can.

Even better, imagine if all of this worked across the existing web; 95% of the existing browsers on the web could start using these features right now, with no software installs or funky new browsers.

What could you build if you had these tools? How about a truly collaborative, web-based word processor with client-side storage for your private documents, as well as offline access? Maybe an Ajax RSS aggregator with client-side caching of the feeds you read and offline access? An offline, web-based book reader using data from the Internet Archive's Open Library would be cool.

This is the vision, and now the reality, of dojo.storage.

For years I've been tinkering and prototyping to see if it was possible to achieve both persistent client-side storage as well as offline access across the existing installed browser base, cross-platform and cross-browser; there were many times I thought it was impossible, but I kept at it because it was an obsession I wanted to see exist. For the last eight months I have worked on it for at least two or three days a week, on my off days when I am not working for my consulting clients. Also, a few months ago, Julien Couvreur had a breakthrough when he figured out the HTTP header to send from a server to achieve offline access.

I've finally finished; dojo.storage is now done, in beta, and fully open source under a business-friendly BSD license.

I've put together an example application to show dojo.storage called Moxie, a web-based word processor with persistent, client-side storage, no server, and full offline access. It is open source and is in the Dojo repository. It works across the big three browsers: Internet Explorer 6+, Firefox, and Safari.

Take it for a spin. Use the rich editing control to enter a document, type in a file name, and select the Save button. If the document is above 100K, you will be prompted on whether to accept this storage request; user's can deny storage requests above 100K. Choose the file from the Load File pull down to reload the file, even if you have closed the browser. Follow the directions on the page on how to do offline access. I've done a bunch of QA around all of this, but the dojo.storage framework I created that powers this is still in beta; if you run into a bug or problem email me at bkn3@columbia.edu with the details.

Moxie was put together in one day using the power of the Dojo framework; it uses Dojo Widgets (the rich editing control); Dojo Events; and the newest member of the Dojo team, Dojo Storage.

What is dojo.storage? Dojo.storage is a unified API to provide JavaScript applications with storage. It is a generic front-end to be able to provide all JavaScript applications a consistent API for their storage needs, whether this JavaScript is in a browser, a Firefox plugin, an ActiveX control, using Windows Scripting Host, etc. Further, the storage backends can use whatever mechanism is appropriate; dojo.storage automagically detects its environment and available storage options and selects the most appropriate one.

You might have heard about AMASS, the Ajax MAssive Storage System, and are wondering about the relationship of AMASS and dojo.storage. AMASS was a proof-of-concept prototype of Flash based storage I released in October 2005. It was not widely QAed and only worked in Firefox and IE. Dojo.storage now supersedes and replaces AMASS.

The dojo.storage architecture is simple; a JavaScript application interacts with the Dojo Storage Manager, which selects the best available Storage Provider and makes it available. Storage Providers implement a generic interface, which makes the underlying storage system look like a simple hash table that can be saved and loaded from. Storage provider can optionally be persistent, and can provide metadata about their capabilities (isPersistent, hasSettingsUI, getMaximumSize, etc.)

Some possible storage providers:
Right now the dojo.storage system includes one provider, the Flash Storage Provider; others are not implemented yet but should be straightforward if folks want to contribute.

The Flash Storage Provider uses features available since Flash 6, including Flash's SharedObject's.

Why Flash? Flash now has a greater installed base than Internet Explorer; Flash 6+ has a 97.1% penetration across the installed base of PCs, while IE 5, 6, and 7 have 64.7%. Flash is probably one of the most installed pieces of software on the planet.

The Flash Storage Provider uses Flash as a hidden runtime to extend the browser, because of its ubiquity and cross-platform/cross-browser qualities. When browsers get native persistence support, your dojo.storage applications will continue to run since you write them against the generic dojo.storage APIs.

I'm going to quickly take you through Moxie, to show you how to use dojo.storage to build applications.

Moxie has two files: editor.html and editor.js. It has a singleton JS object named Moxie, which you can see if you look at the editor.js file.

It's HTML is straightforward, except for one thing: we use the Dojo Editor Widget, which we specify in HTML as follows:
<div id="storageValue" dojoType="Editor"
items="textGroup;;blockGroup;;
justifyGroup;;colorGroup;;
listGroup;;indentGroup;;
linkGroup;">
Click Here to Begin Editing
</div>
The items attribute tells the Editor widget what toolbars we want visible, and the between them says to put a seperator between them.

The next big step, inside edit.js, is to import our packages, including dojo.storage; this is at the top of the file:
dojo.require("dojo.dom");
dojo.require("dojo.event.*");
dojo.require("dojo.html");
dojo.require("dojo.fx.*");
dojo.require("dojo.widget.Editor");
dojo.require("dojo.storage.*");
Next, at the bottom of the file, we subscribe to find out when the storage system is ready for us to work with it; we can not load or save values until the underlying storage provider is ready. In this case we want to call Moxie.initialize when storage is ready; if by the off chance it is already ready when we get to this code block, then we want to wait until the page is fully loaded before working with it:
if(dojo.storage.manager.isInitialized() == false){
dojo.event.connect(dojo.storage.manager,
"loaded", Moxie,
Moxie.initialize);
}else{
dojo.event.connect(dojo, "loaded",
Moxie,
Moxie.initialize);
}
When we are all loaded up, we can start to play with the storage system. For example, to load a value that was previously saved with some key, we just do the following:
var results = dojo.storage.get(key);
To save some value:
try{
dojo.storage.put(key, value,
saveHandler);
}catch(exp){
alert(exp);
}
Value can be a string or even a complicated JS object; we internally JSON everything before storing it as a flat string, and turn it back into an object on later retrieval. Dojo.storage actually does some nifty autodetection if it is working with strings, to avoid the JSON performance hit of evaling() and bypass this, which gives much better performance for storing large strings into storage, like XML files or digital books as you will see in a later demo in this post.

Notice the saveHandler above; a storage system can optionally ask the user if they want to accept or deny your storage request, so you must be ready for a save request to fail.

saveHandler is a callback function that receives two arguments. The first is the status, which can be one of three values:
The second argument is the keyName that is being saved; since saving is asynchronous, it is sometimes useful to know which key is being worked with on the callback.

The Flash Storage Provider pops up an underlying Flash storage dialog to the user after 100K, and every order of magnitude increase after that:

Here's an example saveHandler:
var saveHandler = function(status, keyName){
if(status == dojo.storage.PENDING){
// ...
}else if(status == dojo.storage.FAILED){
// ...
}else if(status == dojo.storage.SUCCESS){
// ...
}
}

try{
dojo.storage.put(key, value,
saveHandler);
}catch(exp){
alert(exp);
}
There is more to the code, but those are the important bits in terms of understanding dojo.storage. It's pretty straightforward to use.

I'll briefly describe dojo.flash here, which is the layer that seperates out JS and Flash communication. Cross-browser, fast, reliable JS+Flash communication is really hard and ugly, so I encapsulated this portion out into it's own layer. The great thing is you don't have to know any of this externally; dojo.flash and dojo.storage work together to figure out your Flash capabilities, and use the appropriate mechanism internally. Zero hassle.

Dojo.flash provides several major services:
dojo.flash.Communicator was the real doozy to create; it was a pain in the butt, to put it lightly. This area of the system provides a method abstraction between Flash + JS. For example, JavaScript can call sayHello(), which is some Flash method, while Flash can execute DojoExternalInterface.call("dojo.storage.save", resultsHandler) to run some JS method.

The heart is something called DojoExternalInterface, which is a backport of Flash 8's ExternalInterface to Flash 6. I didn't want to create this backport, but the complexity of handling all the internal tricks I was doing to make this stuff work required that I wrap this magic in some kind of maintainable API. DojoExternalInterface makes it possible to register Flash methods that can be called from JavaScript:
DojoExternalInterface.initialize();
DojoExternalInterface.addCallback("put",
this, put);
DojoExternalInterface.addCallback("get",
this, get);
DojoExternalInterface.addCallback("remove",
this,
remove);
DojoExternalInterface.loaded();
There are three ways to do Flash+JS communication:
  1. Using LiveConnect/ActiveX + fscommands - Flash 6
    Pro: Extremely fast, can send very large data, mature
    Con: Only works on IE and Firefox
  2. Using ExternalInterface - Flash 8
    Pro: Easy to use, Works on Safari
    Con: Unbelievably slow, performance degrades O(n^2), serious serialization bugs
  3. getURL/LocalConnection/New Flash object for each call - Flash 7
    Pro: Very cross platform
    Cons: Destroys history, serious data size limitations and performance issues
Only methods 1 and 2 are acceptable for the Flash Storage Providers needs. It turns out that we use method 1 for IE and Firefox, and method 2, ExternalInterface, for Safari. I found workarounds to fix ExternalInterface's serious bugs, so that it works on Safari and is fast and serializes correctly, but these workarounds only work on Safari.

The Flash 8 communication support for Safari took 3 months to figure out; pain in the behind. I won't go into detail on the method and workarounds, but here is a bit of info on the workarounds:
With these workarounds, performance and reliability on Safari are great. Unfortunately, these workarounds only work on Safari, so we can't use ExternalInterface cross browser.

Check out this testing page, which provides a lower level interface to dojo.storage; there are quick links on the left to fully save an entire book into the storage system, in this case Faust by Goethe at 250k courtesy of Project Gutenberg; there is also a quick link to save an example RSS XML feed into the storage, in this case an atom feed from my weblog. Try this on Safari; in the past it took minutes and froze the machine to save the book, now it takes seconds.

I won't go into the Flash 6 communication necessary for IE and Firefox support; there were lots of fun different things that had to be figured out, like how to center the Flash dialog even if you are across many different kinds of HTML doctypes, browsers, and platforms, or isolating timing issues like the fact that the fscommand infrustructure on some versions of IE comes up after your Flash applet is already running.

dojo.flash was really hard to make, but I tried to isolate the pain into that module so that it is encapsulated and ready for folks to use. Externally its easy, internally it was hell.

Dojo.storage has had lots of what I call poor-mans QA testing, which means I went to coffee shops and Kinkos, paid them money, and used their machines to test all of this code; I also begged random people to go to the pages and saw how it works. I've spent a bunch of money and done alot of QA testing, but this is still beta; if you find a bug or have something wierd email me your browser and platform details at bkn3@columbia.edu.

For offline, Julien Couvreur discovered the necessary HTTP headers. The core part of it is to use HTTP caching; your site must have the following kinds of HTTP headers on:
Etags and Last-Modified are on by default on Apache 2.0; the rest have to be turned on in httpd.conf with mod_expires:
LoadModule expires_module modules/mod_expires.so

<Directory "c:/dev/dojo/">
ExpiresActive On
ExpiresDefault "access plus 1 month"
</Directory>

The page is now in the browser cache after the first access. In IE and Firefox, the user has to go to File > Work Offline to work offline, and then can simply just navigate to the app's URL. In Safari, this is not necessary, and you can just go to the URL. I like to provide a link that can be dragged to the toolbar.

The offline and dojo.storage work together, because whether you are offline or online you can access the same persistent storage, saving data while offline then syncing when online. Expect a dojo.offline and dojo.sync in the future that will provide abstractions for common operations like this. I'm looking for financial sponsors on this if you are interested.

Dojo.storage is in beta and is inside the Dojo Subversion now. It will be bundled with the next release of Dojo, including with Moxie, but you can download it now. Links:
If you try out dojo.storage and run into problems, remember that I did this all for free and make my money and food from consulting, so I'm not available to provide free support; please consider a consulting contract. I do want bug reports though.

Comments:
Hey Brad,
Nice work!

The link to the Moxie demo was broken for me ("take it for a spin") though.

I especially like the idea of DojoExternalInterface (I looked at Javascript/Flash communication in Flash 6 and it seems a pain; I'll definitely try it) and the choice of different storage capabilities for users who don't have Flash installed. Hooking a rich text editor is definitely a plus too.

Actually, I think the end-to-end integration was the biggest issue for TiwyWiki.
HTTP headers are a small part of the solution ("Expires" is the key header by the way), but the online/offline proxy and synchronization code were the biggest challenges. Interfacing with navigator.onLine also helped for making the experience smoother.

I have some questions:
1) How do you handle the synchronization process, for example saving offline modified content to the server?
2) Have you solved the problem of content being pushed out of the cache when running in an offline state for a while?
3) The Moxie demo seems to have the same problem as TiwyWiki: the back button is broken in IE. Did you have any ideas for that?
 
Julien, I fixed the 'take it for a spin' link; thanks for pointing that out.

The further work you've done on synchronization and automatically detecting online/offline access is very cool; I'm only using what you discovered about the HTTP headers so far. We should talk on the phone or over Skype some time about how we could fashion a dojo.offline or dojo.sync API. Are you ever in the Bay Area? Will you be at the Ajax Experience conference in two weeks? We should meet and get a beer.

Answers to your questions:
1) We don't handle this yet; right now I've delivered dojo.storage; the offline stuff is just a bonus and not really fleshed out to the degree you have done with TiwyWiki.
2) In terms of busting the cache, I don't have a solution for that. I haven't run into it yet.
3) I did not integrate RSH into the Moxie demo or explore history issues.
 
Hi Brad,

Very Nice!

There is a WYSIWYG editor created by Moxiecode Systems AB called TinyMCE, WYSIWYG users might be confused with your work. TinyMCE isn't implementing xmlHTTPrequest and the features your Moxie is sporting though.
 
This is really really cool (let me state that again, this is totally $@#%^&* cool!). I have a question though, would it ever be possible to save the file to the file system somehow (and be able to control the format with which its saved), or is this a pipe dream just waiting for some new iteration of browser based technology?
 
Very cool work, Brad... care to add a storage wrapper for Amazon's S3 service as discussed here: http://musingmarc.blogspot.com/2006/04/maintaining-client-side-state-but-not.html
 
Thanks for all your work on this Brad. I can think of a few things this will be handy for.

I had trouble getting your Moxie test to run until I remmeber that all of my firefox installs have flashblock enabled. This was silently killing your code. Could there be some kind of warning for detecting when flash was installed but not availible? Once I added the domain to the list of allowed flash domains this was not an issue, it just did not know how to handle detecting flash but not being able to talk to it.

Caleb
 
Hi Brad,
I looked at the Moxie; this some good stuff. Will it be possible to use apache derby as the data store
such as this demo:
http://blogs.sun.com/roller/page/FrancoisOrsini?entry=derby_apachecon_demo

Ransford
 
Hi Brad,

After talking to you about dojo.storage, I have find some use on my other project. It seems ideal for use to store some personalization data on the client side. But first there is an issue I have to resolve, is there any stragegy to backup the storage on the client side? With my current plan there won't not be a server. Everything is so simple that they would be deliver as a few static files that the user open on his computer.

Secondly, sadly this does not work in Opera. On the demo page everything is grayed out. What is the magic ignition code that launch this thing in a browser?

Wai Yip
 
Hi, would it be much trouble integrating this with jQuery library as a plugin ? I'd love using it but I don't like that it's strongly linked to a specific library ... at least another choice would be good. Thank you.
 
I have included Moxie in some web application named Rut. The worst problem that I have found is how to save documents from Moxie to mi hardrive. I have used Save Text Area plugin for Firefox but it is not available for IE. I have used a web page editable that I have named NavEdit but it doesn't work in IE.

In Firefox, anything I copy from Moxie to NavEdit is downloable as an editable html page. I can copy texts and images but I can not copy tables from Moxie to NavEdit because any table turn itself a single row but I can do it fron Open Office to NavEdit.

Moxi has not a print option on its bar. It is another awful problem.

Rut may be found embedded in a tiddler of my TiddlyWiki here http://tron.blogdns.org/esterilizacion/documentos/esterliz2.html when my server is running.
 
There is a WYSIWYG editor created by Moxiecode Systems AB called TinyMCE, WYSIWYG users might be confused with your work. TinyMCE isn't implementing xmlHTTPrequest and the features your Moxie is sporting though.
 
Post a Comment

Subscribe to Post Comments [Atom]



Links to this post:

Create a Link



<< Home

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]