Well, however, I don’t want to get carried away, I guess you get what I mean. So, I decided to start something else. Something that would be easier to realize for a non-3D-guy like me. And something using three.js, as I’ve always been using GLGE for my WebGL experiments (RavenJS was built using GLGE, too). And, most important, something that could be open source and live on GitHub. Something that everybody could play, download and fork. Something people could contribute to, modify, extend, make better.

And so I did. It’s all in the very beginnings, but it’s really fun already. I’m calling it “Ascent” for now – probably to a big deal in remembrance of an old breathtaking game where you would fly around in mine tunnels. The idea is that it should not be only a game, but instead something that would make it incredibly easy for other people to add to it – more of a framework, so to say, than just a game. The goal is that you’d be able to write your own level in a ridiculous amount of time, like, say, ten minutes. Stuff like menus and appearance are easily customizable. Adding your own models is a piece of cake. You name it.

Yeah, I know, that’s quite a challenge. And even though it has a space setting, which solves a lot of problems I had with RavenJS, there’s so much I have just no idea how to achieve. And if it’s just for math – I quit studying math after two years or so, and now it all comes back to me… I just recently learned that there is quaternions, a number system quite frequently used to work with numbers describing objects in euclidian space. Dude. A new number system. But, I’ll manage. I’ve rarely been that enthusiastic about a project.

And I hope that people will help me out and correct my mistakes and foolish assumptions. So, here’s the marketing:

This will be awesome! You’ll hack a new level in no time, call your friends and start playing away, shooting rockets at your enemies from your fancy spaceships! It will be epic!

Right now, everything is a little bit hacky, and there’s not much to show, but the foundations are laid. I’m still a little bit afraid to put it up on GitHub just yet, as it’s really so-far-from-anything-close-to-presentable. I’ll try to add a couple of things and do some cleanup, just to prevent people from pointing their fingers at me and start laughing So, here’s a video I quickly whipped up showing it in action:

Three.js' Ray caster successfully detecting a Pulsar transport vehicle.

Unfortunately, three.js’ Ray class currently doesn’t detect intersections with imported Collada objects. Unfortunately, because I heavily rely on imported models and I’m too lazy to do the detection manually.

But, the good new is, the collada objects carry all information needed for the ray caster to work properly; you just need to do some manual tweaking.

Step 1: Modifying the Ray Class

The ray class checks whether the passed object is either a THREE.Particle or a THREE.Mesh. Usually, when adding a Collada object to the scene, you add it’s scene property. And this is not a THREE.Mesh, it’s a THREE.Object3D. So we modify the check (in src/core/Ray.js, at the time of writing on line 72):

These are the two things the ray caster needs to do it’s work.
 

Step 2: Modifying the object added to the scene.

Update: Please check the “Update” section further below.

Let’s assume your callback for the Collada loader looks something like this:

This is where we need to add the geometry object needed for the ray caster. The collada object has a dae property that contains loads of stuff, including animations, images and cameras. And, there is a geometries property, this is what we are looking for. It contains all geometries that were found in the xml file, but, sadly, as an object. That means, you need to know the name of your mesh. You can look it up in the file or just check the property name in the console:

Expand it further, and you’ll see a mesh property, containing a geometry3js property that holds a THREE.Geometry instance. Bingo! That’s the one we need. In my case:

Now just add that thing to the object that’s going to be added to the scene:

 

Step 3: Fun (and profit, maybe)

Now the modified Ray class can detect intersections with the Collada object. Of course, you have to look up the mesh id. And you’d better have only one mesh in your Collada. But given the fact that you’ll probably have a limited amount of imported models and that these should ideally contain only one mesh, this should be a doable task. If you still want an automated approach and do not fear assumptions: Iterate over the geometries object, go into the mesh property of the thing you encounter and return the instance stored in geometry3js:

You could even do that in the Collada loader class. But keep in mind that if you have multiple meshes in the original file you have no idea which one is going to end up in the object; I for one prefer doing it manually during the process of adding the Collada to the scene.
 

Update:

There is another, very convenient way to add the geometry to the object, that also works for Colladas containing multiple meshes. As always, reading the source code just one more time really helps… After loading, the Collada object contains everything that was imported from the XML file in a children property: Lights, cameras – and the mesh. So what I’m doing now is this:

For large models containing multiple meshes, the results still are a little awkward, but it works reliably for models containing one mesh, and it saves you from looking for the mesh property by hand.

Update: Stephan recommended to put the video on top,  so here it goes: Some impressions from RavenJS, enjoy (fullscreen and headphones recommended)!

The Beginning

When we started, I had little to no knowledge about working with 3D and OpenGL – I’m a JavaScript guy, after all. When I first fired up Blender and tried to work with it I was pretty close to dropping the whole idea again. Same happened when I saw how shader code looked like. But I so much wanted this – and the existing wrappers like GLGE or three.js enable us to still work with WebGL even if we don’t have the background knowledge.

So I decided to go with GLGE as I had already played around it with it’s examples before and because it already had nice collada support, which I decided to have as format for meshes and animations.
The first step was to extract some assets from the original game, collect mesh and texture data and get it into Blender. Luckily, Gothic has a highly active modding community, and this community is supported by the original game developers. So there were a couple of tools that could be used to aid in this, and it didn’t take long and we had a sword rotating in a browser.

But if it was possible to load a sword into GLGE, then shouldn’t it also be technically possible to load parts of the world into it? Yes, it was. It took some time, and it’s still far from perfect, but it worked. The next step was to create a first person camera that would react to mouse and key events – and had collision detection. Real collision detection, a height map would get us nowhere, as the levels had bridges and indoor locations, and I needed moving objects, too. Luckily, GLGE had some demos using raycasting to achieve this that I could use as start and modify until it worked as needed.

To recreate the dark intense atmosphere I played around with the right ambient lighting, fog types and added a sky box. Well, I would have loved to have a real sky dome, but, alas, I have no idea how to do this… Finally, I added objects, both static and animated.

In action

I can’t and won’t put online a live demo, as I obviously have no rights to do so. The copyright of the meshes and textures used belongs to the original developers, Piranha Bytes, and the RavenJS project is in now way endorsed, encouraged or supported by nor affiliated with them. But I can show you some impressions of what it looks like so far (fullscreen and headphones recommended):

Update: Video is now on top

(The song in this video is called “A special thing” by “amanyth” and published under a Creative Commons license.)

Issues

Besides the mere amount of free time that is needed and lacking, there are currently four main issues that are a bit of a show-stopper:

Animations caused a big deal of problems, and it’s still not prossible to run more than one animation sequence – running multiple animations will screw up bone position. While the animations still run fine in Blender, something goes wrong during the export. I’m still to find out if this can be overcome by using an alternate format for export.

Another thing that is still unsolved is “double-faced textures” that have transparent parts (look at the trees in the video). Those transparent parts become plain black when rendered in the browser. I’ve read up on this, and it seems that this scenario is indeed not trivial… From what I understood so far, in our case there’s two shapes “glued” to each other and each one has its own material, but I still have no idea what exactly goes wrong.

A real biggie, performance-wise, is that there is no logic yet that handles display of objects that are not visible. In the original game e.g., doors and the likes are “closed” by vertices with solid (non-transparent) textures and everything that’s behind it is taken out from the whole rendering/object interaction/whatever process. Only when you get really close you can see through these and see the intereour of the adjectant room. If these vertices get actually removed or if you “portal” through them I don’t know.

Mouse lock. Of course. No further explanation needed, we’re all looking forward to seeing this in the browser.

Future

I always loved 3D games. And the fact that it’s possible to render 3D worlds in a browser – with acceptable performance – is just stunning. The majority of work is still ahead, I’m well aware of that, but I *so much* want this to happen… So I just hope that some of the major problems can be solved and this project can go on!

Please grab all phones you have and navigate to http://jensarps.de/tests/storage-tests/.

There are four tiny tests there. Please do them all and report your results for each phone. If you like, include your twitter handle so I can say thank you!

Your help is greatly appreciated!! Thanks a ton!!

Among other tests, the main thing I wanted to know is how long does it take to read or write 1000 entries from/to localStorage. The keys were of the form “key000″, the values “val000″, with increasing numbers. So the tests were using minimal data – only 6  bytes – but the results are still useful to get an idea of how it goes – and how the browsers perform compared to each other.

Desktop

All test were run on a Mac Mini. Opera and IE were tested on a virtual machine, so there might be an additional performance penalty based on virtualization. However, here are the numbers (average time needed to run the respective method 1000 times):

Safari 4:
- getItem: 1.4ms
- setItem: 1.7ms
- key: 0.8ms

Chrome:
- getItem: 240ms
- setIem: 280ms
- key: 230ms

Firefox:
- getItem: 77ms
- setItem: 2100ms
- key: 25ms

Opera:
- getItem: 19ms
- setItem: 20ms
- key: 18ms

IE8:
- getItem: 5ms
- setItem: 72ms
- key: 3.6ms

Mobile

As we can see, Safari is lightening fast. That made me want to run the tests on a mobile phone that has a modern Safari browser. I took an HTC Desire, wich comes with Android 2.1, and thus has a Safari capable of localStorage. Here are the results:

HTC Desire:
- getItem: 16.4ms
- setItem: 19.5ms
- key: 8.7ms

This is pretty amazing, considering that it is a phone. It still beats most of the browseres on the desktop.

If I find that test page again (uh, yeah, it’s lost somehow), I’ll run the tests again with newer browser releases and publish the link here, so you can run the tests yourself.

So here you go: dojox.storage.encrypted, a Blowfish encrypted storage. It sits on top of dojox.storage, and you get all the dojo storage manager goodness, mainly the automatic selection of the best storage provider available. It exposes the complete API that dojox.storage does. If an attacker gains access to the storage area, he can still nuke the storage, but the data found within will be useless.

I’m not completely happy with it’s architecture, and I didn’t have the time yet to fully test it with all providers, but it works pretty well with localStorage and the performance is not too bad. If you have a spare minute and are interested, feel free to to test it yourself (ah I know, you don’t have the time yourself, but I’d be thankful!)…

Usage

Just use it as you would use dojox.storage, except that you need to set your passphrase before you start working with the storage:

dojo.require("dojox.storage.encrypted");
var sto = dojox.storage.encrypted;
sto.setPassphrase("my super secret passphrase");
sto.put('key','value');
var value = sto.get('key');

It uses/requires dojox.encoding.crypto.Blowfish but that’s been in dojo for ages, so no worries.

To use it, just put the encrypted.js file in your dojox/storage directory.

Files

the code (encrypted.js)
the test page

Update II: Ben Hockey proposed to do the whole thing without using the pub/sub system. I for one think its a good idea, but in case you want the topic version, I’ll leave the test page and add another one with the topic-less version. Thanks Ben!

Update III: Dojo 1.5 introduces robust Promises with dojo.deferred as Krys Zyp explains in this SitePen post.

—

One of the things/ideas/concepts that I really like about CommonJS is Promises.

Promises? A brief explanation from the CommonJS API:

Promises provide a well-defined interface for interacting with an object that represents the result of an action that is performed asynchronously, and may or may not be finished at any given point in time.

To hear more about Promises, I highly recommend these two posts.

That concept is not entirely new. The dojo toolkit has dojo.Deferred, which does a similar job. You create a Deferred, add (multiple) callbacks and error handlers to it, and when it fires off, all your handlers get called. But dojo.Deferred is pretty heavyweight. It’s loads of code. And it works with .addCallback() and .addErrback() methods – not with .then() methods, which provide maximum readability.

The goal

What I want to have is the follwing code running (with dojo):

asyncComputeTheAnswerToEverything().
    then(addTwo).
    then(printResult, onError);

(This is the example code from the proposed CommonJS Promise API)

And I want the execution to continue right after doSomethingAsnyc() – the async call should still be non-blocking. The .then() methods should recieve the results of the previous part of the chain, but still be non-blocking. Luckily, dojo provides a tool that allows us to achieve this: the publish/subscribe system.

Let’s do it

When we have unique topics for each promise made, we can subscribe the .then() method to the current topic in the chain, and publish the next topic when the data handlers are done computing. Within the then() method we need to check if the data we recieve is an error or not, to determine wich handler we need to call. The spec also demands a progress handler – if we wanted to implement that, we needed to add another topic, but I left that out.

We only need to kick off  the whole thing when we create the first promise, then the rest takes care for itself. Including the changes proposed by Micheil, what you have to do is pretty trivial and you don’t have to care about unique topics or the pub/sub system at all. The asyncComputeTheAnswerToEverything() function mentioned above would look like the following:

var asyncComputeTheAnswerToEverything = function(){
	var promise = new dojox.Promise();
	window.setTimeout(function(){ // some async operation
		var data = 42;
		promise.emit(data);
	},2000);
	return promise;
};

You create a Promise and store a reference to it. You return the promise and use the reference to it to kick off the chain when the data is there. Everything else runs on it’s own now.
Go to the test page or theand check the source – it contains the Promise code and the example functions. Open the console and run testSpec(); to start off the example code from the CommonJS API proposal.

Usage

As a namespace to put the Promise into I chose dojox – but choose whatever floats your boat. You can just drop the promise code somewhere into your code, or, if want to dojo.require("dojox.Promise") it, create a file named Promise.js that contains the code and place a dojo.provide("dojox.Promise"); as the first line in it. Save the file in the dojox directory and you’re done. (Well, to be honest, this violates the dojox naming convention, but if its just in your project, it’s ok and it will work.)

Any questions left? Feel free to leave a comment, @ me at twitter or drop me a line.

References / Files / More

• CommonJS Promise API Proposal
• Post by Kris Zyp @ SitePen blog about CommonJS
• Post on www.michaelharrison.ws about promises and more
• the test page (using topics and pub/sub)
• the test page (w/o topics, recommended)

The Curve

Every dojo.Animation needs a curve. If you call animateProperty or it’s shorthand anim, you don’t have to provide the curve for yourself, dojo will do it for you – if you build an animation using new dojo.Animation(), you have to provide your own curve. You can think of the curve as of a line that the animation walks on. The line is already present before the animation starts to walk it, and it can always take a break and pause in the middle and eventually continue walking where it had stopped. To create a curve, you need to call new dojo._Line(start, end). In return, you get an object with three properties: end, start and a getValue() method.

var anim = new dojo.Animation({
	curve: new dojo._Line(0,100)
});

Easing

The curve will then be modified by a easing function. By default, this is the linear dojo._defaultEasing. Easing breathes a bit of life into the curve, you can say. Other properties of dojo.Animation that affect the curve are rate and duration. rate is the timespan between two frames. To calculate the targeted fps, use (1000/rate). By default, dojo aims at 50 fps.

var anim = new dojo.Animation({
	curve: new dojo._Line(0,100),
	easing: dojo.fx.easing.quadInOut
});

Waking Life

To get some numbers out of dojo.Animation, let’s let it go wild and play(). When using animateProperty or the likes, you just fire and forget. In our case, we need to get the numbers. To achieve this, we connect to our animation’s onAnimate event. This is fired on every frame of the animation. When your connected function is called, you get – who would have guessed – the current value of the curve. Now you have a function called at a given rate, for a given period of time and you get get a numeric value that is already calculated for you. Decent!

dojo.connect(anim,'onAnimate',function(curveValue){
	// do something
});

To see some curves in action, go to this test page, open the console and type play(1); and play(2); and check the source code to see what’s happening.

Multidimensional Curves

dojo._Line generates only one-dimensional curves. But, there is dojox.fx._core, which contains an (old) version of dojox.fx._Line, which is capable of creating multi-dimensional curves. To use it, you have to explicitely dojo.require("dojox.fx._core"). It’s a bit odd, but the _core file contains only the _Line, and it is not loaded when requiring “dojox.fx”. Well, whatever, you can pass arrays of numbers into dojox.fx._Line:

new dojox.fx._Line([0,2],[100,6]);

Now you have a multi-dimensional curve. Or, if you could say, you have two curves in one. You can (technically) pass an unlimited amount of numbers into dojox.fx._Line. When you hand this curve to a dojo.Animation, it will animate all dimensions of the curve. And it will modify all dimensions of the curve according to easing, rate and duration. If you connect a function to onAnimate, it will now recieve an array of numbers.

var anim = new dojo.Animation({
	curve: new dojox.fx._Line([0,2],[100,6]),
	easing: dojo.fx.easing.quadInOut
});

Head over to the test page again, and type play(3); into the console. Now, the width of the bars is also animated – by the same animation function that calculates the height of the bars.

What now?

If your mind is still not blown away with ideas about how to use this, let me give you a start:

• animate position, width and height of a browser window and use it as a screensaver for some unknowing, yet thankful visitor…
• take some CSS 2D and 3D transisitions/transforms and animate away…
• canvas! No need to say another word…
• Still too banal? Go WebGL! Grab some 3DContext and animate an evil ball of jelly jumping around!

In short: dojo’s animation system allows you to not only animate CSS properties, but gives you a good tool for  animations in general. You get the curves on the one hand, but you also get the “player” for these curves, including play/pause/stop/gotoPercent methods. And dojo.fx.easing has a lot of easing functions in stock.

Uh, persistant local storage?

Ahm, yeah, in case you don’t know, that means a client-side storage, be it in a browser, or in an Air app. It’s not very popular, but that concept has been around for a long time. So why local? There are two major reasons for it: First, we need this for apps and tools that work offline – and apps and tools that work online but need an offline backup and sync later. Kris Zyp wrote a post about the JsonRestStore and OfflineRest back in 2008, he goes a little into detail there. Secondly, we need it for apps and tools that rely on persistence for other reasons, no matter if online or offline – like it happended to me. When I ran into Lawnchair, I had the idea to build a tool that was monitoring CSS selector usage on websites and apps (you know how you sometimes lose control about CSS selectors in webapps…). To achieve this, I needed to store data locally, and persistent.

What dojo offers

As I was using dojo’s excellent CSSRuleStore to find out what selectors where used, I wanted to switch from Lawnchair to a dojo solution. Dojo has dojox.storage (the above mentioned OfflineRest uses dojox.storage), which provides wrappers for the following backends: Gears, Flash, globalStorage and Adobe Air (globalStorage is Firefox’s early implementation of the early WhatWG storage draft). Hm. Not that much. Gears and Flash are 3rd party solutions, globalStorage is a Firefox-only-thingy (and old, though it’s still supported in current Versions) and Air is non-browser. That means: No IE without plugins. No Safari when Flash is not available.

What dojo does not offer (but should)

Guaranteed, reliable support for *any* browser. Without depending on 3rd party plugins. As of now, to have this, you’d need to use Lawnchair or persist,js, which is pretty complete. Speaking of it, let me quote from it’s readme (it’s a good read on local persistent storage in general):

The most notable attempt at addressing this problem [storing client-side persistent data] is probably Dojo Storage. Unfortunately, Dojo Storage does not support Internet Explorer without Flash, and it does not support Safari or other WebKit-based
browsers at all (at least, not without Flash). Also, Dojo Storage is
not standalone; it requires a several other Dojo components in order to
operate.

Whoa, even other JS tools mention the lack of support in dojo…

New providers

Attached to this post are wrappers for the following mechanisms:

• localStorage: this is the current spec, and it is supported by IE8, Firefox 3, Safari 4, Opera 10, Chrome 5.
• userData behavior: this is Microsofts client-side storage for IE < 8.
• cookie: worst case fallback solution…

With these three, there’s an almost 100% coverage of browser situations (see storage overview). There are still no-go scenarios, of course, such as IE7 with userData disabled and cookies disabled, but there’s no way to address them all.

Notes on BehaviorStorageProvider

This one uses the userData behavior, present in IE from version 5 up. It works with DOM elements that have a special behavior assigned. You can then use  set/get/removeAttribute() methods to store data. It has some limitations though – of course…

• Maximum size varies from 64k to 1M – depending on the zone the site is in. So, you can’t rely on more than 64k.
• UserData can be switched off in security settings.
• I couldn’t find a way to read out stored keys if you don’t know them. Maybe this is some security thingy or just a bug – but it means that some methods in the provider don’t work. (that is, getNamespaces, getKeys and clear). To work around this, we have to take some of the precious storage space and store the keys and namespaces seperately.

Usage

Copy the the providers into your /dojox/storage directory and add the needed dojo.requires to /dojox/storage/_common.js:

dojo.require("dojox.storage.LocalStorageProvider");
dojo.require("dojox.storage.GearsStorageProvider");
dojo.require("dojox.storage.BehaviorStorageProvider");
//>>excludeStart("offlineProfileExclude", kwArgs.dojoxStorageBuildOption == "offline");
dojo.require("dojox.storage.WhatWGStorageProvider");
dojo.require("dojox.storage.FlashStorageProvider");
//>>excludeEnd("offlineProfileExclude");
dojo.require("dojox.storage.CookieStorageProvider");

What’s left?

Well, it would be pretty cool to have a dojo.data API compliant store that uses dojox.storage as backend, so that you could write your apps using dojo.data calls and have a reliable offline backup. I started working on that, but then came Christmas, parents, vacation, new year and a lot of beer. But hey, it’s 2010 now, and there’s plenty of time left!

Storage Overview

Files

LocalStorageProvider.js
BehaviorStorageProvider.js
CookieStorageProvider.js
Storage Test Page

Besides indexOf(), there are some other ways to achieve this, so – let’s have a competition and find out who’s the fastest!

The Contestants

replace and length

One possibility is to replace the searched string with an empty string, read the length property of the modified string and compare it to the length of the haystack. If it’s different, the haystack contains the needle. Reading the length property is extra work, but comparing two integers is faster than comparing two strings, so maybe it’s faster in general.

replace

Again, replace the searched string with an empty string. Then compare the the modified string with the haystack. If they are different, the haystack contains the needle.

split

Take the haystack and try to split it using the needle as the seperator. If the result’s length is greater than one, the haystack contains the needle.

indexOf

Find the first occurance of needle in haystack; if the result is something else than -1, the haystack contains the needle.

Results

For testing, I did 10,000 iterations and retrieved the execution time in ms. The methods were tested in the order presented above.

On Chromium (Mac build, Version 4.0.203.0 here), replace + length is slightly faster than replace, and both are faster than split. indexOf is by far the fastest.

1) true: 7 / false: 3.5
2) true: 7.5 / false: 3.5
3) true: 10 / false: 8
4) true: 2.5 / false: 3

Safari 4 has nearly the same results as Chromium, but the numbers tend to differ a lot from test to test.

1) true: 4-18, avg 10 / false: 2.5
2) true: 4-19, avg 10 / false: 3
3) true: 6-22. avg 13 / false: 6-23 avg. 16
4) true: 2 / false: 3

On Firefox 3.5, split a bit faster than the two replace methods, but indexOf is again by far the fastest.

1) true: 13 / false 10
2) true: 13 / false 10
3) true: 10 / false 9
4) true: 1.5 / false: 2

On IE 8 (run in a VM), both replace versions perform almost the same, and faster than split. Again, indexOf is fastest. Only 2 – 3 times faster than the replace methods, but still the fastest.

1) true: 35 / false: 25
2) true: 30 / false: 25
3) true: 60 / false: 50
4) true: 15 / false: 15

You can run the tests yourself, if you are interested, the test page is here.

Summary

The results are pretty obvious: indexOf() outperforms the other contestants. Which is not really a surprise, considering that whatever Javascript does during indexOf() –  it has also to do the same before being able to do a split() or replace(). So, the proposed way for a contains() method is the following:

dojo.string.contains = function(/* string */ needle, /* string */ haystack, /* bool */ caseInsensitive) {
    if(caseInsensitive) {
        needle = needle.toLowerCase();
        haystack = haystack.toLowerCase();
    }
    return haystack.indexOf(needle) !== -1;
}

If you want to use contains() in your code, just copy the above lines somewhere in your code. Again, don’t forget to dojo.require(“dojo.string”) before.

Or, if you want to have beginsWith(), endsWith() and contains() all-in-one, use this: dojo.string.addons.js