Website Still Down

The Render Engine website it still down.  I've spoken with my hosting company and they tell me that the machine died but that the site data is safe.  I'm not sure when the site will be live again.  Progress on v1.5.4 has been hindered by this, and other factors (namely the holidays), but I intend to get it finished as soon as I possibly can.

Dirty Rectangles III

The website is still down - Not sure when it will be up again...


More time has been spent on dirty rectangles to try and eek out some performance during rendering to canvas.  There are basically two methods which I am currently fighting with:

1. Brute force method
2. Joined rectangles method

The first method would go through the collection of objects, and for each dirty object it would grab an image of the space behind it and store that.  For the next frame, it will restore those stored rectangles and then the process starts again.  Objects which move, but then stop, would need to update the layer to which they draw to keep this process somewhat automatic.  Otherwise, objects that stop being dirty would be lost during the next frame redraw.  The second method would use something similar, but instead of capturing an image for every dirty object, it would combine rectangles which overlap.  The difference between the two is timing.

With the first method, we just go ahead and grab dirty rectangles for each object regardless of whether another object is grabbing that rectangle (or part of it) as well.  Thus, if there are 30 dirty objects, we grab 30 rectangles each frame, draw 30 objects, then for the next frame we restore 30 rectangles and do it again.

If we combine rectangles, it might add up to a savings if the objects overlap.  However, time will be spent determining the overlap and joining rectangles before actually grabbing the images to later restore.  I'm not sure if the trade-off is worth it without actually testing it.  The actual method is pretty intensive and requires visiting objects, which have already been visited, repeatedly.  This might end up causing more overhead than just a simple brute force method.

The other issue, with dirty rectangles, is the particle engine.  Capturing hundreds of tiny little rectangles for the particle engine is messy, and time consuming.  The actual time saved is lost because we're doing twice the work for all of those particles.  For larger objects it makes sense.  If we can eliminate the need to redraw 30 objects, if only 3 are moving, then we save time.  There are some tricks for particles, such as using bitmaps for the explosions which simulate particles but that might look bad.

Maybe using the Asteroids Evolution demo to test all of this isn't the best idea, after all.

Dirty Rectangles II

The site is currently down.  I've sent an email to my provider and hope to know something shortly.

Yesterday I was working on a dirty rectangles method.  What it has boiled down to is something similar to a SpatialGrid, whereby I subdivide the context into smaller blocks and determine which ones contain dirty objects.  A dirty object would be anything that would change how an object is rendered: line style, points, position, rotation, etc.  Additionally, I'm using the z-index to group objects together into "layers" which are rendered together.

Object2D has the idea of a z-index which allows us to position objects in front of, or behind, other objects.  Using this z-index would allow me to organize objects into rendering "bins" and perform updates for each bin as a whole.  So each frame that is generated would:

1. Go through the bin and see which of the rectangles contain dirty objects
2. Capture the contents of those rectangles (the dirty rectangles), and 
3. Draw the objects for the bin.  

Each bin would process separately, so it would be possible to have some background layers (objects that don't change very often), and active layers where things do change often.  Then, on the next frame we would restore those dirty rectangles, and start the process again.  This should reduce the number of times that we need to reset the entire context before redrawing, which in turn reduces the amount of drawing that must occur each frame.

Additionally, each object that isn't dirty doesn't need to be redrawn because it hasn't changed in any way, further reducing the number of drawing calls that must occur.  Objects still need to be processed, but most of the time is spent rendering the objects anyway.

Tonight I plan on implementing this method.  If anyone has any input for this method, let me know.  I've already considered sparse rectangles, but the simplest (and quickest) way I figure it could be done is via subdivision of the viewport.

v1.5.4 Coming

There are some things that have been going into v2.0 that would just make sense to get into a new point-release. So, I'm going to get a v1.5.4 together and tested.  The new stuff is mainly some new functions in Math2D and some updates to the math primitives, all regarding transformations and stuff along those lines.  No new features, per se, but something that would be useful to people currently using the engine.

Asteroids and SAT

Well, it's been about a week since I last posted something so I wanted to get a new entry out to let people know I'm still actively working on the engine.  As those who follow my posts may know, I've been working on getting a real narrow-phase collision system into The Render Engine for a while now.  Tutorial 11, in v2.0, is a simple example of using SAT.  However, I wanted to really push the extreme and get a good test for the new SAT convex collider, collision hulls, and fix up a lot of part of the engine.  This stuff is all new, but it's finally to a point where I can show it off:

Asteroids with SAT

The full game isn't working, there are bugs, but at least the attract screen is showing promise.  All of the asteroids are using the convex collider component and collision hulls.  The link above has debug mode enabled so you can see the debug helpers in action as well.  You can see the blue outline of the convex hulls (which are automatically calculated from points of the shape).  It doesn't run super fast, but removing the "?debug=true" from the link will run it without the helpers and provide a speed increase.  This runs fastest in Chrome and Firefox 4, at the moment.  Also, you should know that asteroids don't start checking collisions until they've been "alive" for at least two seconds (it's not a bug).

What this also points out is that I need to get the work that I've been doing with dirty rectangles working.  The framerate is dependent on how quickly a browser can redraw the entire frame, which isn't the best way to do things.  I should be able to push a lot more collisions on that screen, but at the moment it's shaky.  Additionally, the particles are really slowing things down because there are so many of them.  This is just a good point when it comes to using all of the effects in the engine - don't do it unless you're okay with a low framerate.  The Asteroids Evolved demo is intended to be a demo, not an example of a full game... Maybe in the future it will run a lot better and I can polish into a real game, but for now it's a tech demo and nothing more.

Debug Helpers

When you're developing a game, it can be very helpful to "see" what's going on with your objects.  Thus, in v2.0 I've been merging the debug helpers into the components directly.  When you enable engine debugging with Engine.setDebugMode(true), or use the debug=true query parameter, it will enable all of these debug helpers.  What they are, are things like bounding boxes or collision hulls for collider components, the origin axis for transformations, and image outlines for billboards.

These kinds of helpers just make it easier, at least on me, to verify that something is occurring and that it's occurring correctly.  For example, last night I was trying to figure out why some changes I made to the Asteroids demo were causing the objects to spin about an origin that didn't seem to be where I thought it was.  Enabling the debugging helped me to see that the origin had been moved and correct the issue.  Using debug mode increases the frame render time, since it has to draw these helpers, but it's only while debugging is enabled.  All of the helpers are in pragma:DEBUG blocks, so they won't be compiled into the production version of the engine.

There have been some changes to some of the components, including the Vector2DComponent.  I noticed that render components were setting the bounding box for their host object.  Originally I intended this to be the way render components worked, but I've realized that while this is nice to have, it can also be painful to debug.  Plus, when I changed the way bounding boxes are created (they now have a top-left origin of 0,0 and only a width and height) this broke because most vector-drawn objects have their origin at the center.  Transforming the origin would typically put it at the lower-right corner of the object box, so it was breaking all across the board.  As such, a vector component will calculate a bounding box that you can retrieve from it (and now a convex hull) but it becomes a manual process, rather than an automatic one.

Also, I got the abstracted SoundSystem working last night with the Sound Manager 2 system plug-in.  It was a nice feeling being able to plug-in and remove the sound system entirely, and still have the game function regardless.  Now I will start, in earnest, making the HTML5 system plug-in.

I suspect that when people try to migrate from versions of the engine, prior to v2.0, they will be a bit shocked.  I've been noting these changes in the release notes for v2.0 mainly because there are a lot of little things like this.  It's all in an effort to make the engine faster, more optimized, and cleaner.  The difference between the v1.5.3 engine and the v2.0 engine will be fairly dramatic.  But overall, I think people will be happier with it...  It's really grown up a lot.

Debugging with Aptana

If you notice a reduced framerate and  you can't figure it out, try opening your debugger's "Breakpoints" tab and removing them all.  I've been hunting (for days) where the slowdown was occurring, profiling the engine and all of it's parts to no avail.  Finally, when I couldn't see my current breakpoints (valid ones) anymore, I removed them all... lo and behold!  The framerate issue disappeared.

Apparently Aptana still transmits all of the breakpoints, but just because they aren't valid doesn't mean that Firebug isn't checking them.  <sigh>  What a waste of my time - but a valuable lesson learned!

Profiling the Engine

Last night I did some research into what's eating up cycles when running a simple test (I used tutorial 11 from the v2.0 branch).  It appears that rendering is where the majority of the time is being spent.  I tested everything from Container to PooledObject, Iterator to SpatialGrid in the hopes of finding something that was really devouring time.  When I finally got to SpriteComponent, RenderContext, and CanvasContext, it came down to drawing paths and drawing sprites.  While most of the profiles were returning < 1ms average for every run, these areas (especially drawing images) were taking the most time.  Most profiles would execute 2000 times with a combined total of 300 or less milliseconds, while the actual drawing of images was doing 1800-2300 ms total for 2000 execution cycles.  While it isn't something I can directly improve upon, there are some things which can be done to mitigate the issue a bit.  But this brings up the question: is it worth spending the time on?

Dirty Rectangles

Recently I've been looking for a way to optimize rendering in the engine.  A lot of ideas are out there, but the best one (by far) is "dirty rectangles".  This is something that will apply to the canvas context since it's typically wiped for each frame and completely redrawn.  I'm just starting on this so I don't expect to have anything for a little bit.  But, if this results in increased framerates (like I hope it will) it'll push the engine to a place where people can expect more from it.  As it stands now, a game is completely limited by how quickly a browser can redraw the entire canvas.

Really Close Now

Yet another thing I'll soon be able to check off my list of stuff to get done for v2.0... Yay!  SAT collisions are almost done, and are working fairly well.  If you click the link below you can try it out.  All three types of collision hulls are in use in the tutorial now.  I ended up reverting Tutorial 9 to what it was and creating a new tutorial.

Tutorial 11 - Convex Hulls and Particles

There's still a few small things which need tweaking, but it's so close to being done that I'll be able to start working on some of the more interesting demos.  I'll probably also update the Asteroids demo to use collision hulls and SAT.

SAT Collision Detection

I spent time, this evening, working on the SAT implementation.  I got it working, partially, but it needs some tweaking.  I think I am missing something.  When the OBB for the player is rotated, it treats the collision as if it were not rotated.  Very odd, but I'm starting to get the actual math and hope to have a solution soon.  Check out the link below for the updated example.  Once you pick up a powerup, the next collision with a powerup will stop you from collecting (it's supposed to stop movement entirely, like it does when moving from right to left)

Tutorial 9 with SAT collisions

I also wanted to give props for where I found this code.  I tried to decompose the "N Tutorial", but aside from nice flash demos it left a lot up to the inquisitive mind to figure out on their own.  Instead, I found this:

Rocketman Development - SAT Tutorial

Convex Hulls

I know this doesn't look like much, but I've been working on SAT collision detection and just got convex hulls working.  There's not really that much that is done with them, at the moment, but you can see (in the attached demo) that they are getting generated.  In this case, the hull is generated from the sprite's bounding box.  Unlike the world bounding box (in blue) the convex hull (in yellow) rotates with the player.

Tutorial 9 with Convex Hull Test

Like I said, not much to see and the hull isn't even being used yet.  Hopefully I'll have more tomorrow or Tuesday...

CSS Transformations II

I wanted everyone to see that it's in progress, so here's a quick test that is only using CSS transformations.  There is no canvas involved in this example.  This will not work in IE8 or earlier (yet) so don't even try.  =)

CSS Transformations Testing

I feel it's important to point out that this isn't using the support method I previously talked about.  That was actually quite problematic and wasn't an ideal solution.  Instead, HTMLElementContext has transformation support built into it.  It also has a very simple transformation stack so it works like the canvas context.  By the way, this is running out of the v2.0 branch.

CSS Transformations

Most modern browsers support some sort of CSS transformation.  What this means is that a lot of what can be done only with Canvas, can also be done within the DOM.  There has always been a plan to include this and I think that v2.0 will be the place where it is developed.  In particular, the HTMLElementContext has had support for this for a while, but nothing was ever done with it.

Each class in The Render Engine can implement a class method called resolved() that will be executed as soon as the class dependencies are resolved and it can be initialized.  The HTMLElementContext uses this method to load one of two support files, either for Safari or Gecko.  These files would add the functionality for transformation support into the context, thus completing methods which are currently just abstract in RenderContext2D.

I just wanted to point this out because so many people have been asking about the HTML contexts recently.  While support is limited in v1.5.3, the next major version of the engine should have more complete support for this functionality.

Downloads Fixed

Someone alerted me to the fact that the downloads were broken links.  They have been fixed now.  There is a variable used in the build process that means "engine version" and another that means "build version".  I got them mixed up and lo' and behold, the download links were borked... <sigh>

User Interfaces in v2.0

After spending some time doing the due diligence on user interfaces, I have decided to remove them as part of The Render Engine.  Since the engine is running in a browser, it just didn't make sense to try to abstract what the browser is doing so well already.  While it would be neat to have a user interface be tightly integrated within the engine, I feel that it just eats up cycles better spent on the game.  User interfaces can still be created, loaded, displayed, and interacted with -- it'll just happen outside of the engine.  For example, you could load a UI with jQuery and display it over top of the game.  You'll have finer control over the interface's design and display, and you'll also have access to the game's objects if you want to display things like sprites and so forth.  So, in an effort to get 2.0 going again, and to focus on what makes a great game engine, I'm dropping UI's as a part of the engine core and leaving that up to the developer.  I think most people will agree that this is a better plan all around.

v1.5.3 Released

Just a quick note to let you know that The Render Engine v1.5.3 was released today.  There were a few small bugs that appeared while creating Tutorials 8 & 9.

The Render Engine v1.5.2

Last night I released v1.5.2 of The Render Engine.  This version is a small update to the 1.5.1 release to address some bugs identified in the ColliderComponent and its subclasses.  In addition to the fixes, I've added the ability to specify collision masks which should eliminate the need to perform instanceOf checks in the onCollide() callback.  Plus, a new callback has been added which will let your host object know when collisions have stopped: onCollideEnd().

Other things in this release:

• Transform2DComponent now supports non-uniform scaling
• Container's forEach() method wad updated so it will halt if the "this" object is defined and has been destroyed.  The loop can also be terminated by returning false from the callback
• ColliderComponent's execute() method will not throw an exception if the host is destroyed

See the release notes for full details.

v1.5.2 is Coming...

Some of you might have noticed that v1.5.2 showed up for a moment on Wednesday night.  I have pulled the tag and rolled back to v1.5.1 for the time being.  There was a regression in the collision components which needs to be addressed.  The API still shows as v1.5.2, but still applies to v1.5.1.  I hope to have v1.5.2 up by tomorrow night.

Sorry for the mistake!

v2.0 Refactoring

In v2.0 I have moved everything related to the engine into the "/engine/" folder.  Components, render contexts, resource loaders, etc. have all been moved in there.  Since the files are considered "engine files", it makes a lot of sense to make them subfolders of "/engine/".  The folder structure is a lot cleaner since the files are now "hidden" in the engine.  These files were always intended to be the foundation classes for a game, and not supposed to be modified directly.

Putting them in "/engine/" sort of enforces that concept.  Doing so allows someone to see the separation between their game and the engine.  I know this will cause some confusion, especially for people who want to upgrade to v2.0 of the engine.  However, the only real change is that files which were previously loaded via:

Engine.include("/engine/...");

Will now need to be:

Engine.include("/...");

That and the index page will need to load the libraries from the "/engine/libs/" path now.  This change will not affect previous versions of the engine, so only if you're using v2.0 and beyond.

Abstracting the Sound Engine and Collisions

Currently The Render Engine uses SoundManager 2, which was a great solution at the time since the audio tags weren't formalized and Flash seemed like the only way to play sounds reliably.  Now that the spec is formal, and after analyzing SM2, I have decided that abstracting the sound system would be a good idea for v2.0.  This would allow a developer to plug in a different sound system if they have a better alternative.  Instead of loading SM2 and using its HTML5 capabilities (originally what was planned) I'm going to create a layer of abstraction which allows for just such a thing.

SM2 will continue to work, but will instead be a pluggable sound system.  The actual SoundLoader and Sound objects won't change, but instead the loader will interact with the abstraction.  This is all planned for v2.0 but should be in the repository soon.  I really want a lightweight sound system, and depending on SM2 is pretty heavyweight but it does have lots of nice options...

I played around a bit with getting the fundamentals in for the Separating Axis Theorem last night.  While The Render Engine will support any collision system, I've decided to provide both GJK and SAT.  GJK is mostly implemented, and SAT is getting there.  It'll be nice to have some choices when using the collision system.  I still think that the SpatialGrid will continue to be the broad-phase of choice, but I'm also considering BSP which would open up the engine for raycasting and such.

v1.5.2 is in the works...

I was working on Tutorial 7 last night and realized that the collision system wasn't acting appropriately.  It was yelling, getting really drunk, and generally being annoying... I kid, I kid.  There were some issues with arguments being out of order in the ColliderComponent subclasses of BoxColliderComponent and CircleColliderComponent.  Plus, CircleColliderComponent was doing way too much for simple circle-to-circle collision checking so I've moved the code into Math2D for now.  Also, I realized that I was depending on instance checks which can be costly when executed repeatedly.  I'm always looking for a win when it comes to getting some milliseconds back to use elsewhere.

Well, of course this means that I need to fix up some code in v1.5.1 and release a small upgrade.  v1.5.2 will include the fixes for the two subclasses and also have the implementation of collision masking.  Plus I'm trying to figure out what's going on with IE and FlashCanvas when run remotely.  That's going to be all that will change, since there really isn't anything new that's ready.  Plus, I need stuff for v2.0!

If anyone's interested, here's a link to Tutorial 7.  You can't download the tutorial code yet and the example isn't hooked up yet.  If you see anything crazy, let me know.  Oh, and if you're trying to get the tutorial working in v1.5.1... well, you're gonna have to wait for v1.5.2.

Update to v1.5.1

After releasing v1.5 of The Render Engine we found some bugs which were quite critical.  Internet Explorer support was not working and some of the tutorials were faltering.  After some quick debugging, we've gotten these working and have released an update to version 1.5.1.

Version 1.5 is unsupported and should be replaced with v1.5.1 instead.

The Render Engine v1.5 has been released!

The v1.5 incremental release of The Render Engine has been tagged!  This version includes the Box2d Physics Engine and two examples of its usage.  Additionally, this release has numerous bug fixes and enhancements to the engine as a whole.  The API documentation has been updated and is available, along with the full and minified versions of the engine.

Now I can start to focus on v2.0 again...

v1.5 Ragdoll Demo

It's almost complete, just working out the kinks (no pun intended) in the joints.  Wanted to give you all a peek at what's coming with the v1.5 release...

http://renderengine.googlecode.com/svn/branches/v1.5/demos/physics2/index.html

You can't do anything with it, except restart it and watch the doll fall again, but it's almost there.  I also have the FlashCanvas support working in IE prior to IE9.  It's certainly not very fast since it is emulation using Flash, but at least all of the demos are beginning to work.

Asteroids demo:
http://renderengine.googlecode.com/svn/branches/v1.5/demos/vector/index.html

New physics demo:
http://renderengine.googlecode.com/svn/branches/v1.5/demos/physics/index.html

I'm still hoping for an end-of-week release.  My fingers are crossed!

v1.5 Delayed

Sorry to be the harbinger of bad news, but the release of the 1.5 update will be delayed.  There are a few things that haven't been fully tested and I want to make sure that everything is solid before releasing the update.  I apologize for the late notice, but I didn't want people waiting around for something that isn't going to happen.  I don't have a specific date yet, but I'm feeling pretty strongly about it happening before the end of the week.

Collision Detection

After reading some comments about The Render Engine it became apparent that the engine doesn't offer any really good way to do narrow-phase collision detection.  The SpatialGrid gives developers a quick broad-phase method of finding potential collisions, but when two objects are determined to be possibly colliding, it's up to the developer to implement some sort of narrow-phase detection.  I figured with the engine being what it is, it should at least offer one implementation out-of-the-box, so I've been looking into the GJK algorithm to solve the problem.

In the trunk I have added a method to quickly determine an approximate convex hull from a set of points, and also a method to compute the Minkowski Difference of the two hulls.  Eventually, there will be a method to perform GJK on the two hulls so that the amount of interpenetration can be determined and the appropriate collision response performed.  There are other methods, which I may implement in the future, such as the Separating Axis Theorem since it really isn't that far off from what I'm doing with GJK.

As an engine, The Render Engine needs to provide these operations to make game development that much easier on a potential developer.  Even though the engine does resource loading and rendering, it has to provide a more robust feature set for all potential problems which must be solved.  With the scene graph being fairly solid, and resource loading working as it should, I can focus on adding features which developers can use to enhance their game experience.

To me it seems like I've done a lot already, but I've only just scratched the surface.  To be a complete engine, there needs to be at least one implementation of certain features such as:

• Narrow-phase collision detection
• AI/Pathfinding
• Multiplayer/Networking
• Tilemap management
• UI Management

Of these, I was planning on everything except the collision detection.  As it stands now, the engine is already quite robust but it has a ways to go before it is truly a complete solution.

The Render Engine on Hacker News

An in-progress game, made with The Render Engine, called Pistol Slut was #1 on Hacker News today for a while.  The engine has also recently appeared in some beginner tutorials.  This only emphasizes the need to get v1.5 out the door as soon as possible!  I'm a little disappointed that the comments for the engine, by its author, were:
 

"The engine I used is mostly a drawing engine, with some basic game stuff included (particles, the beginnings of collision detection)."
 

It makes the engine sound a little amateurish.  Yes, it's only v1.0 of the engine, but The Render Engine is much more than just a drawing engine.  It's a framework that gives a game developer time to work on game mechanics rather than core functionality.  Is there something else that the engine should include that isn't there?  Anyway, it's still good to see it being used!

Announcing v1.5!

In an effort to get fixes out the door which have come up as part of the v2.0 development track, I am going to select some features and accelerate their development to get them out the door as v1.5.  The features I am thinking of including are:

1. Container and Iterator enhancements
2. IE support via FlashCanvas integration
3. New profile display
4. Updated resource loaders
5. Particle engine improvements
6. Basic physics (circle & box bodies, distance & revolute joints)
7. 2 Physics demos

I'm hoping to get v1.5 out before the end of October.  v2.0 will continue to be developed, but I want to get something out there because claiming support for IE, and having a fix for Google Chrome are pretty high on my list right now.  I'll have more specifics shortly.

New Website Launched

The look of the website was amateur and looked like it was done by a programmer.  Well, that's because it was -- me.  As of Oct. 13th, the new website has launched!  Interestingly, this website was done by myself as well but I borrowed design ideas from around the web.  Primarily, I liked the design style of Dojo Toolkit so I used their look and made it my own.

The new site took a week to assemble and a few hours to roll out.  I hope you enjoy the look of the new site!  Some things may be broken, but I believe I caught all of the "bugs".  Now back to the engine...

Cleaning up

It's been a thorn in my side, for a while, that I couldn't shut the engine down cleanly.  Spent a bit of time researching the issue and determined that it wasn't necessary to Assert every time an object didn't exist and it was being destroyed.  Instead, I've written them as Console warnings.  That way, a developer can still find orphaned objects or NULL objects and make sure they are cleaned up properly.  This lead to a few other issues which I was able to clean up and fix.  Now the engine shuts down as it did in v1.0 -- cleanly.  I'm a happy camper again.

Android Development Path

I've spent a lot of time trying to decide how best to approach development on the Android platform.  I tried to implement a bridge between a native rendering engine and the JavaScript engine but it was quickly becoming a real headache.  The direction I've decided to take is to convert the codebase to be a native Android library which will be available for purchase.

I really wanted to be able to keep the engine open, but the amount of time I spend on it does add up.  It'll be an inexpensive game engine for the Android platform, however.  I haven't even thought about price yet, but expect it to be a good value for the included features.  What this means for development is that it will follow behind the browser version of The Render Engine by about 2 months.

It will include all of the features of The Render Engine, and will include the exact same API.  Most of the work to convert a game from the browser based version to an Android version will only require you to rename your ".js" files to ".java" and build an Android project for it.  Some code will require changes, but I hope to create ANT tasks to simplify the majority of these common patterns.

On another note, after considering the results of the poll, The Render Engine will focus its future rendering context work on the Canvas, but will still support the DOM.  Separation of the two contexts will be more clearly defined, in the future, and support will be removed to allow simply switching between the two.  Once you've decided the direction you want to go with your game, it won't be as simple as switching to a new context.

Tilemaps and Isometric Rendering

A lot of engines seem to focus on tilemaps and games rendered with such.  They also seem focused on isometric rendering for that 3D'ish look.  This past weekend I wrote a quick test to load a set of tilemaps and then I wrote a map loader to initialize an isometric world.  So far the tests are going nicely and it's looking like something that would benefit the engine.  I plan on adding the isometric demo to the engine demos so that people can get a look at loading the tiles, loading the world, and then streaming a world over the web.  I'll probably also use it as a foundation for a couple of other demonstrations such as simple multiplayer and generic AI with pathfinding.  Eventually this demo will be a showcase of technologies provided by The Render Engine.

It doesn't supplant the 2D sprite-based demo and game editor, but it's a step in a new direction that should help show the flexibility of the engine.  It's hard to get people to understand that The Render Engine is just a game engine and not a full game.  Showing them a mostly functional Asteroids clone, a simple physics ball bouncing demo, and a text renderer doesn't really showcase what the engine can do.  Instead, most people think that The Render Engine is an Asteroids clone or "a game from the early 1980's" which it isn't.  It's a framework for creating games, and the demos are just examples of the capabilities of the engine.  None of the demos are intended to be full games.

However, after spending some time cleaning up the Asteroids clone it occurred to me that it might be interesting to take that and expand on it.  Make it into a full game which can be played on Facebook, for example.  Add some more features to it and work out an interface with the Facebook API.  That might really show people what is possible with The Render Engine.  Not that I want people to think that the engine is there to make the next "Farm-whatever", but it might just be another person's stepping stone.

What to do?

I've been spending a lot of time thinking of ways to optimize the engine.  But I've also realized that I'm wasting time trying to optimize when I should be doing new development.  I'm missing out on creating the new technology to push The Render Engine over the top.  So, starting tonight I'm stopping my optimizations and going back to working on the new stuff.  Maybe it's silly, but I get caught up in trying to make everything perfect out of the gate.  So, back to the UI, tilemap renderer, multiplayer, and other things.  I'll be sure to record what I was working on so I can return to it, but for now it's back to the new tech.  =)

Engine Configurations lead to Optimizations

Yesterday I revisited the new configuration module in The Render Engine.  What I probably forgot to blog about was that I've been working on a way to load browser-specific configurations to help tune and optimize the engine.  The first iteration was simple and only loaded a simple object with configurations for the type of browser being used.  Turns out, that was too generic to handle some of the things I want to support.

Being ambitious, I've updated the config module to load a configuration which should be more flexible.  For example the Firefox config looks something like this:

{
   "versions": {
      "4.": {
         "transientMathObject": true
      }
   },

   "platforms": {
      "Windows 7": {
         "4.": {
            "hardwareAccel": true

         }
      }
   }
}

What this configuration does is allow us to set up some options which differ from the defaults.  For example, transient math objects and hardware acceleration are disabled by default.  With Firefox v4.0b6, hardware acceleration on Windows 7 has been enabled by default so this config sets a flag which the engine (or a game) could use to take advantage of that.  Also, in Firefox v4.x, using transient math objects tests faster than pooling them.

So, they're just flags but they do help in making determinations while the engine, or a game, is running.  For example, knowing that hardware acceleration is available, the drawImage() call of the CanvasContext is much faster.  So, it makes sense to use the Billboard2DComponent in the vector demo instead of always drawing the vector object's polys.  When I profile in Firebug, the calls to drawImage() in Firefox 3.6.9 take up > 5% of the overall time (right at the top) with only 3000 calls, compared to just drawing the polys which takes ~ 1% of the overall time with over 120,000 calls.  However, using the billboards in Firefox 4.0b6 those 3000 calls take up < 0.01% of the overall time.

You should know that these are just guesses based on the defaults that the different browsers use.  As such, if someone disabled the Direct2D acceleration in Firefox 4.x, the engine can't determine that purely in JavaScript.  So, while these aren't foolproof, they will help the majority of people who don't mess with their configurations.

Spatial Containers

I'm realizing that there are a number of places within The Render Engine that need to be tightened, cleaned up, optimized, or rewritten entirely.  That isn't a bad thing, because it means that the engine will either be faster or easier to work with.  One such example is that I've been trying to figure out how, or where, I can use threading with WebWorkers in the engine.  I had thought that maybe collision detection could take advantage of just such a thing.

As I was studying the code, I realized that the methods which update the collision model are contained everywhere except the spatial containers.  How odd, I thought to myself, that I should make such a simple mistake.  It would take a very specific knowledge of the collisions containers to know how to create your own or to utilize an existing one!  So, I did some rewriting and moved the code out of the ColliderComponent and into SpatialContainer.  This way, as objects move about, all they need to do is let the collision model know where they are (and, of course, who they are).  That simplified a lot, and cleaned up the code dramatically.

By componentizing the system, from the start, I've been able to keep these inner workings well enough hidden from the typical engine's user.  But this was just atrocious (in my opinion) and needed to be redone.  It's not a speed improvement, but it does take the burden off of the user.  That is a win to me that was worth implementing.

I know there will be more, and some will be more difficult to implement than others, but in the end I believe that The Render Engine v2.0 will be a better engine because of it.  Now, back to trying to figure out how I can take advantage of threading within the engine...

Red-black Trees and the Container

The problem with the Container class is that it is a linked list.  While it is not subject to fragmentation, like Array, it is extremely fast for linear operations, and it gracefully handles the removal of items while it's being iterated, it absolutely blows when doing lookups.  A lookup in a linked list is an O(n), or linear, operation which is bad, bad, bad on large collections.  Particles are an example of a very large, very active collection which needs a lot of lookups.

I remembered something from my earlier days of doing game programming with the Torque engine.  They use a red-black tree for their memory management.  It is a nicely balanced tree with O(log n) lookups.  I read something, earlier today, that said looking up an item in a red-black tree with 4 billion entries is at most 32 operations.  So, I found a nice implementation of a red-black tree in Java and did a port to JavaScript.  Unfortunately, it doesn't implement the remove() operation...  I've found a decent example of the operation, but I've got to translate it as well.

What I'm hoping to accomplish is that a new subclass of Container will use a backing red-black tree for speedy lookups.  I think this will offer the best of both worlds when trying to keep things relatively fast.  It will mean more data overhead, but then again I don't care as much about a small footprint -- only that the engine perform well.

Particle Engine Rewrite

Last night I rewrote the particle engine to use a Container instead of an Array.  The problem with arrays was that they could get messy when considering the way in which particles are created and removed from them on a regular basis.  I haven't fully decided if it's any faster, as of yet, but it doesn't appear to be a slowdown.  I had run into an issue in the latest version of Chrome where the ParticleEngine.sortParticles() method was "freezing" the browser when called.  I have determined that it's due to just calling Array.sort() with a compare function.  If the comparison did anything with the actual objects (even just checking if they were null), the browser would stop for a moment.

By replacing the Array with a Container, I can use an Iterator which will inherently skip destroyed objects.  Containers are simply linked lists, so when an entry is removed the link is broken, rather than coalescing the list.  Some operations, such as adding one Container to another, are simply pointer operations.  Adding one Container to another results in a single list which contains pointers to both.  Modifying the second Container will affect the first, so one need to proceed with caution.  At some point, I'll probably add "deep copy" methods which will be slower, but will ensure that the two lists won't affect each other.

UI Development

Over the last few days I've been working on the new UI system, going back and forth about how best to implement it.  Originally I had thought that simple objects which aren't pooled or anything would be easiest.  They would simply be a DIV overlayed on the rendering context, and it was going fine until I realized how much work I was going to create for myself since I already had components to render sprites and images, play sounds, etc.  So, back and forth I went...

Turns out, I already had everything I needed to generate UI's.  The new UI design is finalized and development is continuing.  Here is how it all came out:

• UI's are HTMLDivContext objects
• UI elements are Object2D objects
• Positioning of UI elements is handled automatically via Transform2DComponent
• Sprites and Images are rendered with components

There is a UIManager which loads user interface files and parses them into a UI with all elements generated from the UI file.  The part I'm working through right now involves how to position elements within the UI.  I want to give some flexibility so that UI elements can be positioned relative to each other.  That way, if you have a series of buttons which all need to be left aligned together, you only need to absolutely position the first then relatively position the remainder.  Next up, after that, will be enabling some sort of way to trigger methods from the UI using jQuery event binding.  UI elements currently in development:

• Image/Sprite
• Button
• Textbox
• Text Input
• Check/radio box
• Slider
• Sound

I figure that those should cover most of the basics for creating a UI.  Of course, all of this will be extendable so you can create your own UI elements.  That way, UI's can be created and used within your games as well (not just for menu and option screens).  While doing all this, I modified the multi-resource loader a bit.  I've decided that I want to be able to use the loader without having to specify what is being loaded.  As such, I've created my own extensions for files which describe meta formats such as bitmap fonts and levels:

• Level: *.level (*.lvl)
• Bitmap Font: *.font (*.fnt)
• Image: *.image (*.img)
• Text: *.text (*.txt)
• Sprite: *.sprite (*.spr)
• User Interface: *.ui

Formats like JSON, XML, and sounds have their own extensions already so I don't need to specify those.  Each of the meta formats is still a JSON object that describes the actual contents to load, so you can call the loaders in the same way, but the multi-resource loader will simplify things for you.

When the multi-resource loader is used, you can either specify the type of loader to use or let it decide based on the extension of the file being loaded.  That way, you can just use the multi-resource loader as if it were all the loaders rolled into one without thinking about what it's actually loading.  It will use the correct loader for you.  The only difficulty comes when requesting cached resources from the multi loader by name.  The name doesn't indicate what it is, so I'll have to tackle that one at a later date.

HTML5 - What direction should we take?

I've been spending my time, lately, working on an Android port of The Render Engine to allow developers to create games in JavaScript, yet be able to take advantage of the platform in a native way.  While working on this approach, I've also been thinking about the best way to market the engine.  It seems that a lot of people have a preconceived notion in their heads about what, exactly, makes a game an "HTML5" game.

The feeling I get is that most people think "tile based game rendered in the DOM" when they imagine an HTML5 game.  However, it goes so much further than that... With the advent of the Canvas, people have been trying (myself included) to utilize it much the same way as developers did for games when EGA and then VGA showed up on the PC.  Rather than textual characters and lengthy text descriptions which made up a lot of games, the game players were wanting more so they invested time in learning how to access the graphics, directly.  This doesn't necessarily mean  that games written to use the DOM as their rendering "context" are bad.  But it can be limiting when trying to do advanced effects.

I don't want people to think that The Render Engine is solely a Canvas-based game engine.  It still supports the DOM, and it can do so with very little difference when writing a game for Canvas.  This allows developers, to make a choice when writing their game, to choose what their target audience is.  The way the Canvas works is similar to the way modern games are written for the PC, Xbox, Wii, or other gaming platform.  You render all of your content to a buffer which is then displayed to the player a frame at a time.

When writing a DOM-based game, however, it isn't the same.  The DOM is always live which means that a change to the DOM happens immediately.  There isn't the "clear and redraw" that occurs for "typical" games.  If the standard method of drawing a frame using CanvasContext were applied to HTMLDivContext, the engine would end up creating thousands of objects in the DOM.  Instead, the engine uses the DOM-created object as a mini-canvas (if you will) which is updated, as needed, to run animations and draw the world.  Ideally you wouldn't have to know about this and the engine would take care of all of that for you, but this isn't the case.

To truly make a game run in every browser, you have to design for the least common denominator.  In this case, that would be the DOM since IE (at least until IE9) doesn't support the Canvas.  While the engine tries to alleviate some of the work from the developer, not all of it will work as expected.  As such, a developer writing a game should probably make the decision from the start as to whether they want to use the DOM or the Canvas and stick to it.  Originally, I had intended The Render Engine to be able to "hot swap" rendering contexts but that paradigm just isn't feasible.  So I'm starting to wonder if I should do one of two things:

1. Focus on only creating the best engine for graphical contexts (canvas, web-gl, etc.)
2. Still support any context, but make the separation cleaner/clearer

Some feedback would be great because this would be a fundamental shift in the platform.  Neither one is a bad choice, but ideally the engine would be just an engine which allows the developer to make decisions for themselves rather than being limited by what the engine can do for them.  So, I ask you for some help in determining the future of The Render Engine... Vote in the poll and let us know your thoughts.

Further Android Development

So, I spent some time last night trying to figure out how to duplicate some of The Render Engine's functionality in Java.  Unfortunately, this is no simple task since JavaScript is such a flexible language and Java isn't.  While converting the Engine class to Java, I ran into my first problem.  It needed to deal with PooledObject, and that uses the "create" method to reduce garbage collection.

However, the "create" method is a class method on PooledObject and is actually inherited by objects which derive from it.  Overall, the complexity of the method could only be matched by either a factory class, or using reflection.  I don't want to have a super complex factory, and I also don't want reflection (which can be slow), so I got to thinking.

Maybe the objective here should be to determine what parts of the engine are slow and try to speed those up by re-making them in Java.  When I profiled the engine, on all the platforms, the largest amount of time is spent rendering objects.  So, the first tests I work through tonight will be creating an AndroidRenderContext which wraps the SurfaceView and Canvas Android objects.  Hopefully this will result in some speed ups.  If that proves out, I will see about re-making some of the components as Java implementations to see if that adds to the performance.

After speaking with a friend of mine today, he agrees that rendering is probably where 90% of the time is spent (and the numbers back that idea up).  He feels that things like math, input, collision, etc. are probably not taking up as much time as rendering.  Thus, if I can create a context which takes advantage of the Android, somewhat natively, I should be able to gain the performance I need without having to rewrite everything.

Android Development

Originally I had figured that it would be nice to have a simple way to package the engine and a game as an installable package for the Android platform.  However, after doing some testing and preliminary benchmarking, I've determined that running a game in an engine completely written in JavaScript isn't going to perform well on that platform.  So I've expanded the task to develop a launcher app into a complete port of The Render Engine to the Android platform as an app.

I know it might sound foolish, but most people aren't as interested in extending the engine as much as they are to use it.  Instead of running the engine in JavaScript, I plan to create a simple prototype to test implementing the engine in Java and using the WebView to expose the engine to a game.  The methods will all be the same, but the internals will run in native Java rather than JavaScript.  It's going to take some time, but the eventual plan is to be able to write a game for The Render Engine which runs in any web browser, plus the Android app, without changes.

What this would encompass is rewriting the engine in Java and creating interfaces to the parts of the engine using the Java to JavaScript bridge.  If this proves, a) plausible, and b) practical, then the result might be to have games that run quickly on Android and take advantage of the hardware more natively.  As such, I'm expecting most of the processing to speed up and for games to run without feeling sluggish.  Eventually, it would be possible to navigate to a game (or possibly distribute it through the Android Market Place) which runs the same on any desktop browser, and with the best possible performance on Android devices.

If all goes well with this, I might see if I can either develop (or commission) someone to do this for the Apple iPhone.  However, don't cross your fingers yet since that might violate their policies and allow people to develop apps for iPhone which aren't iTunes-approved.

Performance Notes

This morning I decided to add back the DOM rendering context into the "bounce the ball" demo.  How it got removed, I'm not sure, but it's back now in trunk.  You simply enable it by appending "context=dom" in the URL -- by default, it uses the canvas context.  After doing this, and relating to a blog post I read last night, I decided to get some performance numbers using this demo.  Now, realize that this is still very early in v2.0 development and their will be time to optimize, but I wanted to see where some browsers were at regarding engine performance.  The test was simple and not super scientific, but I ran the demo at 30fps (33ms available frame time) and added bouncing balls until the metrics topped out at an engine load of ~100%.  Below are the quick results:

Chrome 6.0.472.41
-----------------
DOM Context - 32 balls
Canvas      - 66

Firefox 3.6.8
-----------------
DOM Context - 17 balls
Canvas      - 17

Firefox 4.0b3
-----------------
DOM Context - 17 balls
Canvas      - 21

Computer specs:

Windows XPsp3

Intel Core2 Quad CPU (Q9300 @ 2.50GHz)

4GB

While this doesn't really tell much, it does show that Chrome's canvas rendering is pretty amazing.  66 physically animated balls bouncing around is pretty intense.  Especially considering that the best Firefox could muster was 21.  Like I said, not very scientific but still interesting none the less.  This points to the fact that The Render Engine's scene graph needs to be optimized a bit.

Understand that the physics demo really puts a strain on any computer.  While it performs quite well, considering how much it does, it's still performing a ton of calculations each and every frame.  It's currently set at 10 integrations per frame to calculate collisions, so I could lower that in favor of more speed for less accuracy.  There is a lot of tuning that could be done to eek out as much performance as possible, but it's still going to really test the capabilities of the engine.

Once all of the features are in the engine for v2.0, I plan on spending some time optimizing The Render Engine for each platform it supports (within reason, of course) to get the most performance out of it.  Some things will be major wins for all platforms, while others will be specific tweaks per browser.  By far, Chrome is still the best when it comes to performance.  I'm still amazed at how much horsepower the V8 engine can put out.

User Interfaces

Started working, in earnest, on the new user interface objects.  It will consist of a manager class to load UI's, an object which represents a loaded UI in game, and a set of base UI elements.  While creating the UI manager class, I decided to embark on creating a multi-resource loader.  This resource loader class can take the place of many resource loaders by lazy loading the ones you need, and then managing the instances for you.  When you need to access a single type of loader, you instead address the multi-resource loader and tell it what type you want to use.

This will allow for the UI manager to load resources from multiple sources automatically.  Your user interfaces will be able to have sprites, images, sounds, and more all in play.  A JSON object file is used to describe your UI, which makes creating user interfaces simpler and faster (I hope).  It's still pretty early in development, so it isn't ready to use even though it's in the source repository.  Everything is in flux, and subject to change as you probably already know.  As things come together, I'll either put together some screenshots or a simple demo to show off the new UI stuff.

End of Week Summary

Here it is, the end of the week, and what do I have to show?  Well, get yourself a chair and take a seat... You may have played around with The Render Engine when it was a child -- back in v1.0.  Additionally, you probably took a look at the "bounce the ball" demo and thought it looked pretty jittery and not very well simulated.  You'd be right on both counts.  Here, take a look:


http://renderengine.googlecode.com/svn/tags/v1.0/demos/wii_test/index.html

Well, v2.0 of The Render Engine is starting to grow up... Mind you, this wouldn't be possible without the fantastic JavaScript port of the Box2d physics library, but it's just another example of how flexible The Render Engine really is.  Everything is still in flux, at this point, and is subject to change in the future so don't count on this being the final implementation.  However, take a gander at the new "bounce the ball" demo:

http://renderengine.googlecode.com/svn/trunk/demos/wii_test/index.html


So, when anyone asks what I did this week, just point them at that there link and smile... I know I will.

Gettin' There

I have got physics working... almost.  The link below is a very rough version (early, early) of the "bounce the ball" demo using the Box2d physics code.  It runs surprisingly well, but as I said it is still quite rough.  There are a number of things to do to get this to where I want it to be for v2.0 of The Render Engine, but it's getting there.  This is untested on the Wii, so I wouldn't run out and try it on that platform.  However, it does run quite nicely on Chrome and Firefox.  Give it a go and send me some feedback.  I'm interested in hearing what people think of this particular update.

http://renderengine.googlecode.com/svn/trunk/demos/wii_test/index.html?metrics=true

Let's get Physical!

I've gotten the physics module loading, but it's still not working yet.   I don't get any dependency errors, so the files and classes are all loaded and configured.  The thing that I don't see is a physically animated object.  It's not a big deal -- the big deal is that it all loads without errors!  Now to try and get a physics object into a demo and play with it a bit...

In other news, I've been playing around with how to do UI's for The Render Engine.  I've got ideas about a couple of approaches.  The first would be to just use a div layered over the render context and have jQuery do the layout for me.  Another option is to create renderable components for the context itself.  I'm also fighting with how to represent UI objects for quick loading.  JSON just doesn't seem like a good fit, but it doesn't require a lot of parsing since it's inherent to most browsers.  I may still change it to be a textually descriptive format.

Physics and Billboards

Having a hell of a time integrating Box2d-JS into The Render Engine, right now.  I'm not saying it can't be done, but it's proving to be a major pain in the neck.  Soon I hope to get the bouncing ball demo re-written to use the new physics module, but for the time being everything to do with it is a little broken.  I wouldn't go trying to use the physics module just yet... I'll let you know when it's in there.

You may wonder how Billboards and Physics are related -- well, they aren't.  But one thing that was recently completed in v2.0 is the introduction of the BillboardComponent.  This component allows an object to render itself to a billboard (2d picture) and present that to the render context rather than re-rendering each and every frame.  Things like the vector text renderer benefit from this since it doesn't have to redraw each character's glyph for each frame.  It does it once, and from then on (or at least until it changes) it uses the billboard to draw itself.  This, my friends, is another win for speed!

Containers, rewritten

There have been a number of things, recently, that have come up during v2.0 development that could negatively affect performance in the future.  Additionally, these issues had started to manifest themselves in Chrome and Firefox 4.0.  So, to try and fix the main issue I rewrote the Container class to be a linked list, rather than a simple array.  It may seem counterproductive to change from the native Array object, especially since it has all those fantastic methods for filtering and such, but overall I've seen a slight improvement in performance.  It isn't huge, but it's there...

Doing so also allowed me to update the Iterator class so that it would not be negatively affected by changing the backing container.  As such, the iterators can be safely used without needing to worry about whether an object in the container has been destroyed.  This is due to the fact that iterators will intuitively detect which objects are included in the iteration and skip over those that should be excluded.

Unfortunately, as with any change to a core class such as Container has its drawbacks.  One such issue is that when a container is destroyed, it no longer destroys the objects it contains.  This isn't a big issue, as you can still call the cleanUp() method of Container.  But it will be a sticking point for some who've already started using The Render Engine for their game development and choose to work from the SVN trunk.