A Client/Server Tools Architecture


I mentioned Insomniac’s client/server tools architecture during my GDC talk earlier this year, and this topic has generated considerable interest. This article gives a very high level overview of the system as it is currently implemented.

All interactive productivity tools developed over the last two years at Insomniac Games have been built on this architecture. The basic idea is that the edited document is not kept in the memory space of the editor application itself, but rather each individual modification is transmitted to a server application, running on the same machine as the editor. The server maintains the authoritative document. Any number of documents may be open at any time. Any number of editors may communicate with the same local server application. The server provides various document related services.

There are several benefits from this architecture:

  • Crash Proofing
    The server application is comparatively simple, and matures early on. Editors are in constant development and may therefore be unstable. But since your work is managed by the server, up to the very last edit, it will survive a crash. Just restart the editor application and all your changes will still be there. Even the undo queue will survive.
  • Multiple Views
    LunaServer allows multiple editors to display/edit the same document. Changes that are made in one will immediately appear in the others.
  • “Free” Undo/Redo
    Undo/redo is handled by the server. All editors (clients) get undo/redo without a single line of code.
  • “Free” Load/Save/Revert
    All disk operations are handled by the server. All editors get this for free.
  • “Free” Perforce Integration
    Perforce integration is handled by the server. All editors get this for free.
  • Consistent Behavior
    Because undo/redo, file operations and Perforce integration are handled by the server, the user is presented with a very consistent interface

LunaServer

At the heart of the system is LunaServer. This is a server application running on the user’s own machine. No network connection is required. Although LunaServer uses network protocols for communication with the editors, and it is quite capable of running remotely, we have not found it necessary to do so. It was never intended to be used over a network. Every user machine runs its own independent LunaServer.

LunaServer implements a RESTful HTTP protocol.

Any editor that we write at Insomniac Games runs as a LunaServer client. We currently have implemented a world editor, an animation set editor, a visual script editor, an effect editor, a material editor, and several more. Any combination of editors and any number of instances of the same editor may be running at the same time. The same document may be opened in any number of editors. Most editors are written in JavaScript, one is written in C++, and one is written in Flash.

JSON

LunaServer acts as a JSON document manager. All assets are JSON documents. All editors “speak” JSON, even the C++ and Flash based editors. Even though the C++ application maintains a version of the document in binary form, this is considered a cache for 3D rendering and mouse interaction. The JSON document is the authoritative version, and it is always synchronized immediately with any changes made in the binary version.

Delta JSON

Changes are transmitted between LunaServer and the clients in what we call a “delta JSON” format.

Say, LunaServer contains a document representing an asset named DreamCar. The document might look like this:

{
  "model":"veyron.dae",
  "color":"black",
  "topSpeed":267
}

Now imagine that the user is using one of the editors to change the “color” property. The editor would update its local copy of the document, and transmit the following delta JSON:

{
  "color":"red"
}

In order to transmit this change, the delta JSON itself is wrapped into a “change document” like this:

{
  "targetDocument":"DreamCar",
  "deltaJSON": {
    "color":"red"
  }
}

When this is received by LunaServer, it knows to update the “color” property of the DreamCar document with the new value.
New properties can be added and existing properties can be removed. We have adopted the convention that a null value in a delta JSON object means “remove”. (As a consequence, LunaServer documents cannot contain null values, only delta JSON can)
For example, if an editor needs to send a change to LunaServer to add a new property “price”, and remove property “topSpeed”, it would make those changes locally, and send the following change document to LunaServer:

{
  "targetDocument":"DreamCar",
  "deltaJSON": {
    "topSpeed":null,
    "price":1.7e6
  }
}

After processing both change documents, DreamCar will read the same on the client side and in LunaServer:

{
  "model":"veyron.dae",
  "color":"red",
  "price":1.7e6
}

Property “model” was unchanged, “topSpeed” was removed, “color” was modified, and “price” was added.

Synchronizing Clients

As I mentioned, multiple editors may be running at the same time. They may even display and edit the same document. In addition to transmitting changes as delta JSON to LunaServer, each client must poll for document changes from LunaServer. If any editor changes a document, the delta JSON that is received by LunaServer is also transmitted to others, as they poll for changes.

(LunaServer implements a RESTful protocol, and therefore cannot push changes to the clients. So clients must poll)

MongoDB

LunaServer itself is backed by MongoDB. This was a natural choice. JavaScript, JSON, and MongoDB work very well together.

Perforce

Asset documents are shared between team members through Perforce, and Perforce maintains a document version history. LunaServer will interact with Perforce when an editor attempts to modify an asset document that is not currently writable. LunaServer manages load, save, revert, check-in, check-out, and other operations. It will prompt the user for file names and confirmations as needed. LunaServer does all of this automatically, relieving all editors from these responsibilities. All the editor code needs to do is send and receive asset changes in delta JSON format.

LunaTracker and files on disk

Perforce does not interact directly with LunaServer or MongoDB. Instead, it synchronizes files on the user’s hard drive. Every document in LunaServer’s database has a counterpart on disk. Synchronization of the database with changes on disk is the responsibility of LunaTracker. LunaTracker is a service that watches the asset document folders, and updates LunaServer when a new version of the file is detected.

Undo/Redo

Before LunaServer applies a delta JSON to its copy of the document, it will compute an inverse delta JSON.

If you recall, our DreamCar document started out looking like this:

{
  "model":"veyron.dae",
  "color":"black",
  "topSpeed":267
}

And our first change was to set the “color” to “red”:

{
  "targetDocument":"DreamCar",
  "deltaJSON": {
    "color":"red"
  }
}

LunaServer will store this change document in its redo queue, and transmit it to clients that poll for changes. It will also store the inverse change in the undo queue. The inverse change document will look like this:

{
  "targetDocument":"DreamCar",
  "deltaJSON": {
    "color":"black"
  }
}

When told to perform an undo operation, LunaServer will simply process the last document in the undo queue, and the car is black again.

Sessions

If multiple editors are open at once, their undo queues must kept separate from each other. When an editor is launched, LunaServer creates a unique session ID and a session document. Undo records are organized by session ID. The session document contains various bookkeeping data such as the list of documents currently opened by the editor instance, object selection, and other settings.

Multiple editor instances may make changes to the same document. Changes that are made in one instance are transmitted to all. But undo for any particular change is only available from the instance or session where it originated.

In some cases (actually only one), what appears as a single editor to the user is in fact implemented as two (or more) separate executables. This is the case with the world editor. The 3D view is a C++ application, and the supporting 2D user interface is a separate application written in JavaScript and HTML 5. In such a case, multiple editors may share the same session ID and session document. Their undo records and selections are shared and synchronized.

Session document changes are handled in the same manner as asset document changes. They are transmitted, received and processed in the same delta JSON format. The target will be the session document. This makes changes in selection, opening and closing of documents undoable operations.

Limitations

Although the architecture allows for multiple simultaneous editors, it is designed for a single user on a single machine. There is therefore no need to handle collisions, and there is no provision for it. Although multiple editors may have the same document open, the single user can operate only one at a time. If somehow simultaneous changes are made to the same property in the same document, in two different editor instances, some changes may be lost.

Conclusion

The Insomniac LunaServer architecture was started about two years ago. It seemed like a good idea at the time. I don’t think that anyone involved realized back then, how good the idea really was. The server side document storage, the simple synchronization between instances, the centralized undo/redo system and disk access have made our tools more robust, more flexible, easier to use, and their development much simpler.

13 comments » Write a comment

  1. Very interesting, I would like to know if you’re engine is a client to Luna server ( in that case, you’re engine is reacting to data change induced by your editors ) or if you bake you’re data from the server and then injected in the engine ( which in that case, you cannot edit in realtime data )…

    Thanks you very much…

    • It could be the case that the game/engine runs in two modes:

      * at development time, getting real-time data from the server, baking then and there then passing onto the game.
      * at test/release time, skipping the build and using pre-built data.

      But I would definitely be keen to hear more details about how the Insomniac guys do it.

      • Peter is exactly right. User editable data is “baked” in real time, as the user is altering it, and presented to the engine in freshly baked format. This works if the baking process takes milliseconds.
        Data that takes too long to bake is “faked”. For example the game may use a single collision mesh that combines all static geometry in a level. However during editing, it is not really static, re-baking takes several seconds, so we use dynamic collision meshes instead.
        In other words, when in editor mode, we sacrifice some offline optimization passes for the sake of keeping the UI responsive.

        Also, there is a lot of data that comes in from third party tools: bitmap data, model data, animation data, sounds. Our tools don’t offer any way to edit those assets. They are always baked and loaded by the engine directly.

    • In the current generation of our tools, the engine is NOT a LunaServer client. Rather, the editor executable is a client, and it pushes changes into the engine. It is this way mostly for historical reasons: the engine predates LunaServer and the editor.

      Making the engine, and indeed the game a LunaServer client is definitely on the radar — once FUSE is boxed and shipped.

      http://www.insomniacgames.com/games/fuse/#/home

  2. Thanks for sharing, this post blew my mind. Now i’m looking at web tech in many new ways.
    Did you consider using the WebSocket protocol instead of HTTP to avoid polling?

  3. Very nice technique! I’m thinking of ways that I can adopt this in my own pipeline. I did a hobby project recently that serialized user-created levels with JSON to a remote SQL server and used the same concept as your deltaJSON to minimize bandwidth. I didn’t really consider that it could be used for a generic solution to remote editing – quite novel! The architecture you’ve laid out here makes a lot of sense

  4. Pingback: Insomniac의 LunaServer – disk 접근과 undo/redo를 책임지는 로컬 서버 » ohyecloudy programming notes | ohyecloudy programming notes

  5. This is an excellent breakdown of how your architecture works, thanks!

    Before performing “undo”, do you first verify whether current document state matches the original delta using the redo queue before attempting to apply the inverse delta? and then if the state is mismatched, clear the entire undo history to avoid corruption?

    For example, given 2 editor clients [A] and [B]:

    1. Initial data for object { “a”: 1 }
    2. [A] submits delta { “a”: null, “b”: 2 } //–> { “b”: 2 }
    3. [B] submits delta { “b”: 3, “c”: 4 } //–> { “b”: 3, “c”: 4 }

    If [B] were to perform undo then the data would become { “b”: 2 }, as I would expect.
    If [A] were to perform undo then the data would become { “a”: 1, “c”: 4 }, which is corrupt right?

    Please excuse my ignorance, this is just something which spring into mind when I was reading through.

  6. Not quite. It seems that you assume that each client has its own undo queue. That is not the case. The undo queue is associated with the session, and if [A] and [B] both need write access to the document, they must connect to the same session. Therefore they both share the same undo queue. It doesn’t matter whether [A] or [B] requests the undo. The most recent change is always undone first, regardless of which client it came from.

    • Thanks for clarifying this point, yeah I was assuming that each client had its own session, but it certainly makes sense to share the session for this.

Leave a Reply