I'm some days away from releasing a major upgrade and one of the features that is now included is a self-updating mechanism. When I say self-updating I mean that in the launcher screen (a small window that pops up before the main game launches that let's you set visual preferences and similar) there is a nice friendly button named "Update". Press it, wait for a few seconds while the app downloads new data, and presto! you are running the latest version. If you update your game regularly this is a major benefit to your players. My main goals were (in order of importance): 1. Use freely available components and only write "application" code 2. Can update running executable 3. Portable (Win32/OSX/Linux) 4. Optimized network transport layer 5. Easy to manage releases and server The system I'll describe fits all goals. In the end I wrote about 300 lines of code in the game and managed to avoid all server coding. Of that about 100 lines is for swapping executables, quite straightforward but tedious. The major design features are: - The library used for transport and version management is Subversion, the open source version control system created to replace CVS. The application uses the svn_client.h interface and just 2 functions (svn_revert and svn_update). The library is compiled with support for only standard WebDAV without encryption as transport. The server runs a standard Apache2/WebDAV/SVN server on a Debian box. - A lot of things result from using SVN. There are excellent tools for managing a repository and it's portable across all platforms. SVN is also free (as in speech and beer). The best part is that SVN is optimized for minimizing network load. All the files are stored separately and when you update only changes are propagated over the wire. You also get all future optimizations for free. - To facilitate updating of the running program all binary files (*.exe and *.dll) are placed in a "binary_win32" folder (under version control) with local copies in the installation directory. The update sequence is as follows ("\" is the installation root): 1. \simulation.exe updates the "\binary_win32" folder 2. \simulation.exe launches "\binary_win32\simulation.exe -hotcopy TASKID" and quits 3. \binary_win32\simulation.exe waits for TASKID ("\simulation.exe") to quit 4. \binary_win32\simulation.exe copies all *.exe and *.dll files from \binary_win32 to \ 5. \binary_win32\simulation.exe launches "\simulation.exe -postupdate TASKID" and quits 6. \simulation.exe waits for TASKID ("\binary_win32\simulation.exe") to quit - Just packing up a checked out copy would be wasteful as SVN stores pristine copies of all files in the .svn directory ("text-base"). To get around this the installer is built only using the contents of the .svn directories without the "normal" files. After this structure is unpacked on the players machine (resulting in en empty directory tree with a lot of hidden .svn directories) the installer runs svn_revert on the repository directories thus "reverting" the whole directory tree to the pristine copies stored in the .svn directories. All in all composing the installer from a live working copy adds just a few 10's of KB. I'm most happy about the fact that everything is based on proven tools that I use daily anyway and that the amount of code to write was minimal. I hope you find this technique useful. Take care!
That is a very slick solution, I think it's great how you've leveraged SVN. (I need to upgrade from CVS soon.) Very cool stuff!
nice one! That exact project is one my To Do List I hadn't thought of using Subversion, that's a great idea, I'll have to squirell it away for later.
I'm not familiar with subversion, but I wondered if you could address a few things that popped into my head while reading you post. It sounds like a great idea, but.... - aren't you worried about exposing your scm to the world, hacking etc.? - "only changes are propagated over the wire" does this mean just changed files or the changed parts of files? - you didn't mention it in the steps but I assume it also updates non-exe/dll files at some point? - it sounds like the installer is a small file that then does the work of pulling down the rest of the files (is this correct?) or put another way... - when you say "normal" files, do you mean all the other files except the ones needed to use you install method? I hope my question makes sense, thanks.
Good questions, thanks. I needed a bit clearing up .. (I took the liberty to number your questions). 1. WebDAV runs essentially like normal HTTP requests through Apache2. The repository on the public server contains only release files (ie. final compiled assets, meshes, images etc.) and is a separate setup from our development server. The area is password protected using mod_auth_mysql so all users have a distinct account. The http: server has read only access to the repository and I do my updates through secure server with a separate authentication setup (passwd files). This setup keeps honest people honest .. 2. Only delta information is propagated, ie. only the actual changes in the files are transfered. AFAIK the delta algorithm is binary (ie. non-textual) and it uses the fact that there is a pristine copy of the file in the client working space. The whole design of Subversion has been to optimize for minimum bandwidth consumption (it's a distributed version control system). 3. Yes. I didn't mention it specifically as it isn't as problematic as updating the running exe. All the game assets live in a folder named "assets" which is updated at the same time as the binary folder. The launcher is self-contained, ie. it has all the assets compiled in so things are needed from the disc only when the actual game starts. 4. and 5. No. The installer contains all the files so after downloading it you can install and play without being connected to the net except if you want to use the updating feature. The problem was how to turn these raw files into a valid subversion working copy. What the installer does is to package up a working copy residing on my disk that is checked out from the correct URL. You could do this directly and it would work nicely enough. However, to speed up certain operations SVN stores a copy of the file in the .svn directory. So for every file "./foo" you also have a file roughly named "./.svn/text-base/foo-base" that is an exact replica of "foo". (The "normal" file in this case is ./foo). This doubles the size of the installer. So instead of packagin up ./foo the installer packages ./.svn/text-base/foo-base and then uses the SVN command svn_revert that recretes ./foo based on the version in text-base. This way your installer takes the same space as it would normally take, but you end up with a properly created working copy on the players machine (that can then be used for updating). As with all SVN working copies this ends up taking 2x the required space on the target disk. (The rationale being that space is cheaper than bandwidth), Just ask if everything isn't clear..
Great thanks Mistaril for the extra detail. I think I understand now. The only lingering question is about the 2x space requirements. Is the increase in update speed worth it? Or is just a matter of how subversion works?
This sounds good! You've inspired me to do something similar with subversion. Self updating is much easier for testing than distributing new installers to your testers. Anything to make testing quicker is great! What happens if someone cancels the update mid process(kills program, or computer shuts down)? Ie, does it leave the old stuff there until all of the new stuff is finished updating? Will your game still work? How much disk space does the subversion library add to your installer?
The 2x space requirement is just how Subversion works. It's really not very useful for the update process but there isn't really a way around it. Good question about the update.. I don't know yet. I know the commit process is atomic but I suspect that the update is non-atomic. Ie. you need to flag the start of an update and then force it to completion before starting the program. The update is recoverable (ie. you can stop it anytime through) and with this setup the Launcher can start even if the assets are out of sync. I'll test it and write back when I know better.
It would be great if more applications could be upgraded without having to restart. For example, Igor Engraver, will download any required patches and upgrade itself without interrupting any work in progress. -Luke
An easy way of doing this is to start your game with a stub.exe. You can call this with the name of the final exe and whatever info it needs to check for the update. Once it's finished (or can't get a connection) it'll just run the game as normal.