How to manage resources?

[ZeHa]

Hello,

at the moment, I'm thinking about programming a little Resource Manager for my engine, but not only for the stuff on the inside, like loading the sprites into memory and things like that, but also highly for development.

For the games, it's a good thing to store graphics and sounds numbered, like A053.bmp or something like that. But although I try to find a clever way of numbering, it gets really cryptic after a while. So I thought about writing a little app where I can manage all my resources for a game. I imagine a comfortable UI with lists and trees etc, where I can put sprites into categories and have clearly readable names on my screen. But on the backend, they will be saved with their cryptic filenames. So I won't touch the filesystem anymore, but work with my app, just like some database client.

Of course this could be further expanded, all resources could be packed into one big file and perhaps also compressed etc, and of course the manager app could also include viewers and even simple sprite editors etc.

But before I'll start something like that, I'd like to hear what YOUR ways of "being the master of your chaos" are ;) and if you perhaps also found some similar solution, or even something completely different?

[luggage]

We just name our files so they can be identified - mouse.png for example. Then when the game is done use molebox (http://www.molebox.com/)to pack the game up.

[soniCron]

For the games, it's a good thing to store graphics and sounds numbered, like A053.bmp or something like that. You know, we've broken the 8.3 filename barrier ages ago... :p

In all seriousness, I keep XML files with a handy processing format that allows me to load a series of images in a variety of formats. For example, to load all images "Sprite0001.png" to "Sprite0050.png" I just use the following:

<sprite name="Mysprite">
<frameset filemask="Sprite####.png" />
</sprite>

For a series of different named images:

<sprite name="Mysprite">
<frame file="Sprite1.png" />
<frame file="Sprite1b.png" />
<frame file="Othersprite1.png" />
</sprite>

If I have a single image with all frames laid out horizontally, I do the following:

<sprite name="Mysprite">
<frame file="Sprite.png" width="64" />
</sprite>

Then, in the engine, I just do LoadSprite("Mysprite") and it gets the sprite. If the sprite hasn't been defined, it tries loading it as a file, trying all acceptable extensions. (PNG, JPG, TGA, etc.) If it does that, it tries to auto-detect if it's a single image in a series by looking at trailing numbers and makes a best-guess at loading it.

Of course, I could just default to loading images without defining them explicitly, but there are additional options I can use to further specify playback. Like:

<sprite name="Mysprite" rate="30" loop="-1">

would play the sprite at 30fps and loop indefinitely. It can get far more complex than that, because you can load not only images or frames, but other sprite definitions, so you can mix and match various sprite sequences. I can even apply different "tracks" of sprite animation and switch between them. (Running, walking, idle, etc., for example.)

The rest of the engine is split into a hierarchical tree like this, from game states to the scenegraph to audio playback. Finally, I can load either directly from disk or from a zipped datafile. (Or series of data files, or a combination of all three.)

This is a little overkill for most people's needs, but I like the flexibility and laziness allowed. I like to spend time making games, not fighting with semantics! ;)

[Sybixsus]

Finally, I can load either directly from disk or from a zipped datafile. (Or series of data files, or a combination of all three.)

Ah, I'm not the only one who does this then. All my file accesss is abstracted so that the engine automatically looks to all archives which have been "attached" to the current project to see whether the indices ( a series of binary lists for speed of access ) includes the file or not. If so, reading/writing takes place within the archive, and if not, it looks for it on disk instead.

[jcottier]

> A053.bmp
????

Why do you want to use such a complex system to name your texture?

You can use pack file and still have human friendly names for your textures.

JC

[ZeHa]

Ah, I'm not the only one who does this then. All my file accesss is abstracted so that the engine automatically looks to all archives which have been "attached" to the current project to see whether the indices ( a series of binary lists for speed of access ) includes the file or not. If so, reading/writing takes place within the archive, and if not, it looks for it on disk instead.

Yes, that's indeed a good technique. I planned to use that also for properties-files, so that I can develop with simple "fullscreen = yes" properties, but also use binary files later for release (of course, the fullscreen example is a bad one because it would be actually a good thing if the player could change the basic settings directly via some .ini file - but there's more "sensitive" data for the game, which should be protected).

@ jcottier:

The reason for me to do that was because when we developed our first game, we had "real" names (at least some abbreviations using meaningful letters), but then I had to load every sprite manually, and since single tiles were stored as single files, it was a lot of code (actually it was okay, but our new games will have more tiles than the first one).
So my intention was to make the loading process easier by using some simple cascaded for-loops that load sprites by some hexadecimal pattern. That way, I'd just add some tiles to my gfx-folder without touching my code at all. It's perfect for the INTERNAL resource manager, as you can see, but of course it's not perfect for me, the "external resource manager" ;)
But of course I don't mind some good proposals of what I could do instead - that's why I started this thread, actually :D

[James C. Smith]

Finally, I can load either directly from disk or from a zipped datafile. (Or series of data files, or a combination of all three.)

Our framework is very similar. We use logically named files in a data folder and then ZIP the data folder into a data.dat file to ship the game. (not numbers or 8,3 names). The file system abstraction layer makes it transparent to load data from individual files in the data folder (during development), or file in the main data.dat archive (that we ship), or in add-on archives (that are used for patches and localizations), or all 3. We also support storing the archive inside the EXE but we haven't used that since Swarm.

The file system abstraction layer handles all that file IO and archive parsing. But sitting on top of that is a resource manager that keeps track of which resources (images, sounds fonts) have been loaded and where they are in RAM. If you request the “GameOver.OGG” file from the resource manager it will return the CSound object containing the waveform. It will check to see if it has already been loaded and if not it will allocate the object, perform the IO, and process the data into a usable form. You can ask it what has been loaded so far, tell it to unload things not used recently, or force it to load things.

Out resource manager also known how to check for dependencies between files. It can tell you which levels contain which character and what sounds and images those characters use so that you can pre-load only the needed resources at the start of a level. So the resource manager lets you just ask for any data at any time, and it will just load it on the fly if it needs to, but in practice that never happens because you usually end up pre-loading the things you need before they are needed.

But the system for manage what to load when, know what is currently loaded, and know how to load things is not part of file system. And the system for storing resources inside of archives is not part of the file system abstraction. The file system abstraction and the resource manager are two very separate systems. (Of course one is dependent on the other but not the other way around)

[TunaBreeze]

I'm developing a 3D engine and I'm currently using INI files to track the resources for game objects. An example would be...

[factory]
normal=data\whatever.ms3d
damaged=data\kjdghkjdsahg.ms3d
destroyed=data\kdfjhjdsadjk.ms3d

[objectname]
objectstate1=data\asdf.ms3d
objectstate2=data\asdf2.ms3d

I parse the file in to my list of game objects. The text in between the brackets is the name of the object, and the whatever=whatever are the different states for that object. I parse using the = sign, the stuff on the left is the name of the state, and the stuff on the right is the milkshape 3d model for that state. You can have multiple resources such as your model, audio, or whatever is necessary by just adding a space then the file name for that resource. Example...

[factory]
damaged=data\whatever.ms3d data\explosionsound.mp3

[objectname]
objectstate=data\modelfile.ms3d data\soundfile.mp3 data\2diconfile.jpg

This system has worked pretty well for me so far. When I eventually get my GUI system running I will set up a menu so I can edit the file graphically and be able to see and hear the different contents of the files for the different states of the objects. That way when I bring more people on to the team they can easily edit the game objects without having to reference various folders or worry about the internal format I'm using for the INI file.

[Emmanuel]

We also use the same technique as James C. Smith and Daniel, where a data folder and its subfolders are archived into pakfiles automatically when making a release build. The I/O layer looks for assets into the pakfiles that the game registered, and if it doesn't find them in there, looks for the same assets in independent files. We use explicit names; my number one goal when I start a new game is to have a working build where the artists can update the datafiles and see the result live in the game right away. The build process automatically splits data between common and distribution channel-specific files so that we can ship all builds to all portals and offline distributors by patching the default base build (BigFish's) with a single datafile.

In funpause days, I've contributed the code to ptk (the KArchive classes), so ptk will automatically load any asset from archive files, as well as let you use the classes to load your own data. We still use the same concept, with a different fileformat, for Big Fish games. It's very convenient and I don't see the need to over-architect beyond that.

Best regards,
Emmanuel

[ZeHa]

But since you use "real" names (I imagine something like redplayer_walking_left.png or so), how do you manage it on the inside? You can't build a loop to load all the graphics, so do you have a separate "ingredients list" where every graphic file to be loaded is named on?

[soniCron]

But since you use "real" names (I imagine something like redplayer_walking_left.png or so), how do you manage it on the inside? You can't build a loop to load all the graphics... Sure you could:

Color
red
green
blue
yellow

Action
walking
idle
running

Direction
down
up
left

Then just concatenate them: Color + Action + Direction + ".png" Although, on second thought, I suppose that's the ingredients list you were talking about... ;)

[Emmanuel]

We group all frames in a single texture; save it as a film strip, or use one of the bazillion tools (http://www.phelios.net/supersprites.html) available. If you're rendering using '2d in 3d', it's also critical to do that in order to minimize state changes. The framework can do the grouping for you, but we just do it in the asset files directly.

Otherwise, it's indeed not very hard to use naming conventions (so that artists can edit your assets easily) and use that in a loop.

Best regards,
Emmanuel

[chanon]

Well, here's how we do it.

First, all our resources use human readable names such as "greenMonster.jpg".

Then for each game we create a folder structure so that the resources that are used together are grouped into folders.

ex.

\res\level1\
would include the resources that only level1 uses.

When we are beginning level1 we can do something like
VQAnimDataMgr::instance()->preloadPackage("/res/level1/");

This preloads all resources under the \res\level1\ folder. (It iterates the folder to see what image resource files are under it.)

Then in the game when we create a sprite it is just
VQAnimSprite* gameBg = new
VQAnimSprite("greenMonster");

The VQAnimSprite constructor calls
VQAnimDataMgr::instance()->acquire("greenMonster") inside.

The game coder doesn't have to know/remember/care where "greenMonster" is located he just needs to know the name. Also this allows us to re-arrange the folders without having to change anything in the code except the preloading lines.

[TunaBreeze]

I'm not sure what kind of engine the original poster intended to make but one thing your resource loading should contain is a scalability. For instance my engine is a 3D engine. The different game objects can have multiple states as I have demonstrated in my first post and each state for that object has it's own model, audio, and possibly more content later down the road. Many of the examples give by other posters don't explicitly state a method to create different states for the game objects, they only allow you to specify one texture, sprite, or whatever it is they're using.

Loading from a folder is fine so long as each type of game object has its own folder otherwise you will have to hard code a method for letting your engine know what geometry, texture, audio, what have you goes to what game object.

With my method of object creating I have the object type list which looks like this...

[objecttype1]
walking=whatever1.ms3d whatever1.jpg whatever1.mp3
running=whatever2.ms3d whatever2.jpg whatever2.mp3

[objecttype2]
walking=whatever3.ms3d whatever3.jpg whatever3.mp3
running=whatever4.ms3d whatever4.jpg whatever4.mp3

Then my engine accesses a different file that has a list of object instances. The instance contains the name, location, and state of the object, then when the game starts both files are loaded and a scene can be created.

I don't want to say the other methods listed are bad, maybe they just didn't explain enough about their method, but in my game the system needs to have more flexibility and scalability, maybe yours doesn't, and maybe their's doesn't.

[ZeHa]

My engine will be a tile-based 2D-games engine.

Well I think my numbering comes from the fact that my tile-based level files contain maps that are stored in 16-bit-numbers for each tile.

Of course, for "dude"-like objects, it's not a big problem to use real names, but if the whole map was basically "tree tree water bigtree woodenbridge..." etc it would be a little bit complicated ;) well it would be possible to map every name to a number somwhere, but this wasn't my first thought. So instead I used a system of hex numbers where the first digit would be the class (like a tree, or a piece of ground) and the other three digits could be the number. That way I can load everything with two nested for-loops and without mapping anything, but of course it's not the most comfortable way.

I'll think about it a little bit and remember your approaches, perhaps I'll be able to make something more usable. But of course I still like my initial idea of making a little app that controls everything related to resources, that way I could "check in" graphics, sound, and scripting files, perhaps edit them directly, and have it readable and clear on a single screen, while the app will manage how it will look like in the filesystem. Perhaps then I'll also include a "pack everything together into one file" function for release.