Best practices for periodically saving game state to disk

Question

I'm working on an MMO. All of the player and environment data lives on a server and is kept in memory. There's a "world" object which keeps track of all of the maps, characters, etc. and their relations to each other.

To avoid data loss in case of a crash, I've been periodically serializing the world to disk. The trouble is, this object can be quite large, so when the server starts writing, there's noticeable in-game slowdown for a few seconds, which I'd like to avoid.

Any pointers on how to go about this in a more efficient way?

What is the data structure of the world object and where is the larger chunks of data? — AbstractChaos
– AbstractChaos, Commented Jul 9, 2012 at 16:02

Kylotan · Accepted Answer · 2012-07-09 20:20:12Z

(1) Only save the really important bits, (2) only save them when they have changed since the last time they were saved, (3) save them individually, and (4) save them via an asynchronous system.

Much of the world is likely to be static - that data doesn't need to be serialised at all, as it probably already exists on disk in some form. At the other end of the scale, several variables are so transient that they don't need to be serialised either. An example would be hit points - it's hard to imagine a situation where your business is harmed if a server restarts and resets hit points to full, so don't save them.
Set a flag on each part of the data, set it to True when you perform an update or insert operation on that part of the data, and only save the data if that flag is True. Otherwise, leave the previously written version in place. (This requires that you're not just writing the whole thing as one lump - see the next section.)
You should separate the specific items that need saving and be able to save them individually. In some games, this is just players. Other games save items on the ground; some may save important NPCs too. Very few are saving absolutely everything. But when you can save things individually, you can easily save the world a bit at a time and make saves quicker.
If you use a database for your persistent data, these often provide an asynchronous interface which will return immediately while saving in the background. (But beware of error conditions that could potentially leave you without a valid save point.) Alternatively you can perform your writes in a background thread, providing you were able to make an in-memory copy of the relevant state first.

Nathan Reed · Accepted Answer · 2012-07-09 17:59:50Z

It wasn't clear to me from your question whether you're already doing this or not, but: do the save in a separate thread (or using asynchronous I/O, available in some OSes). Then your game code can continue to run while the save happens in the background. Of course, you'll have to snapshot the game state in memory so you're not updating state while you're in the middle of saving it. (Hopefully the state is not so large that making a copy of it in memory causes noticable lag!) Also, whether using threads or async I/O, there should be some sort of priority knob you can turn to prevent the background work from taking too much time away from the main game thread.

Ivgeni Segal · Accepted Answer · 2014-12-01 02:26:34Z

If your MMO server is running on a Linux/Unix OS, you might be able to take advantage of the inherent copy-on-write properties of vfork(). This trick is used by some database software such as Redis to make consistent point-in-time snapshots without stopping the server.

When you call vfork() the OS makes a clone of your process, however it does not copy the memory of that process. The clone will still share the same memory pages, as long as they are equal. The OS will create copies of pages as needed when they are modified by either of the process. Page sizes are usually 4kb in linux. If you think about it, this is your delta compression, except you don't have to write any code for it. The OS already does the dirty work for you (utilizing the hardware MMU, etc).

Once you call vfork(), your parent process can continue to operate as normal, and your child process, which is a clone of the game universe at the time of vfork(), should no longer process the game and instead take all its time writing out all of the object states. It can possibly use lower priority, throttle down the IO, etc. Once it finishes, it should exit.

The beauty of this is that all your game state is consistent to the exact point of the fork 'snapshot'. You are essentially making the OS do all the hard work for you of keeping dirty bits etc. You will not get into a situation where some of your objects got modified after you had begun saving the game, while others didn't.

This trick was used in the original Ultima Online, as some of the devs mentioned it in this discussion.

tomsseisums · Accepted Answer · 2012-07-10 06:25:08Z

Base your game around Events, and log every single tiny Event. Whenever you need to load data or something, just iterate over the related Events and woila everything's back.

For Events, you can use either database, text files or whatever storage, it's basically a logging mechanism.

Then you can cache current Event state before moving Events to backup for like 1-3 times a day on a separate server or when the load time is less than X%. Caching works like restore point - if the player had 12313 Events in last 3 hours, that's a lot of loading time the next time he comes online, so, when it's cached, it would read the cached state from memory and log new Events, that would later also be re-cached and so on and so forth.

Only thing here is, you need a nice set of algorithms or something for Event state cache.

I'm in a hurry at the moment, will explain in detail later.

Stack Exchange Network

Best practices for periodically saving game state to disk

4 Answers 4

You must log in to answer this question.

Hot Network Questions

Best practices for periodically saving game state to disk

4 Answers 4

You must log in to answer this question.

Related

Hot Network Questions