I know of at least one (proprietary) open world game engine that effectively stores its world contents as a database in the development version. These systems are made to support efficient insertions/updates/removals, without moving large swaths of the data around for each operation.

Loading a chunk of the world is then just a database query vaguely like:

SELECT * FROM worldObjects WHERE x >= xMin, x < xMax, y >= yMin, y < yMax

The trouble is, this is quite slow to load from when scaled up to millions of objects, so this was only done in dev mode (and all our dev machines had big fast SSDs to hide the inefficiency).

For players, we had a part of the build process that would do these slow database queries and bake their results out to an optimized flat file for each chunk of the world at a given level of detail. That way, as you approach a sector of the map in the released game, the engine can quickly find the file for that sector and load it in one linear pass.

So that shows two strategies you can use:

• If this is just for development, or if the set of dynamic changes is small / short lived, consider a database system that supports random insertions and deletions out of the box.

• For linearly-loaded data, don't store it all in one single file buffer, but divide it into separate files at regular spatial intervals. That way you can still randomly seek to a given zone by its file name, and if any chunk file needs to change in size, you have to modify only that one file, not the entirely of the world"s map buffer "after" that point.

Elaborating on this, you can use the same strategies we use to support dynamically growing collections in RAM too:

• The location of something in the data's byte order does not need to strictly map to its location in the world. To find stuff by location, you can construct a spatial lookup acceleration structure, like a quadtree, during load. That way, adding something near the "start" position of the map doesn't mean you have to move everything else's bytes down: you can add it at the end, and update your lookup structure to point to it.

• Store the volatile parts separately from the long-lived parts. If you have a core terrain that doesn't change (often), put that part of the file first. Then adding/removing dynamic stuff only affects stuff close to the end of the file, not the whole layout.

• Keep a little wiggle room: when you need to expand the container to hold a new item, don't expand it by just one slot, round up to a multiple. When you remove an item, you can leave a gap to fill later, rather than immediately vacuum-sealing the container to its minimal size. That way the next several adds are free, and you amortize the cost of resizing the container over many operations.

  This does waste some disc space, but your OS is likely allocating it in blocks of ~4kiB at a time anyway. You can fit a lot of game entities in 4kiB if you take that extra space all at once.

I know of at least one (proprietary) open world game engine that effectively stores its world contents as a database in the development version. These systems are made to support efficient insertions/updates/removals, without moving large swaths of the data around for each operation.

Loading a chunk of the world is then just a database query vaguely like:

SELECT * FROM worldObjects WHERE x >= xMin, x < xMax, y >= yMin, y < yMax

The trouble is, this is quite slow to load from when scaled up to millions of objects, so this was only done in dev mode (and all our dev machines had big fast SSDs to hide the inefficiency).

For players, we had a part of the build process that would do these slow database queries and bake their results out to an optimized flat file for each chunk of the world at a given level of detail. That way, as you approach a sector of the map in the released game, the engine can quickly find the file for that sector and load it in one linear pass.

So that shows two strategies you can use:

• If this is just for development, or if the set of dynamic changes is small / short lived, consider a database system that supports random insertions and deletions out of the box.

• For linearly-loaded data, don't store it all in one single file buffer, but divide it into separate files at regular spatial intervals. That way you can still randomly seek to a given zone by its file name, and if any chunk file needs to change in size, you have to modify only that one file, not the entirely of the world's map buffer "after" that point.

Elaborating on this, you can use the same strategies we use to support dynamically growing collections in RAM too:

• The location of something in the data's byte order does not need to strictly map to its location in the world. To find stuff by location, you can construct a spatial lookup acceleration structure, like a quadtree, during load. That way, adding something near the "start" position of the map doesn't mean you have to move everything else's bytes down: you can add it at the end, and update your lookup structure to point to it.

• Store the volatile parts separately from the long-lived parts. If you have a core terrain that doesn't change (often), put that part of the file first. Then adding/removing dynamic stuff only affects stuff close to the end of the file, not the whole layout.

• Keep a little wiggle room: when you need to expand the container to hold a new item, don't expand it by just one slot, round up to a multiple. When you remove an item, you can leave a gap to fill later, rather than immediately vacuum-sealing the container to its minimal size. That way the next several adds are free, and you amortize the cost of resizing the container over many operations.

  This does waste some disc space, but your file system is likely allocating it in blocks of ~4kiB at a time anyway. You can fit a lot of game entities in 4kiB if you take that extra space all at once.