9

I'm trying to initialize an array in three dimension to load a voxel world.

The total size of the map should be (2048/1024/2048). I tried to initialize an jagged array of "int" but I throw a memory exception. What is the size limit? Size of my table: 2048 * 1024 * 2048 = 4'191'893'824

Anyone know there a way around this problem?

// System.OutOfMemoryException here !
int[][][] matrice = CreateJaggedArray<int[][][]>(2048,1024,2048);
// if i try normal Initialization I also throws the exception
int[, ,] matrice = new int[2048,1024,2048];

    static T CreateJaggedArray<T>(params int[] lengths)
    {
        return (T)InitializeJaggedArray(typeof(T).GetElementType(), 0, lengths);
    }

    static object InitializeJaggedArray(Type type, int index, int[] lengths)
    {
        Array array = Array.CreateInstance(type, lengths[index]);
        Type elementType = type.GetElementType();

        if (elementType != null)
        {
            for (int i = 0; i < lengths[index]; i++)
            {
                array.SetValue(
                    InitializeJaggedArray(elementType, index + 1, lengths), i);
            }
        }

        return array;
    }
Improve this question
13
  • 6
    Your CreateJaggedArray method isn't creating a jagged array, it's creating a multi-dimensional array which is very specifically not a jagged array. Commented Apr 23, 2013 at 21:07
  • 2
    @MehdiBugnard Just look at the type of the variable int[, ,] matrice. If it were a jagged array it would need to be int[][][]. Commented Apr 23, 2013 at 21:14
  • 4
    If the world is mostly "empty space" then you can do way, way better than a big array. The usual technique is to use an octree: en.wikipedia.org/wiki/Octree Commented Apr 23, 2013 at 21:38
  • 2
    @Filip: Indeed. Moreover: since an int is four bytes, representing all 4 billion voxels naively will consume 16 billion bytes of virtual address space. On a 32 bit process there is only 2 GB of virtual address space available period. (There is of course essentially unlimited virtual memory; remember, memory availability is bounded by disk size not by physical memory size. But there is very limited virtual address space on a 32 bit processor.) An octree can compress this down to a very managable size if there is lots of redundancy in the structure. Commented Apr 23, 2013 at 21:53
  • 2
    @MehdiBugnard: I recommend that you not attempt to load the data into a big array in the first place. Just load it directly into your octree. Commented Apr 23, 2013 at 21:54

4 Answers 4

Reset to default
4

The maximum size of a single object in C# is 2GB. Since you are creating a multi-dimensional array rather than a jagged array (despite the name of your method) it is a single object that needs to contain all of those items, not several. If you actually used a jagged array then you wouldn't have a single item with all of that data (even though the total memory footprint would be a tad larger, not smaller, it's just spread out more).

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Thank you very much. Do you know an effective method to initialize a "jaggedArray" in 3D with one item per case ?
@MehdiBugnard Just stick with good ol' for loops; one per dimension...Why complicate things.
NET 4.5 has an option in x64 to explicitly allow objects to be larger than 2gb by setting gcAllowVeryLargeObjects in the app.config.
I think the OP's problem actually boils down to storing such a structure in the memory (with reasonable performance). That's what I'd like to know, too.
4

Thank you so much to all the staff who tried to help me in understanding and solving my problem.

I tried several solution to be able to load a lot of data and stored in a table. After two days, here are my tests and finally the solution which can store 4'191'893'824 entry into one array

I add my final solution, hoping someone could help

the goal

I recall the goal: Initialize an integer array [2048/1024/2048] for storing 4'191'893'824 data

Test 1: with JaggedArray method (failure)

system out of memory exception thrown

            /* ******************** */
            /* Jagged Array method  */
            /* ******************** */
            
            // allocate the first dimension;
            bigData = new int[2048][][];
            for (int x = 0; x < 2048; x++)
            {
                // allocate the second dimension;
                bigData[x] = new int[1024][];
                for (int y = 0; y < 1024; y++)
                {
                    // the last dimension allocation
                    bigData[x][y] = new int[2048];
                }
            }

Test 2: with List method (failure)

system out of memory exception thrown (divide the big array into several small array .. Does not work because "List <>" allows a maximum of "2GB" Ram allocution like a simple array unfortunately.)

        /* ******************** */
        /* List method          */
        /* ******************** */
        
        List<int[,,]> bigData = new List<int[,,]>(512);
        for (int a = 0; a < 512; a++)
        {
            bigData.Add(new int[256, 128, 256]);
        }

Test 3: with MemoryMappedFile (Solution)

I finally finally found the solution! Use the class "Memory Mapped File" contains the contents of a file in virtual memory.

MemoryMappedFile MSDN Use with custom class that I found on codeproject here. The initialization is long but it works well!

        /* ************************ */
        /* MemoryMappedFile method  */
        /* ************************ */

        string path = AppDomain.CurrentDomain.BaseDirectory;            
        var myList = new GenericMemoryMappedArray<int>(2048L*1024L*2048L, path); 
        using (myList)
        {
            myList.AutoGrow = false;

            /*
            for (int a = 0; a < (2048L * 1024L * 2048L); a++)
            {
                myList[a] = a;
            }
            */

            myList[12456] = 8;
            myList[1939848234] = 1;
            // etc...
        }
Improve this answer

Comments

3

From the MSDN documentation on Arrays (emphasis added)

By default, the maximum size of an Array is 2 gigabytes (GB). In a 64-bit environment, you can avoid the size restriction by setting the enabled attribute of the gcAllowVeryLargeObjects configuration element to true in the run-time environment. However, the array will still be limited to a total of 4 billion elements, and to a maximum index of 0X7FEFFFFF in any given dimension (0X7FFFFFC7 for byte arrays and arrays of single-byte structures).

So despite the above answers, even if you set the flag to allow a larger object size, the array is still limited to the 32bit limit of the number of elements.

EDIT: You'll likely have to redesign to eliminate the need for a multidimensional array as you're currently using it (as others have suggested, there are a few ways to do this between using actual jagged arrays, or some other collection of dimensions). Given the scale of the number of elements, it may be best to use a design that dynamically allocates objects/memory as used instead of arrays that have to pre-allocate it. (unless you don't mind using many gigabytes of memory) EDITx2: That is, perhaps you can define data structures that define filled content rather than defining every possible voxel in the world, even the "empty" ones. (I'm assuming the vast majority of voxels are "empty" rather than "filled")

EDIT: Although not trivial, especially if most of the space is considered "empty", then your best bet would be to introduce some sort of spatial tree that will let you efficiently query your world to see what objects are in a particular area. For example: Octrees (as Eric suggested) or RTrees

Improve this answer

4 Comments

What "above answers" are you contradicting? All of the answers besides yours state that the max size of an object is 2 GB.
That the memory size restriction is the final nail in the coffin. Perhaps I'm misinterpreting the MSDN docs, but it sounds like you can work around the maximum array size in 64 bit environments, but still limited to a maximum array element count.
@Servy Sorry I misspoke. I'm looking for a workaround to create a 3D table with many given. I edited the title
Thank you I just used octree. I cut the world of my "BigArray" in 64 regions and calculating the octree afterwards. But first I declare my world in this array.
1

Creating this object as described, either as a standard array or as a jagged array, is going to destroy the locality of reference that allows your CPU to be performant. I recommend you use a structure like this instead:

class BigArray 
{
    ArrayCell[,,] arrayCell = new ArrayCell[32,16,32];

    public int this[int i, int j, int k]
    { 
        get { return (arrayCell[i/64, j/64, k/64])[i%64, j%64, k%16]; } 
    }
}


class ArrayCell 
{
    int[,,] cell = new int[64,64,64];

    public int this[int i, int j, int k] 
    { 
        get { return cell[i,j,k]; } 
    }  
}
Improve this answer

3 Comments

Thank you I'll try to implement this in my code and I give back my resutlats on.
I only corrected 2-3 syntax error. But there he was not an error on this line -> return (arrayCell[i/64, j/64, k/64])[i%64, j%64, k%16]; to return (arrayCell[i/64, j/64, k/64])[i%64, j%32, k%64]; ?
I'm sorry but I can not understand the logical "return (arrayCell[i/64, j/64, k/64])[i%64, j%64, k%16];". Is it possible to modify your code with an array of 2048,1024,2048

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.