Extracting data from a file that was read into memory

Question

I have a binary data file that contains 2d and 3d coordinates in such order:

uint32 numberOfUVvectors;
2Dvec uv[numberOfUVvectors];
uint32 numberOfPositionVectors;
3Dvec position[numberOfPositionVectors];
uint32 numberOfNormalVectors;
3Dvec normal[numberOfNormalVectors];

2Dvec and 3Dvec are structs composed from 2 and 3 floats respectively.

At first, I read all these values using the "usual" way:

in.read(reinterpret_cast<char *>(&num2d), sizeof(uint32));
2Dvectors.reserve(num2d); // It's for an std::vector<2DVec> 2Dvectors();
for (int i = 0; i < num2d; i++){
    2Dvec 2Dvector;
    in.read(reinterpret_cast<char *>(&2Dvector), sizeof(2DVec));
    2Dvectors.push_back(2Dvector);
}

It worked fine, but it was painfully slow (there can be more than 200k entries in a file and with so many read calls, the hdd access became a bottleneck). I decided to read the entire file into a buffer at once:

in.seekg (0, in.end);
int length = in.tellg();
in.seekg (0, in.beg);

char * buffer = new char [length];

is.read (buffer,length);

The reading is way faster now, but here's the question: how to parse that char buffer back into integers and structs?

Ran Proshan · Accepted Answer · 2013-05-22 17:40:29Z

To answer your specific question:

unsigned char * pbuffer = (unsigned char *)buffer;

uint32 num2d = *((uint32 *)pbuffer);
pbuffer += sizeof(uint32);
if(num2d)
{
    2Dvec * p2Dvec = (2Dvec *)pbuffer;
    2Dvectors.assign(p2Dvec, p2Dvec + num2d);
    pbuffer += (num2d * sizeof(2Dvec));
}

uint32 numpos = *((uint32 *)pbuffer);
pbuffer += sizeof(uint32);
if(numpos)
{
        3Dvec * p3Dvec = (3Dvec *)pbuffer;
    Pos3Dvectors.assign(p3Dvec, p3Dvec + numpos);
    pbuffer += (numpos * sizeof(3Dvec));
}

uint32 numnorm = *((uint32 *)pbuffer);
pbuffer += sizeof(uint32);
if(numnorm)
{
    3Dvec * p3Dvec = (3Dvec *)pbuffer;
    Normal3Dvectors.assign(p3Dvec, p3Dvec + numnorm);
    pbuffer += (numnorm * sizeof(3Dvec));
}
// do not forget to release the allocated buffer

A an even faster way would be:

in.read(reinterpret_cast<char *>(&num2d), sizeof(uint32));
if(num2d)
{
    2Dvectors.resize(num2d);
    2Dvec * p2Dvec = &2Dvectors[0];
    in.read(reinterpret_cast<char *>(&p2Dvec), num2d * sizeof(2Dvec));
}
//repeat for position & normal vectors

user1182183 · Accepted Answer · 2013-05-22 16:55:20Z

1

Use memcpy with the appropriate sizes and start values

or cast the values (example):

#include <iostream>

void copy_array(void *a, void const *b, std::size_t size, int amount)
{
    std::size_t bytes = size * amount;
    for (int i = 0; i < bytes; ++i)
        reinterpret_cast<char *>(a)[i] = static_cast<char const *>(b)[i];
}

int main()
{
    int a[10], b[10] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

    copy_array(a, b, sizeof(b[0]), 10);

    for (int i = 0; i < 10; ++i)
        std::cout << a[i] << ' ';
}

answered May 22, 2013 at 16:55

user1182183

Collectives™ on Stack Overflow

Extracting data from a file that was read into memory

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related