Storing and unpacking a very long array of ints in C

Question

I have a very long array of ints (3,000,000+) that I want to store and be able to unpack quickly. At first, I wrote each int on a separate line in a text file, but that took rather long to unpack as the strings had to be converted back to ints. Then, someone suggested that I just write the array itself to the file, but when I read it back and print out the elements, it only prints zeroes. The code looked something like this:

#include <stdio.h>
#include <stdlib.h>

#define LENGTH 3000000

int main() {
    FILE *f = fopen("file.txt", "w");
    int *arr = malloc(LENGTH * sizeof(int));

    /* initialization of arr not shown */

    fwrite(arr, sizeof(int), LENGTH, f);
    fclose(f);
    free(arr);
    return 0;
}

And then to unpack the data:

#include <stdio.h>
#include <stdlib.h>

#define LENGTH 3000000

int main() {
    FILE *f = fopen("file.txt", "r");
    int *arr = malloc(LENGTH * sizeof(int));
    fread(arr, sizeof(int), LENGTH, f);
    fclose(f);

    for (int i = 0; i < LENGTH; ++i) {
        printf("%d\n", arr[i]);
    }

    free(arr);
    return 0;
}

But, as I said, it only prints zeroes. Why does this not work, and is there a more appropriate way of doing such a thing?

Check the return value of fread and fwrite to make sure you read / wrote the number of items you expected. Also, start by testing with a smaller array size. — dbush
– dbush, Commented Mar 3, 2021 at 13:56
Instead of assuming how big the file is, why not use ftell and find out? — tadman
– tadman, Commented Mar 3, 2021 at 13:59
If your question is "What would be the best way...", then answer wil be opinion based and question should be closed. If your question is "Why does this not work?" then you need have better error handling to figure out where the problem is. Either way this question cannot be answered with information you have given. Please edit your question to make it answerable. — user694733
– user694733, Commented Mar 3, 2021 at 14:05
Problem is not sufficiently presented in code. Please post enough code that demos the problem (after applying this recommended improvement) — chux
– chux, Commented Mar 3, 2021 at 14:18
"initialization of arr not shown" may be not performing as expected. — chux
– chux, Commented Mar 3, 2021 at 14:27

Govind Parmar · Accepted Answer · 2021-03-03 14:09:12Z

Few things:

When reading and writing the ints directly, you must open the file with the mode b as well:

 // Open for writing
 FILE *fp = fopen("myfile.bin", "wb");


 // Open for reading:
 FILE *fp = fopen("myfile.bin", "rb");

When calling malloc you should always check that the return value is not NULL to ensure that the allocation succeeded. When allocating 3000000 * sizeof(int) bytes, not checking for success is a disaster waiting to happen. You should also check the return values of fopen, fread, and fwrite:

 // Checking malloc for success
 arr = malloc(LENGTH * sizeof(int));
 if(NULL == arr)
 {
      perror("malloc");
      exit(EXIT_FAILURE);
 }

 // Checking fopen for success:
 FILE *fp = fopen("file.bin", "rb");
 if(NULL == fp)
 {
     perror("fopen");
     exit(EXIT_FAILURE);
 }

 // Checking fread for success:
 size_t readCnt = fread(arr, sizeof(int), LENGTH, fp); 
 if(readCnt != LENGTH)
 {
     // Did not read file successfully
 }

 // Checking fwrite for success:
 size_t writtenCnt = fwrite(arr, sizeof(int), LENGTH, fp);
 if(writtenCnt != LENGTH)
 {
     // Did not write file successfully
 }

Collectives™ on Stack Overflow

Storing and unpacking a very long array of ints in C

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related