0

I have a program that returns rows of allowed arrays (as follows) which are stored in a file (say output.txt), using python program.py>output.txt on the windows 7 command line:

0 , 0 , 1 , 1 , 6
0 , 0 , 1 , 2 , 1
0 , 0 , 1 , 4 , 1
0 , 0 , 1 , 6 , 1
0 , 0 , 2 , 0 , 0

I later use output.txt as input for another program, which reads it row by row into an array. The output.txt file is about 1.5 GB. I can use 7-zip to reduce it to 33 MB.

I risk running out of memory for the larger output data sizes. I'm aware that we can read zipped files in Python so is it possible to get zipped outputs as well?

My program has multiple nested loops, so I can make a small zip file after a cycle in the innermost loop and append it to another which gradually becomes larger as the loop traverses ahead.

Improve this question
8
  • Are you simply asking how to compress the result before writing it to a file, or how to compress part of the result during your computations before running out of memory? Commented Oct 12, 2017 at 10:25
  • 2
    You should not burden your application with such things. The UNIX philosophy would be something along the lines of python program.py | gzip > output.gz. Commented Oct 12, 2017 at 10:30
  • 1
    It wasn't clear because I'd expect someone to run out of memory before running out of hdd space. And "is it possible to get zipped outputs" seems to ask how to simply compress an output, not compress chunks before running out of memory. Regardless, to compress your output import gzip and gzip.compress(bytestream). Commented Oct 12, 2017 at 10:33
  • 1
    How to append to a gzip file might also be of interest. If you show us exactly how your loops are structured and how you store your data in a python object, we could give explicits answer with code. Commented Oct 12, 2017 at 10:44
  • 1
    Zipping strings in Python is easy. Still, I tend to agree with deceze. Let the shell handle the compression. Doing compression / decompression within the application makes sense if the application's task requires manipulation of data stored in compressed files, eg an epub file editor. Or if you're processing image / media files, which generally used compressed file formats, but even then you should be using a library that handles the compression for you. Commented Oct 12, 2017 at 10:52
Related questions
1
file compression without using built in functions
1
Compression for Python source files
1
Smallest possible windows executable from python script
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.