5

I have a speed/efficiency related question about python:

I need to write a large number of very large R dataframe-ish files, about 0.5-2 GB sizes. This is basically a large tab-separated table, where each line can contain floats, integers and strings.

Normally, I would just put all my data in numpy dataframe and use np.savetxt to save it, but since there are different data types it can't really be put into one array.

Therefore I have resorted to simply assembling the lines as strings manually, but this is a tad slow. So far I'm doing:

1) Assemble each line as a string 2) Concatenate all lines as single huge string 3) Write string to file

I have several problems with this: 1) The large number of string-concatenations ends up taking a lot of time 2) I run of of RAM to keep strings in memory 3) ...which in turn leads to more separate file.write commands, which are very slow as well.

So my question is: What is a good routine for this kind of problem? One that balances out speed vs memory-consumption for most efficient string-concatenation and writing to disk.

... or maybe this strategy is simply just bad and I should do something completely different?

Thanks in advance!

Improve this question
3
  • Why don't you just write each bit of data to the file as you have it. It doesn't seem like there's any need to concatenate all those strings. Writes will automatically be buffered on their way to disk. Commented Feb 25, 2014 at 21:20
  • There just seems to be huge decrease in speed when file.write is called multiple times instead of just one - but maybe I'm wrong? Commented Feb 25, 2014 at 21:31
  • Unless you post your code, there's not much more people are likely to be able to help. Commented Feb 26, 2014 at 16:54

3 Answers 3

Reset to default
7

Seems like Pandas might be a good tool for this problem. It's pretty easy to get started with pandas, and it deals well with most ways you might need to get data into python. Pandas deals well with mixed data (floats, ints, strings), and usually can detect the types on its own.

Once you have an (R-like) data frame in pandas, it's pretty straightforward to output the frame to csv. 

DataFrame.to_csv(path_or_buf, sep='\t')

There's a bunch of other configuration things you can do to make your tab separated file just right.

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

I have never tried using Pandas, so I will give it ago. However, in some cases the actual text file will be a bit more messy than a dataframe
@Misconstruction, if this answered your question, please mark as answered with the check mark.
5

Unless you are running into a performance issue, you can probably write to the file line by line. Python internally uses buffering and will likely give you a nice compromise between performance and memory efficiency.

Python buffering is different from OS buffering and you can specify how you want things buffered by setting the buffering argument to open.

Improve this answer

3 Comments

How can you set buffering arguments?
open("foo", "wb", 1024) would set a 1k buffer. 0 and 1 have special meaning.
You shouldn't really need to set the size of buffers manually. The defaults work well most of the time.
0

I think what you might want to do is create a memory mapped file. Take a look at the following documentation to see how you can do this with numpy:

http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.