python how to read files with block structure

Question

usually, I use python to read csv files, the structure of which looks like:

date1, value1_1, value1_2, value1_3, ...
date2, value2_1, value2_2, value2_3, ...
...

in this case, one line is one piece of data and I just use numpy.loadtxt() to read them.

but today, my colleague gave me a file with block structure, which looks like:

date1
value1_1, value1_2
value1_3, ...
date2
...

and this gives me a headache...

Does anyone have any good solution for this? Is there a function I can use to deal with this file, or do i have to write a reading_messed_files() function myself?

It might be easier to fix this at the source, and ask your colleague is she or he can give you standard a csv-formatted file instead. — user707650
– user707650, Commented Dec 14, 2015 at 4:09
Are these still line delimited strings? Is there some structure? — tdelaney
– tdelaney, Commented Dec 14, 2015 at 4:12
Without a decent example of what this file looks like, we can't make a guess on how to parse it. — tdelaney
– tdelaney, Commented Dec 14, 2015 at 4:27
the second file is line delimited strings. In the csv case, each line contains one complete piece of data, whereas in the second file, the combination of line1, 2 and 3 is one complete piece of data — Fei
– Fei, Commented Dec 14, 2015 at 4:33

hpaulj · Accepted Answer · 2015-12-14 04:53:42Z

This isn't a full answer, but a little long for a comment.

numpy csv readers like loadtxt and genfromtxt accept any iterable as input. While typically it's a filename that it opens and reads line by line, it can also be a list of lines, or a generator that returns one line at a time.

So you could open the file, read it line by line, rework the blocks into normal csv lines, and pass them on to loadtxt.

I remember examples using this to read multiple files (with the same columns), to skip lines, or to read blocks. Also examples process the lines to replace awkward delimiters.

I frequently demonstrate loadtxt using a list of lines derived from a cut-n-paste example.

I'm thinking of something like:

def foo(afile):
    header=None
    for line in afile:
        strings = line.split(delimiter)
        if len(strings)==1:
             header = strings[0]
        else:
            line = delimiter.join([header]+strings)
            yield line

with open(filename) as f:
    A = np.loadtxt(foo(f),....)

Collectives™ on Stack Overflow

python how to read files with block structure

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related