reading a big file in binary with custom line terminator and writing in smaller chunks in python

Question

I have a file that uses \x01 as line terminator. That is line terminator is NOT newline but the bytevalue of 001. Here is the ascii representation for it which ^A.

I want to split file to size of 10 MB each. Here is what I came up with

size=10000 #10 MB
i=0
with open("in-file", "rb") as ifile:
    ofile = open("output0.txt","wb")
    data = ifile.read(size)
        while data:
            ofile.write(data)
            ofile.close()
            data = ifile.read(size)
            i+=1 
            ofile = open("output%d.txt"%(i),"wb")


    ofile.close()

However, this would result in files that are broken at arbitrary places. I want the files to be terminated only at the byte value of 001 and next read resumes from the next byte.

is the byte just \x01?

Joran Beasley
– Joran Beasley

2017-08-25 19:26:04 +00:00
Commented Aug 25, 2017 at 19:26 — Joran Beasley
– Joran Beasley, Commented Aug 25, 2017 at 19:26
@JoranBeasley yes

brain storm
– brain storm

2017-08-25 19:32:27 +00:00
Commented Aug 25, 2017 at 19:32 — brain storm
– brain storm, Commented Aug 25, 2017 at 19:32

Joran Beasley · Accepted Answer · 2017-08-25 19:38:15Z

1

if its just one byte terminal you can do something like

def read_line(f_object,terminal_byte): # its one line you could just as easily do this inline
    return "".join(iter(lambda:f_object.read(1),terminal_byte))

then make a helper function that will read all the lines in a file

def read_lines(f_object,terminal_byte):
    tmp = read_line(f_object,terminal_byte)
    while tmp:
        yield tmp
        tmp = read_line(f_object,terminal_byte)

then make a function that will chunk it up

def make_chunks(f_object,terminal_byte,max_size):
    current_chunk = []
    current_chunk_size = 0
    for line in read_lines(f_object,terminal_byte):
        current_chunk.append(line)
        current_chunk_size += len(line)
        if current_chunk_size > max_size:
            yield "".join(current_chunk)
            current_chunk = []
            current_chunk_size = 0
    if current_chunk:
        yield "".join(current_chunk)

then just do something like

with open("my_binary.dat","rb") as f_in:
    for i,chunk in enumerate(make_chunks(f_in,"\x01",1024*1000*10)):
        with open("out%d.dat"%i,"wb") as f_out:
            f_out.write(chunk)

there might be some way to do this with libraries (or even an awesome builtin way) but im not aware of any offhand

answered Aug 25, 2017 at 19:38

Joran Beasley

114k13 gold badges167 silver badges187 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

brain storm Over a year ago

It doesnt seem to split on the terminal_byte. The terminal byte I used is bytes(chr(1))

brain storm Over a year ago

I just noticed that terminal byte is not written in output file. I want to join on "\x01"

brain storm Over a year ago

I modified "".join(iter(lambda:f_object.read(1),terminal_byte)) to "\x01".join(iter(lambda:f_object.read(1),terminal_byte)) and yield "\x01".join(current_chunk). but that is not working

Joran Beasley Over a year ago

what does "that is not working" mean? you probably just want "\x01".join(current_chunk)+"\x01"

brain storm Over a year ago

that did not work. however, this worked. def read_line(f_object,terminal_byte): return ''.join(iter(lambda:f_object.read(1),terminal_byte)) + "\x01"

|

Collectives™ on Stack Overflow

reading a big file in binary with custom line terminator and writing in smaller chunks in python

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related