Dividing a .txt file in multiple parts in Python

Question

I'm a begginer in Python, and I have a question about file reading : I need to process info in a file to write it in another one. I know how to do that, but it's reaaally ressource-consuming for my computer, as the file is really big, but I know how it's formatted ! The file follows that format :

I won't explain what it is for, as it would take ages and would not be very useful, but the file is essentialy made of four lines like these, again and again. For now, I use this to read the file and convert it in a very long chain :

inputfile = open("input.txt", "r")
output = open("output.txt", "w")
Chain = inputfile.read()
Chain = Chain.split("\n")
Chained = ' '.join(Chain)
Chain = Chained.split(" ")
Chain = list(map(int, Chain))

Afterwards, I just treat it with "task IDs", but I feel like it's really not efficient. So do you know how I could divide the chain into multiple ones knowing how they are formatted? Thanks for reading !

Yosef O · Accepted Answer · 2017-11-18 11:07:57Z

1

How about:

res = []
with open('file', 'r') as f:
  for line in f:
    for num in line.split(' '):
      res.append(int(num))

Instead of reading the whole file into memory, you go line by line. Does this help?

If you need to go 4 lines at a time, just add an internal loop.

Regarding output, I'm assuming you want to do some computation on the input, so I wouldn't necessarily do this in the same loop. Either process the input once reading is done, or instead of using a list, use a queue and have another thread read from the queue while this thread is writing to it.

Perhaps the utility of a list comprehension will help a bit as well (I doubt this will make an impact):

res = []
with open('file', 'r') as f:
  for line in f:
    res.append( int(num) for num in line.split() )

edited Nov 18, 2017 at 11:07

answered Nov 18, 2017 at 10:50

Yosef O

3672 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Fathin Luqman Tantowi · Accepted Answer · 2017-11-18 11:22:53Z

1

hmm there's some method to write to a file without reading it i believe

Add text to end of line without loading file

https://docs.python.org/2.7/library/functions.html#print

from __future__ import print_function
# if you are using python2.7
i = open("input","r")
f = open("output.txt","w")
a = "awesome"
for line in i:
    #iterate lines in file input
    line.strip()
    #this will remove the \n in the end of the string
    print(line,end=" ",file=f) 
    #this will write to file output with space at the end of it

this might help, i'm a newbie too, but with better google fu XD

answered Nov 18, 2017 at 11:22

Fathin Luqman Tantowi

215 bronze badges

Comments

ritchie46 · Accepted Answer · 2017-11-18 10:49:20Z

0

Maybe do it line by line. This way it consumes less memory.

inputfile = open("input.txt", "r")
output = open("output.txt", "a")

while True:
    line = inputfile.readline()
    numbers = words.split(" ")
    integers = list(map(int, numbers))

    if not line: 
       break

There is probably a newline character \n in the words. You should also replace that with an empty string.

answered Nov 18, 2017 at 10:49

ritchie46

15.6k2 gold badges45 silver badges71 bronze badges

Comments

Łukasz Szczesiak · Accepted Answer · 2017-11-18 11:00:26Z

0

If you don't wanna to consume memory (you can run of it if file is very large), you need to read lien by line.

with open('input.txt', 'w') as inputfile, open('"output.txt', 'w') as output:
    for line in inputfile:
        chain = line.split(" ")
        #do some calculations or what ever you need
        #and write those numbers to new file
        numbers = list(map(int, chain))
        for number in numbers
            output.write("%d " % number)

answered Nov 18, 2017 at 11:00

Łukasz Szczesiak

2241 silver badge10 bronze badges

Collectives™ on Stack Overflow

Dividing a .txt file in multiple parts in Python

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related