Write to file without redundant lines in python

Question

I'm writing python script to read line from a input file and write a unique lines(if the same line is not already in output file) to output file. somehow, my scripts always append the first line of input file to output file even if the same line is already in output file. I can't figure out why this happens. can anyone know why and how do I fix this? thanks,

import  os

input_file= 'input.txt'
output_file = 'output.txt'

fo = open(output_file, 'a+')
flag = False
with open(input_file, 'r') as fi:
    for line1 in fi:
       print line1
       for line2 in fo:
           print line2
           if line2 == line1:
               flag = True
               print('Found Match!!')
               break
       if flag == False:
           fo.write(line1)
       elif flag == True:
           flag == False
       fo.seek(0)
    fo.close()
    fi.close()

You are opening it in append mode. Do you think it is because of that ? Also, when you use with, you dont need to close explicitly. — karthikr
– karthikr, Commented Aug 30, 2015 at 3:53
So to be clear, you want to write lines from the input file to the output file only if they don't already exist in the output file; right? What if a line doesn't occur in the output file, but it occurs more than once in the input file? — Cyphase
– Cyphase, Commented Aug 30, 2015 at 4:08
my input files are already composed of unique lines but output file doesn't as output file get update by multiple input files. — Joohun Lee
– Joohun Lee, Commented Aug 30, 2015 at 4:40

kmacinnis · Accepted Answer · 2015-08-30 04:12:50Z

3

When you open a file in append mode, the file object position is at the end of the file. So the first time through, when it reaches for line2 in fo:, there aren't any more lines in fo, so that block is skipped, and flag is still true, so that first line is written to the output file. After that, you do fo.seek(0), so you are checking against the entire file for subsequent lines.

answered Aug 30, 2015 at 4:12

kmacinnis

6486 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Joohun Lee Over a year ago

I wasn't aware of "a"pend option moving the pointer to the end of file and that makes sense that why it didn't compare the at first line. Followed your suggestion, moving fo.seek(0) before for loop fixed my problem.

Cyphase · Accepted Answer · 2015-08-30 04:30:19Z

1

The answer by kmacinnis is right on as to why your code isn't working; you need to use mode 'r+' instead of 'a+', or else put fo.seek(0) at the beginning of the for loop instead of the end.

That said, there's a much better way to do this than reading the entire output file for every line of the input file.

def ensure_file_ends_with_newline(handle):
    position = handle.tell()

    handle.seek(-1, 2)
    handle_end = handle.read(1)
    if handle_end != '\n':
        handle.write('\n')

    handle.seek(position)


input_filepath = 'input.txt'
output_filepath = 'output.txt'

with open(input_file, 'r') as infile, open(output_file, 'r+') as outfile:
    ensure_file_ends_with_newline(outfile)

    written = set(outfile)

    for line in infile:
        if line not in written:
            outfile.write(line)
            written.add(line)

answered Aug 30, 2015 at 4:30

Cyphase

12k3 gold badges33 silver badges33 bronze badges

2 Comments

kmacinnis Over a year ago

If the files could be enormous, holding the entire contents in memory would be a problem. That said, this is much better if you know you have small files to work with. Also, I didn't realize you could have multiple file objects in a single with statement, so I'm thrilled to learn that! Thanks.

Cyphase Over a year ago

@kmacinnis, I considered that, but if your files are that big, that's a HUGE cost in IO to be reading the entire file once for every line in the input file. If you are dealing with files so big they won't fit in memory, there are better ways to do this than doing all that IO.

user4119159 · Accepted Answer · 2015-08-30 04:15:27Z

0

Your flag was never set to False.

flag == True is an equality

flag = True is an assignment.

Try the latter.

import  os

input_file= 'input.txt'
output_file = 'output.txt'

fo = open(output_file, 'a+')
flag = False
with open(input_file, 'r') as fi:
    for line1 in fi:
       #print line1
       for line2 in fo:
           #print line2
           if line2 == line1:
               flag = True
               print('Found Match!!')
               print (line1,line2)
               break
       if flag == False:
           fo.write(line1)
       elif flag == True:
           flag = False
       fo.seek(0)

answered Aug 30, 2015 at 4:15

user4119159

3 Comments

kmacinnis Over a year ago

That doesn't fix the problem that he asked about, but it does fix another problem I noticed, which is that no lines after the first match are written to the output file.

user4119159 Over a year ago

I must have misunderstood his question. Ran the code and only noticed this second bug. Nice catch.

Joohun Lee Over a year ago

thanks for picking this up. I made a change as well.

Collectives™ on Stack Overflow

Write to file without redundant lines in python

3 Answers 3

1 Comment

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related