1

I have been trying to do some text manipulation in Python and am running into a lot of issues, mainly due a fundamental misunderstanding of how file manipulation works in Python so I am hoping to clear that up.

So lets say I'm iterating through a text file called "my.txt" and it has the following contents: 

3 10 7 8     
2 9 8 3  
4 1 4 2

The code I'm using to iterate through the file is: 

file = open ("my.txt", 'r')
for line in file:
    print line`

I copied and pasted the above code from a tutorial. I know what it does but I don't know why it works and it's bothering me. I am trying to understand exactly what the variable "line" represents in the file. Is it a data type(a string?) or something else. My instinct tells me that each line represents a string which could then be manipulated(which is what I want) but I also understand that strings are immutable in Python.

What role is memory playing into all this, if my file is too big to fit into memory will it still work? Will line[3] allow me to access the fourth element in each line? If I only want to work on the second line can I do: 

if line == 2:

within the for loop?

It might be worth noting that I am pretty new to Python and am coming from a C\C++ background(not used to immutable strings). I know I squeezed quite a few questions into one but any clarification on the general topic would really be helpful :)

Improve this question
3
  • 3
    I believe you should read the python tutorial. line is, as you thought, a string. In particular is a "line", i.e. at every iteration line gets assigned to a whole line in the file. Strings are immutable in python(as in most other languages), but they provide a whole lot of methods to manipulate them(creating new strings). Regarding memory: iterating over a file does not read the whole file into memory. It is read by chunks(avoiding both too high memory consumption and too low performances). Commented Aug 20, 2013 at 19:41
  • If you want to have both the line content and the line number you can use enumerate: for index, line in enumerate(file_object) This will assign index = 0 to the first line read, then index = 1 etc. (the syntax for a,b,c,d,e in iterable is called tuple-unpacking which may be worth reading about, since it is used a lot). Commented Aug 20, 2013 at 19:45
  • As a note, you should always use the with statement when working with files in Python. Commented Aug 20, 2013 at 21:16

5 Answers 5

Reset to default
3

line is a line of text, represented as a string. Strings are immutable, but that's not an issue for manipulating them; all variables in Python are references, and assigning to a variable points the reference to a new object. (In C++, you can't change where a reference points.) Iterating over a file iterates over the lines, so on each iteration, line refers to a new string representing the next line of the input file.

If you're familiar with range-based for loops or other language's for-each constructs, that's how Python's for works. The loop variable is not a counter; you can't do

if line == 2:

because line isn't the index of the line; it's the line itself. You could do

for i, line in enumerate(f):
    if i == 2:
        do_stuff_with(line)
        break  # No need to load the rest of the file

Note that file is the name of a builtin, so it's a bad idea to use that name for your own variables.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Is file a built in? It's a NameError in 3.3.
Side note: to iterate only over certain lines(e.g. every second line starting from the third up to 1531 lines), itertools.islice can be used: for line in itertools.islice(file_object, 3, 1531, 2)). @user2387370 file was a built-in in python2. In python3 it isn't anymore and can be safely used.
@user2387370: They changed it in Python 3. I think the new API is in the io module.
Yeah, file is a perfectly acceptable in 3.x, and in 2.x, it wasn't really something you used often (open() should be the function to use, file was just the type), so, while smashing it wasn't the best idea, it was seldom a problem.
2

In Python, you can iterate straight over a file. The best way of doing this is with a with statement, as in:

with open("myfile.txt") as f:
    for i in f:
        # do stuff to each line in the file

The lines are strings representing each line (seperated by newlines) in the file. If you only want to operate on the second line, you could do something like this:

with open("myfile.txt") as f:
    list_of_file = list(f)
    second_line = list_of_file[2]

If you then want to access part of the second line you can split it by spaces into another list as so:

second_number_in_second_line = second_line.split()[1]

With regards to memory, iterating through the file directly does not read it all into memory, however, turning it into a list does. If you want to access individual lines without doing so, use itertools.islice.

Improve this answer

2 Comments

This does not answer the question.
What if I only wanted to do something to a specific line/lines and access certain elements in each line
1

Suppose you have your same file:

3 10 7 8\n     
2 9 8 3\n  
4 1 4 2\n

There are many file methods that operate on a file object

In Python, you can read a file character by character, C style:

with open('/tmp/test.txt', 'r') as fin:     # fin is a 'file object' 
    while True:
        ch=fin.read(1)
        if not ch:
            break
        print ch,                           # comma suppresses the CR

You can read the whole file as a single string:

with open('/tmp/test.txt', 'r') as fin:
    data=fin.read()
    print data

As enumerated lines:

with open('/tmp/test.txt', 'r') as fin:
    for i, line in enumerate(fin):
        print i, line

As a list of strings:

with open('/tmp/test.txt', 'r') as fin:
    data=fin.readlines()

The idiom of looping over a file object:

for line in fin:                 # 'fin' is a file object result of open
    print line

is synonymous with:

for line in fin.readline():
    print line

and similar to:

for line in 'line 1\nline 2\nline 3'.splitlines():
    print line

Once you get used to the Python style loops (or Perl, or Obj C, or Java range style loops) that loop over the elements of something -- you use them without thinking about it much.

If you want the index of each item -- use enumerate

Improve this answer

Comments

1

In each iteration the line variable is filled with contents of subsequent lines read from the file. So, you'll have:

"3 10 7 8" in first iteration
"2 9 8 3" in second iteration
etc.

To get the numbers separately, use the split method: link.

So comparing line with 2 doesn't make sens. If you want to identify line numbers, you can try:

lineNumber = 0
for line in file:
  print line
  if lineNumber == 2:
    print "that was the second line!"
  lineNumber += 1

As suggested in the comment, you can simplify this by using enumerate:

for lineNumber, line in enumerate(file):
  print line
    if lineNumber == 2:
      print "that was the second line!"
Improve this answer

3 Comments

better to use enumerate here. it's made for things exactly like this
@BartoszKP So on the second iteration are the elements "3 10 7 8" completely gone if I don't store them as a variable on the first iteration? Lets say I need to do a file manipulation in which the numbers I want to split depend on the numbers that were split on the previous iteration. Can this be done?
@user2202911 Yes, in each iteration contents of previous lines are forgotten, unless you store them somehow. Then you can parse the lines depending on previous lines. But I suggest you read some more about Python, and read more example programs first. Then all this will come to you naturally, and you won't waste your time on asking such elementary questions and waiting for someone to answer :)
0

You can iterate over a file of any size, with the code you have shown, and it should not consume any significant amount of memory beyond the size of the longest single line.

As for how it works, under the hood, you could dive into the source code for Python itself to learn the gory details. At a higher level just consider that the implementor of file objects, in Python, chose to implement line-by-line iteration as a feature of their class.

Many of the collection data types and I/O interfaces in Python implement some form of iteration. Thus the for construct is the most common type of looping in Python. You can iterate over lists, tuples, and sets (by item), strings (by character), dictionaries (by key), and many classes (including those in the standard libraries as well as those from third parties) implement the "iterator (coding) protocol" to facilitate such usage.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.