Extract data from a particular text file row using numpy

Question

I have a file that contains some numerical data. What I need to do is find the third to last row and need to extract that data out, and put it into another file called newyork.csv.

However, I am having a hard time extracting the 3rd 4th 5th etc row; this is the code that I have. If I would guess I think the problem starts at----- for line in file2:.

How can I get this code to extract the 3rd to last row?

Weather = open("weathernewyork.txt",'r').read().split('\n')

csvholder = [] 
i = 0
while i < len(Weather):
    with open(Weather[i], 'r') as file2:
        reader = csv.reader(file2)
        wtr__12 = open(Weather[i]).read().split('\n')            
        new_york_d  = wtr__12[-3]
        new_york_d = numpy.array(new_york_d)

        for line in file2:
            line = line.strip()
            new_york_d = line.split(",")[4:]


        xx = numpy.array(new_york_d).reshape(-1,8)
        csvholder.append(xx)        
        i = i+1

xxz = numpy.array(csvholder).reshape(-1,8)
numpy.savetxt(newyork.csv", xxz, delimiter=",", fmt='%s')

This is how the data file looks like.

0.0002%,3/30/2005,0.205130307,-0.001238007,1,0,0,0,0,1,1,1 <- I want to extract this one
0.0004%,3/31/2005,-0.10252641,-0.010432191,1,0,0,0,1,1,1,1
-0.0009%,4/1/2005,0.101510875,-0.000877706,1,0,0,0,0,1,1,1 <- Python extracts this one which I don't want

UPDATE: I forgot to mention that the file

Weather = open("weathernewyork.txt",'r').read().split('\n')

inside has other files to open. I am looking at all counties.

Your code is very confusing. You're reading each of the files named in Weather twice. Once you read the whole contents into a list, and take the third from last line. But then you overwrite the numpy array you created from that line with each of the other lines you read from the file, in turn. You also have a csv.reader you never use.... — Blckknght
– Blckknght, Commented Feb 26, 2017 at 7:31
UPDATE: I forgot to mention that the file Weather = open("weathernewyork.txt",'r').read().split('\n') inside has other files to open. i am looking at all countys,states,cities etc I think you are right I am overwritting the file what is the correct way to code that — ben olsen
– ben olsen, Commented Feb 26, 2017 at 8:20

Filip Kilibarda · Accepted Answer · 2017-02-26 18:27:32Z

1

I'm not sure what exactly you're trying to do in the greater context, but if all you want is "find the third to last row and need to extract that data out". Then I'd do this.

Sample Data

a,b,c,d,a,b,c,d
a,b,c,d,a,b,c,d
a,b,c,d,a,b,c,d
t,h,i,s,t,h,i,s
a,b,c,d,a,b,c,d
a,b,c,d,a,b,c,d

Get third last row

with open("data.csv") as f:
    content = f.readlines()

print(content[-3])
print(content[-3].split(",")[4:])

Out:
t,h,i,s,t,h,i,s
['t', 'h', 'i', 's\r\n']

Then you can store that in another csv like you've shown.

edited Feb 26, 2017 at 18:27

answered Feb 26, 2017 at 7:41

Filip Kilibarda

2,7082 gold badges24 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ben olsen Over a year ago

That doesnt work we need to incorporate that we are only pulling data from [4:] - this part 1,0,0,0,0,1,1,1

Daniel · Accepted Answer · 2017-02-26 07:36:43Z

0

Use deque to limit the amount of lines stored and take the first element:

from collections import deque
csvholder = [] 
with open("weathernewyork.txt",'r') as filenames:
    for filename in filenames:
        with open(filename.strip()) as datafile:
            data = deque(datafile, 3)
        csvholder.append(data[0].strip().split(',')[4:])
xxz = numpy.array(csvholder).reshape(-1,8)
numpy.savetxt("newyork.csv", xxz, delimiter=",", fmt='%s')

answered Feb 26, 2017 at 7:36

Daniel

42.9k4 gold badges57 silver badges82 bronze badges

Collectives™ on Stack Overflow

Extract data from a particular text file row using numpy

2 Answers 2

Sample Data

Get third last row

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Sample Data

Get third last row

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related