Pandas DataFrame.to_csv last column missing when field is empty

Question

I am extracting data from a database into a csv file.

I realize that each time the last field of a row in Null, then DataFrame.to_csv omit it. This only happens when the empty field is at the last position.

Here is an exemple :

dframe_iterator = pandas.read_sql_query(request, engine, chunksize=1000)
for i, dataframe in enumerate(dframe_iterator):   
        dataframe.to_csv('file.csv', index=False, header=True, sep='|', mode='a', encoding='utf-8', date_format='%d/%m/%Y')

Let say one n-uplet returned by the sql query contains 2 Null values :

'blabla','blabla',Null, 'blabla', Null

Then, in the csv file, i get :

blabla|blabla||blabla

You can see that the first Null field is there (||) but the second Null filed is omitted.

I would expect this :

blabla|blabla||blabla|

Do you have any idea how to perform this? Another application is expecting as much fields as returned by the sql query.

Thanks!

can you give a minimum example dataframe? I'm struggling to reproduce this behaviour. — draco_alpine
– draco_alpine, Commented Aug 17, 2016 at 11:23
Sorry Draco. It's normal that you cannot reproduce this. My question was wrong. The pandas behaviour is perfect and exactly how i expected it. Too fast to post, not testing enough, and stupid me! — stockersky
– stockersky, Commented Aug 17, 2016 at 13:04
Please consider deleting the question.. Since nothing is wrong — Merlin
– Merlin, Commented Aug 17, 2016 at 15:11

stockersky · Accepted Answer · 2016-08-17 13:03:11Z

2

Hemm, well, I apologie but my question was wrong.

Actually, the pandas behaviour is perfectly good :

'blabla','blabla',Null, 'blabla', Null

would be :

blabla|blabla||blabla|

I have been troubled by a dataset having lots of Null field at last positions. Working on a different dataset made me realize this. And also wrong client specs which would expect blabla|blabla||blabla||

I really do apologie for being stupid and too fast to post.

answered Aug 17, 2016 at 13:03

stockersky

1,5813 gold badges20 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Eugene Lisitsky · Accepted Answer · 2016-08-17 12:38:30Z

0

Have you tried parameter na_rep? Doc

  na_rep : string, default ‘’
       Missing data representation

answered Aug 17, 2016 at 12:38

Eugene Lisitsky

13k6 gold badges42 silver badges63 bronze badges

Collectives™ on Stack Overflow

Pandas DataFrame.to_csv last column missing when field is empty

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related