Writing list of dict into file in python

Question

I am having trouble in dynamically mapping the dictionary values while writing the db output into a file. Scenario:

new_list = [{'Table':'A', 'Column':'C1', 'DataType':'int'},
        {'Table':'A', 'Column':'C2', 'DataType':'varchar'},
        {'Table':'A', 'Column':'C2', 'DataType':'numeric'}
       ]
# I want to write the data into a file.
Table|Column|DataType
A|C1|int
A|C2|varchar
A|C3|numeric

I am trying to do like below.

header = []
with open('my_log.log', 'w',encoding='utf-8') as log:
   for n in new_list:
      for i in n.keys():
        header.append(i)
   log.write("|".join(set(header)))
   log.write("\n")
   for data in new_list:
      # don't want to hard code the keys like below
      log.write("{Table}|{Column}|{DataType} \n".format(**data))
      # need to do something so that I dont have to hard code the keys as it  is dynamic in nature
      # and also my file output should match with the header generated in the previous line
      log.write("{???}".format(**data))

Any Suggestion!

A dictionary can be serialized using pickle. You could look into that — Jens Munk
– Jens Munk, Commented Apr 4, 2016 at 19:44
You could also write the list out using json.dump more information can be found within Python's JSON documentation. — Cory Shay
– Cory Shay, Commented Apr 4, 2016 at 19:54
I second the approach of dumping to JSON, but if you want to build your own file then seriously consider my approach over the accepted solution. csv.DictWriter is the "one...obvious way to do it" — Adam Smith
– Adam Smith, Commented Apr 4, 2016 at 21:03

Brent Washburne · Accepted Answer · 2016-04-04 19:56:53Z

4

Here's a way to write out the data using a dynamic list of headers:

new_list = [{'Table':'A', 'Column':'C1', 'DataType':'int'},
        {'Table':'A', 'Column':'C2', 'DataType':'varchar'},
        {'Table':'A', 'Column':'C2', 'DataType':'numeric'}
       ]

header = new_list[0].keys()

with open('my_log.log', 'w') as log:
   log.write("|".join(header))
   log.write("\n")
   for data in new_list:
      log.write("|".join(data[h] for h in header))
      log.write("\n")

answered Apr 4, 2016 at 19:56

Brent Washburne

13.2k4 gold badges65 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Adam Smith Over a year ago

Note that this doesn't guarantee the order of the columns, since dictionaries are orderless.

Brent Washburne Over a year ago

@AdamSmith True, but if the data is being read back into a dictionary then it doesn't matter.

Adam Smith Over a year ago

Absolutely! Though if the data is being read back into a dictionary, you could just serialize with pickle or JSON :)

Adam Smith · Accepted Answer · 2016-04-04 20:45:50Z

2

You're writing delimited text, so you should be using the csv module. It happens to have a DictWriter object that is perfect for this.

import csv

new_list = [{'Table':'A', 'Column':'C1', 'DataType':'int'},
    {'Table':'A', 'Column':'C2', 'DataType':'varchar'},
    {'Table':'A', 'Column':'C2', 'DataType':'numeric'}
   ]

with open("my_log.log", "wb") as f:
    writer = csv.DictWriter(f,
                            fieldnames=["Table", "Column", "DataType"],
                            delimiter="|")
    writer.writerows(new_list)

answered Apr 4, 2016 at 20:45

Adam Smith

54.6k13 gold badges85 silver badges120 bronze badges

6 Comments

Pradeep Over a year ago

Is there any way we can set the fieldnames dynamically because new_list will vary depending on the query?@Adam Smith

Adam Smith Over a year ago

@Pradeep Sure, any way you could normally manipulate that. The problem you will run into is the same as in Brent Washburne's answer. Dictionary keys are not ordered, so you can't guarantee that {"A": 1, "B": 2, "C": 3}.keys() will give you ["A", "B", "C"] and not ["B", "C", "A"] (or any other arbitrary ordering.)

Adam Smith Over a year ago

If you need to get a list of arbitrary keys and you care at all about their order, I would define them to be a collections.OrderedDict beforehand, or simple decide "I'm using the alphabetical sort of all these keys." (fieldnames=sorted(new_list[0].keys()))

Adam Smith Over a year ago

@Pradeep glad I could help!

Pradeep Over a year ago

Is there any way we can set the UTF-8 encoding because i get the below error for encoding. "UnicodeEncodeError: 'charmap' codec can't encode character '\u0142' in position 4: character maps to <undefined>" . Also i just changed the mode only to "w" instead of "wb" to work in my scenario. @Adam

|

goCards · Accepted Answer · 2016-04-04 19:58:40Z

0

Use python pandas.

import pandas as pd
tableA = pd.DataFrame([[1,2,3],[4,5,6]], columns=["C1","C2","C3"])
tableA.to_csv("mytables.csv",sep="|")

OUTPUT: mytables.csv

|C1|C2|C3

0|1|2|3

1|4|5|6

answered Apr 4, 2016 at 19:58

goCards

1,44611 silver badges10 bronze badges

Collectives™ on Stack Overflow

Writing list of dict into file in python

3 Answers 3

3 Comments

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related