0

I have a generator object 'results', which when looping through returns a list of dictionaries. I'm trying to convert this into a list of lists, so I can easily loop through and reference each value to be INSERTed into a database. I believe I'm having trouble because this is a generator object, how can I do this?

such as:

def parse(results):
    for r in results:
        print(r)

results:

[{'id': '7229957054', 'repost_of': None, 'name': '1996 Acura Integra', 'url': 'https://monterey.craigslist.org/cto/d/salinas-1996-acura-integra/7229957054.html', 'datetime': '2020-11-12 14:37', 'last_updated': '2020-11-12 14:37', 'price': '$1,000', 'where': 'Salinas', 'has_image': True, 'geotag': None, 'deleted': False}, {'id': '7229839309', 'repost_of': None, 'name': '1990 Acura Integra GS', 'url': 'https://monterey.craigslist.org/cto/d/salinas-1990-acura-integra-gs/7229839309.html', 'datetime': '2020-11-12 11:31', 'last_updated': '2020-11-12 11:31', 'price': '$2,800', 'where': 'Salinas, Ca', 'has_image': True, 'geotag': None, 'deleted': False}]

my code:

def initialParse(results):
    rList = []
    for r in results:
        r_id = str(r['id'])
        r_name = str(r['name'])
        r_url = str(r['url'])
        r_datetime = str(r['datetime'])
        r_updated = str(r['last_updated'])
        r_price = str(r['price'])
        r_where = str(r['where'])
        iList = list(r_id + r_name + r_url + r_datetime + r_updated + r_price + r_where)
        rList.append(iList)
    print(rList)

returns:

[['7', '2', '2', '9', '9', '5', '7', '0', '5', '4', '1', '9', '9', '6', ' ', 'A', 'c', 'u', 'r', 'a', ' ', 'I', 'n', 't', 'e', 'g', 'r', 'a', 'h', 't', 't', 'p', 's', ':', '/', '/', 'm', 'o', 'n', 't', 'e', 'r', 'e', 'y', '.', 'c', 'r', 'a', 'i', 'g', 's', 'l', 'i', 's', 't', '.', 'o', 'r', 'g', '/', 'c', 't', 'o', '/', 'd', '/', 's', 'a', 'l', 'i', 'n', 'a', 's', '-', '1', '9', '9', '6', '-', 'a', 'c', 'u', 'r', 'a', '-', 'i', 'n', 't', 'e', 'g', 'r', 'a', '/', '7', '2', '2', '9', '9', '5', '7', '0', '5', '4', '.', 'h', 't', 'm', 'l', '2', '0', '2', '0', '-', '1', '1', '-', '1', '2', ' ', '1', '4', ':', '3', '7', '2', '0', '2', '0', '-', '1', '1', '-', '1'...]

Moving the rList.append() out one block gives a list in a list containing ALL entries... I need each result r in it's own list inside a list... like this:

[['id', 'name', 'url', 'datetime', 'lastupdated', 'price', 'where'], ['id', 'name', 'url', 'datetime', 'lastupdated', 'price', 'where'], ... ]

what am I doing incorrectly here?

Improve this question
1
  • The problem is iList = list(r_id + r_name + r_url + r_datetime + r_updated + r_price + r_where). Change for iList = [r_id + r_name + r_url + r_datetime + r_updated + r_price + r_where] Commented Nov 13, 2020 at 4:14

2 Answers 2

Reset to default
2

Look like you are trying to get this:

[list(r.keys()) for r in results]

That being said, there might be a better way to do what you're doing; could you explain more about how you plan to INSERT the items? Or do you want the values in the lists:

[list(r.values()) for r in results]

But really, it looks like you are dealing with structured data here, which means you should be adding it into a pandas dataframe and filtering rows or selecting columns as needed. All you need to do is:

import pandas as pd
df = pd.DataFrame(results)

You can read up on the pandas documentation to see how to operate on the data and get the table you want. It would also make it much easier to ultimately load into your SQL database.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks for the suggestion, well once I got the data into an object/format I know how to manage, I planned to use pyodbc and a loop to loop through each of the inner lists. The end goal is to insert each of those values into seperate columns in a table... as in, a separate column for each value in each list.
and yes, I want the values in the lists... don't much care about the keys.
And while your solution does work- I don't want EVERY value contained in the results dictionaries... only some of them... which is why I was trying to specify each of the ones I do want. But I suppose I can pull the ones I want from the list of lists after the fact. Thank you! @teepee
if you want to filter the data, you might as well just put everything into a pandas dataframe and filter from there. Added an edit to describe it.
Cool thank you- I will look into pandas, never used it but I hear about it all of the time. I can accomplish most things with my "hack and slash" style of coding... duct taping the things I know how to do together into one big messy masterpiece... but there's always a better more efficient way... and you guys always remind me of that. Thanks again!
1

Replace

iList = list(r_id + r_name ...)

with

iList = [r_id, r_name, ...]

r_id + r_name ... creates a big string by concatenating all the arguments, and the list() function creates a list of all characters in the string. You don't want that.

Improve this answer

5 Comments

I actually tried that, while it's closer to what I want, it doesn't separate the values into their own list items... it creates a big list containing lists of the values concatenated together, liks this: [['id name url date updated price where'], [...]] instead of [['id', 'name', 'url', ...
Not sure why it isn't working for you. I get the following output: [['2', 'kartik'], ['10', 'kartikSecond']]. Don't add the strings (that concatenates them together).
because my input is a generator. Yours is probably just a list with dictionaries in it... right?
when I remove the ''s I get a keyerror, when I remove the str() i get a "can't add notype to list error. what a pain!
What I meant was that you should use the str() and the ''s to extract the corresponding values. But when you create iList from the extracted strings, don't add the strings. Just write the strings within square brackets separated by commas (iList = [r_id, r_name ...]). Yes, I was using a list with dictionaries and not a generator, but I think this step is after you have extracted the strings. At this stage, it shouldn't make a difference whether top structure was a list of dicts or a generator

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.