How to extract data in one FOR loop from JSON multiple dictionary - Python

Question

In my scrapy project I want to extract data from a website. It turned out that all information are stored in some script that I can easily read in JSON format and from there extract the data I need.

That's my function:

    def parse(self, response):
        items = response.css("script:contains('window.__INITIAL_STATE__')::text").re_first(r"window\.__INITIAL_STATE__ =(.*);")
        for item in json.loads(items)['offers']:
            yield {
                "title": item['jobTitle'],
                "employer": item['employer'],
                "country": item['countryName'],
                "details_page": item['companyProfileUrl'],
                "expiration_date": item['expirationDate'],
                'salary': item['salary'],
                'employmentLevel': item['employmentLevel'],
            }

And json file have that structure:

var = {
    "offers":[
      {
        "commonOfferId":"1200072247",
        "jobTitle":"Automatyk - Programista",
        "employer":"MULTIPAK Spółka Akcyjna",
        "companyProfileUrl":"https://pracodawcy.pracuj.pl/company/20379037/profile",
        "expirationDate":"2021-04-28T12:47:06.273",
        "salary":"",
        "employmentLevel":"Specjalista (Mid / Regular)" ,
        "offers": [
                {
            "offerId":500092126,
            "regionName":"kujawsko-pomorskie",
            "cities":["Małe Czyste (pow. chełmiński)"],
            "label":"Małe Czyste (pow. chełmiński)"}],

Above example of one element. So when I try to extract data like cities or regioName I receive an error. How can I make for loop from throughout two dictionaries and yield that data date to the new dictionary?

So, each offer has multiple "offers". What do you want your output to be? Do you want one entry per inner offer, so you potentially get multiple entries per outer offer? — Tim Roberts
– Tim Roberts, Commented Apr 13, 2021 at 22:11

Tim Roberts · Accepted Answer · 2021-04-13 22:14:19Z

1

You didn't make it clear what you want, but I'm guessing this is close:

    def parse(self, response):
        items = response.css("script:contains('window.__INITIAL_STATE__')::text").re_first(r"window\.__INITIAL_STATE__ =(.*);")
        for item in json.loads(items)['offers']:
            for offer in item['offers']:
                yield {
                    "title": item['jobTitle'],
                    "employer": item['employer'],
                    "country": item['countryName'],
                    "details_page": item['companyProfileUrl'],
                    "expiration_date": item['expirationDate'],
                    'salary': item['salary'],
                    'employmentLevel': item['employmentLevel'],
                    'offernumber': offer['offerId'],
                    'region': offer['regionName'],
                    'city': offer['cities'][0]
                }

answered Apr 13, 2021 at 22:14

Tim Roberts

55.3k4 gold badges29 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Kacper Over a year ago

That is exactly what I need to. Thank you

Collectives™ on Stack Overflow

How to extract data in one FOR loop from JSON multiple dictionary - Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related