Problem extracting data from json format to csv using BeautifulSoup 3

Question

I am trying to export Data from json format in CSV but getting no results. below is the code

import requests
from bs4 import BeautifulSoup
import json
import re

url = "https://www.daraz.pk/catalog/?q=dell&_keyori=ss&from=input&spm=a2a0e.home.search.go.35e34937qjElRf"



page = requests.get(url)

print(page.status_code)
print(page.text)
soup = BeautifulSoup(page.text, 'html.parser')
print(soup.prettify())

alpha = soup.find_all('script',{'type':'application/ld+json'})
jsonObj =`json.loads(alpha[1].text)`

for item in jsonObj['itemListElement']:
    name = item['name']
    price = item['offers']['price']
    currency = item['offers']['priceCurrency']
    availability = item['offers']['availability'].split('/')[-1]
    availability = [s for s in re.split("([A-Z][^A-Z]*)", availability) if s]
    availability = ' '.join(availability)

    print('Availability: %s  Price: %0.2f %s   Name: %s' %(availability,float(price), currency,name))

Here is the code I am trying to export Data in CSV but not getting results in CSV

Create a file to write to, add headers row

outfile = open('products.csv','w', newline='')
writer = csv.writer(outfile)
writer.writerow(["name", "offers", "price", "priceCurrency", "availability" ])
outfile.close()
alpha = soup.find_all('script',{'type':'application/ld+json'})

jsonObj = json.loads(alpha[1].text)

for item in jsonObj['itemListElement']:
    name = item['name']
    price = item['offers']['price']
    currency = item['offers']['priceCurrency']
    availability = item['offers']['availability'].split('/')[-1]
    availability = [s for s in re.split("([A-Z][^A-Z]*)", availability) if s]
    availability = ' '.join(availability)

there is no script type='application/ld+json' tag on this page — chitown88
– chitown88, Commented Jan 7, 2019 at 14:11
All of the product Data i.e. Name , price, currency, availability are in the script. — user10878963
– user10878963, Commented Jan 7, 2019 at 14:16

ewwink · Accepted Answer · 2019-01-07 15:10:07Z

1

you get not result because not writing the CSV in the loop

outfile = open('products.csv','w', newline='')
writer = csv.writer(outfile)
writer.writerow(["name", "type", "price", "priceCurrency", "availability" ])

alpha = soup.find_all('script',{'type':'application/ld+json'})

jsonObj = json.loads(alpha[1].text)

for item in jsonObj['itemListElement']:
    name = item['name']
    type = item['@type']
    price = item['offers']['price']
    currency = item['offers']['priceCurrency']
    availability = item['offers']['availability'].split('/')[-1]
    # forgot this?
    writer.writerow([name, type, price, currency, availability ])

# and close the CSV here
outfile.close()

answered Jan 7, 2019 at 15:10

ewwink

19.3k2 gold badges49 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

chitown88 · Accepted Answer · 2019-01-07 14:44:09Z

0

I personally am a fan of Pandas to write a csv. Some might say its extensive. But it works.

import requests
from bs4 import BeautifulSoup
import json
import re
import pandas as pd

url = "https://www.daraz.pk/catalog/?q=dell&_keyori=ss&from=input&spm=a2a0e.home.search.go.35e34937qjElRf"



page = requests.get(url)

#print(page.status_code)
#print(page.text)
soup = BeautifulSoup(page.text, 'html.parser')
#(soup.prettify())

alpha = soup.find_all('script',{'type':'application/ld+json'})
jsonObj = json.loads(alpha[1].text)


results = pd.DataFrame()
for item in jsonObj['itemListElement']:
    name = item['name']
    price = item['offers']['price']
    currency = item['offers']['priceCurrency']
    availability = item['offers']['availability'].split('/')[-1]
    availability = [s for s in re.split("([A-Z][^A-Z]*)", availability) if s]
    availability = ' '.join(availability)

    row = [name,price,currency,availability]
    temp_df = pd.DataFrame([row], columns = ['name','price','currency','availability'])

    results = results.append(temp_df)

results.to_csv('products.csv', index=False)

answered Jan 7, 2019 at 14:44

chitown88

29.1k6 gold badges34 silver badges67 bronze badges

1 Comment

user10878963 Over a year ago

Pandas Library is extensive and your solution works as well :)

Collectives™ on Stack Overflow

Problem extracting data from json format to csv using BeautifulSoup 3

Create a file to write to, add headers row

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Create a file to write to, add headers row

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related