Scraping json response using scrapy

Question

JSON RESPONSE FROM WEBSITE I am new to python scrapy and json . I am trying to scrape json response from 78751 . But it is showing error . The code i used is

import scrapy
import json 
class BlackSpider(scrapy.Spider):
    name = 'black'
    start_urls = ['https://appworld.blackberry.com/cas/content/2360/reviews/2.17.2?page=1&pagesize=100&sortby=newest&callback=_content_2360_reviews_2_17_2&_=1499161778751']





    def parse(self, response):
         data = re.findall('(\{.+\})\);', response.body_as_unicode())
         a=json.loads(data[0])

         item = MyItem()
         item["Reviews"] = a["reviews"][4]["review"]           

         return item

The error it is showing is ValueError("No JSON object could be decoded")ERROR

It looks like your page is returning HTML content not JSON. Checking it via curl got me an html file that says the site is under maintenance. — Colwin
– Colwin, Commented Jul 4, 2017 at 11:58

Granitosaurus · Accepted Answer · 2017-07-04 13:13:15Z

1

The response you are getting is javascript function with some json in it:

_content_2360_reviews_2_17_2(\r\n{"some":"json"}]});\r\n

To extract the data from this you can use simple regex solution:

import re
import json
data = re.findall('(\{.+\})\);', response.body_as_unicode())
json.loads(data[0])

It trasnslates to: select everything between {} that ends with );

edit: results I'm getting with this:

{'platform': None,
 'reviews': [{'createdDate': '2017-07-04',
   'model': 'London',
   'nickname': 'aravind14-92362',
   'rating': 6,
   'review': 'Very bad ',
   'title': 'My WhatsApp no update '}],
 'totalReviews': 569909,
 'version': '2.17.2'}

edited Jul 4, 2017 at 13:13

answered Jul 4, 2017 at 12:01

Granitosaurus

21.6k6 gold badges64 silver badges88 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Thomas Over a year ago

To add: the format is called "JSONP". It is was often used (before CORS became mainstream) to get around the same-origin policy.

emon Over a year ago

I am new to it where to add the url of json response with regex.

Granitosaurus Over a year ago

@emon I'm sorry, could you rephrase your question?

emon Over a year ago

why it is showing error when i put item["Reviews"] = a["reviews"][4]["review"]

Granitosaurus Over a year ago

@emon, I'm still confused. Check out my edit, I can succesfully get review via data['reviews'][0]['review'].

|

Collectives™ on Stack Overflow

Scraping json response using scrapy

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related