Python-Can't Pull JSON Format from JSON Source

Question

I'm trying to scrape data from Verizon's buyback pricing site. I found the source of the information while going through "Net" requests in my browser. The site is in JSON format, but nothing I do will let me download that data https://www.verizonwireless.com/vzw/browse/tradein/ajax/deviceSearch.jsp?act=models&car=Verizon&man=Apple&siz=large

I can't remember everything I've tried, but here are the issues I'm having. Also, I'm not sure how to insert multiple code blocks.

import json,urllib,requests
res=urllib.request.urlopen(url)
data=json.loads(res)
TypeError: the JSON object must be str, not 'bytes'

import codecs
reader=codecs.getreader('utf-8')
obj=json.load(reader(res))
ValueError: Expecting value: line 1 column 1 (char 0)
#this value error happens with other similar attempts, such as....
res=requests.get(url)
res.json()#Same error Occurs

At this point I've researched many hours and can't find a solution. I'm assuming that the site is not formatted normally or I'm missing something obvious. I see the JSON requests/structure in my web developer tools.

Does anybody have any ideas or solutions for this? Please let me know if you have questions.

It's because Verizon is sending back an HTML page with JSON-looking stuff buried inside HTML tags, not a JSON-formatted string (which is what .json() takes as input). This answer should help you. — Akshat Mahajan
– Akshat Mahajan, Commented Mar 31, 2016 at 23:10

Robᵩ · Accepted Answer · 2016-03-31 23:29:50Z

0

You need to send a User-Agent HTTP header field. Try this program:

import requests

url='https://www.verizonwireless.com/vzw/browse/tradein/ajax/deviceSearch.jsp?act=models&car=Verizon&man=Apple&siz=large'
# Put your own contact info in next line
headers = {'User-agent':'MyBot/0.1 ([email protected])'}
r = requests.get(url, headers=headers)
print(r.json()['models'][0]['name'])

Result:

iPhone 6S

edited Mar 31, 2016 at 23:29

answered Mar 31, 2016 at 23:23

Robᵩ

170k20 gold badges251 silver badges323 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python-Can't Pull JSON Format from JSON Source

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related