getting HTTPError while trying to parse info from webpage

Question

I just started to learn Python and faced with this problem. Trued to parse price from amazon and print it to console.

This is my code:

import requests, bs4

def getAmazonPrice(productUrl):
    res = requests.get(productUrl)
    res.raise_for_status()

    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    elems = soup.select('#addToCart > a > h5 > div > div.a-column.a-span7.a-text-right.a-span-last > span.a-size-medium.a-color-price.header-price')
    return elems[0].text.strip()


price = getAmazonPrice('http://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/ref=sr_1_2?ie=UTF8&qid=1460386052&sr=8-2&keywords=python+book')
print('The price is ' + price)

Error Message:

Traceback (most recent call last): File "D:/Code/Python/Basic/webBrowser-Module.py", line 37, in price = getAmazonPrice('http://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/ref=sr_1_2?ie=UTF8&qid=1460386052&sr=8-2&keywords=python+book') File "D:/Code/Python/Basic/webBrowser-Module.py", line 30, in getAmazonPrice res.raise_for_status() File "C:\Python33\lib\requests\models.py", line 844, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: http://www.amazon.com/Automate-Boring-Stuff-Python-Programming/dp/1593275994/ref=sr_1_2?ie=UTF8&qid=1460386052&sr=8-2&keywords=python+book

Process finished with exit code 1

I get the same error message. __requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url __ even after doing as suggested — Moses
– Moses, Commented May 26, 2021 at 10:51

alecxe · Accepted Answer · 2016-04-11 16:23:03Z

7

Pretending to be a real browser by providing a User-Agent header would fix this particular issue:

res = requests.get(productUrl, headers={
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
})

You would also need to tweak your CSS selector. For instance, .header-price would get you all of the prices on the page (non-prime and prime in this case).

answered Apr 11, 2016 at 16:23

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

getting HTTPError while trying to parse info from webpage

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related