Python BeautifulSoup Returning Empty List

Question

I'm trying to create a Python script to pull prices of Yugioh Card prices from tcgplayer.com using BeautifulSoup. When you search for a card on this website, it returns a page of search results with several prices from different sellers. My goal is to pull all of these prices. In the below example, I'm opening the search result for a card called "A" Cell Breeding Device:

import urllib2
from bs4 import BeautifulSoup
html = urllib2.open('http://shop.tcgplayer.com/productcatalog/product/show?newSearch=false&ProductType=All&IsProductNameExact=false&ProductName=%22A%22%20Cell%20Breeding%20Device')
soup = BeautifulSoup(html, 'lxml')
soup.find_all('span', {'class': 'scActualPrice largetext pricegreen'})

A few days ago, running the soup.find_all line correctly gave me the information I needed. However, running this now gives me an empty array []. I've searched pretty extensively about BeautifulSoup returning an empty array, but I'm not sure if any of them apply to me since it was working just fine a few days ago. Can someone help point me in the right direction? Thank you in advance!

Yohanes Gultom · Accepted Answer · 2017-02-27 01:54:26Z

4

You should use selenium to scrap using real browser:

from selenium import webdriver

driver = webdriver.Chrome('/path/to/chromedriver')
driver.get('http://shop.tcgplayer.com/productcatalog/product/show?newSearch=false&ProductType=All&IsProductNameExact=false&ProductName=%22A%22%20Cell%20Breeding%20Device')
prices = driver.find_elements_by_css_selector('.scActualPrice')
for element in prices:
    print(element.text)
driver.quit()

answered Feb 27, 2017 at 1:54

Yohanes Gultom

3,8522 gold badges28 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Pradeep Thimmireddy · Accepted Answer · 2017-02-27 01:12:55Z

0

This website uses a service called Incapsula. The website Developers configured Incapsula to prevent bots from accessing it's content.

I suggest you to contact their admins and request access or ask them for API.

answered Feb 27, 2017 at 1:12

Pradeep Thimmireddy

1064 bronze badges

3 Comments

Kevin Marume Over a year ago

Using selenium worked for me, but do you think it'll stop working in a few days as well?

Pradeep Thimmireddy Over a year ago

With selenium you are actually opening browser and doing all the operation, so it should be fine for now.But may be in future there are chances.

Pradeep Thimmireddy Over a year ago

And using selenium is not reliable

Collectives™ on Stack Overflow

Python BeautifulSoup Returning Empty List

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related