Web Scraping with Selenium and BeautifulSoup returns Empty list

Question

I am trying to search through websites to determine with certain items are in stock. I want this to search the site, find out if the item is "In stock" or not and notify the user of the found information. I can get most of the code to run but this continually returns empty lists. (I tried in BeautifulSoup4 as well as Selenium with the same outcome. Both versions below).What am I missing? Any help would be greatly appreciated!

#selenium version
import os
import string
import subprocess
import traceback
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

def webscan():
    driver = webdriver.Chrome('/usr/lib/python3/dist-packages/selenium/webdriver/chrome/chromedriver')
    #driver.maximize_window()
    driver.get("https://www.amazon.com/GIGABYTE-GeForce-Windforce-Graphics-GV-N207SWF3OC-8GD/dp/B07WN6RVHH/ref=sr_1_11?crid=OOF3E7T9W54&dchild=1&keywords=graphics+card&qid=1630118746&sprefix=graphi%2Caps%2C190&sr=8-11")

    #wait for the page to load fully

    time.sleep(5)

    status = driver.find_element_by_class('a-size-medium a-color-state')
    ps5_avail = status


    print(ps5_avail)
    if ps5_avail == "Currently unavailable":
        print("Sorry. Still not in stock...")
    elif ps5_avail == "Temporarily Out of Stock":
        print("Sorry. Still not in stock...")
    elif ps5_avail == "In stock soon.":
        print("sorr. Still not in stock...")
    else:
        print("Go NOW!!!!!")


    driver.close()


user_input = (input("Want a PS5?"))

if user_input == 'yes':
    webscan()
    print('scan is done')


####################################################################
#beautifulsoup version

import requests
from bs4 import BeautifulSoup

url = 'https://www.amazon.com/GIGABYTE-GeForce-Windforce-Graphics-GV-N207SWF3OC-8GD/dp/B07WN6RVHH/ref=sr_1_11?crid=OOF3E7T9W54&dchild=1&keywords=graphics+card&qid=1630118746&sprefix=graphi%2Caps%2C190&sr=8-11'
res = requests.get(url)
html_page = res.content

soup = BeautifulSoup(html_page, 'html.parser')

text = soup.find_all(string='In Stock')

if text == True:
    print("Out of Stock")
else:
    print("Get it!")

find_element_by_class_name accepts a single class name. You are providing two names. You'll have to filter twice. And there is no find_element_by_class as far as I (and Google) know. — Tim Roberts
– Tim Roberts, Commented Aug 28, 2021 at 3:44
Thanks for the reply. Ok. that has the same error. It continues to return empty lists.... — EveryDayDev
– EveryDayDev, Commented Aug 28, 2021 at 3:51

cruisepandey · Accepted Answer · 2021-08-28 04:10:27Z

1

You should use this css selector instead :

Code :

driver.get("https://www.amazon.com/GIGABYTE-GeForce-Windforce-Graphics-GV-N207SWF3OC-8GD/dp/B07WN6RVHH/ref=sr_1_11?crid=OOF3E7T9W54&dchild=1&keywords=graphics+card&qid=1630118746&sprefix=graphi%2Caps%2C190&sr=8-11")

#wait for the page to load fully

time.sleep(5)

lngth = len(driver.find_elements(By.CSS_SELECTOR, "span.a-size-medium.a-color-state"))
print("value of lngth", lngth)

try:
    if len(driver.find_elements(By.CSS_SELECTOR, "span.a-size-medium.a-color-state")) > 0:
        print("Inside if")
        status = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div#availability span"))).text
        print(status)
    else:
        print("Looks like element was not found.")
except:
    print("Something went wrong")
    pass

Imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Aug 28, 2021 at 4:10

answered Aug 28, 2021 at 3:58

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

EveryDayDev Over a year ago

Thanks for the response. I must be missing something because this returned an empty list as well.

cruisepandey Over a year ago

@willballentine : try the updated code above, and let me know the output.

demouser123 Over a year ago

@cruisepandey The output is value of lngth 0 Looks like element was not found

cruisepandey Over a year ago

then we have an issue here div#availability span, also I think availability will only work if the element is available on the page. Is it showing you available ?

demouser123 Over a year ago

Yeah it's due to Gelocation. If I change the pin code to 90011 (Los Angeles), I'm able to see that too.

|

Collectives™ on Stack Overflow

Web Scraping with Selenium and BeautifulSoup returns Empty list

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related