Python Selenium Scraping missing image from url

Question

Hello all my task is to scrap source urls from offer link like this one.

But when I try to get the elements like this (Note that I make 2 requests of the url to get the cookies, because the first time is redirecting me to the main page):

driver = webdriver.Firefox(executable_path="C:\\selenium-drivers\\geckodriver.exe")
driver.get("http://www.kmart.com/joe-boxer-men-s-pajama-shirt-pants-plaid/p-046VA92629712P")
driver.get("http://www.kmart.com/joe-boxer-men-s-pajama-shirt-pants-plaid/p-046VA92629712P")
img_element = driver.find_elements_by_class_name("main-image")

No elements are found and when i try to search them in the source code in the browser with Ctrl+U they are missing. Why is this happening? Is anyone who can tell me how to get these images.

alecxe · Accepted Answer · 2016-12-24 16:23:15Z

2

You just need to tell selenium to be patient and wait for element's visibility:

from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

# driver definition here

driver.get("http://www.kmart.com/joe-boxer-men-s-pajama-shirt-pants-plaid/p-046VA92629712P")

wait = WebDriverWait(driver, 10)

# get the main image element
img_element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'main-image')))
print(img_element.get_attribute("alt"))

driver.close()

For the demonstration purposes, it prints the alt attribute of the image, which is:

Joe Boxer Men's Pajama Shirt & Pants - Plaid

answered Dec 24, 2016 at 16:23

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Psyke Over a year ago

Oh my god you are my savior! Thank you i didn't know that there is function for waiting the element. Merry Christmas!

Hisagr · Accepted Answer · 2016-12-24 16:26:48Z

0

Or you can just find by xpath and then get image url

>>> driver.get('http://www.kmart.com/joe-boxer-men-s-pajama-shirt-pants-plaid/p-046VA92629712P')
>>> s = driver.find_element_by_xpath('//*[@id="overview"]/div[1]/img')
>>> s.get_attribute('src') 
'http://c.shld.net/rpx/i/s/i/spin/-122/prod_2253990712?hei=624&wid=624&op_sharpen=1'

answered Dec 24, 2016 at 16:26

Hisagr

6196 silver badges13 bronze badges

Collectives™ on Stack Overflow

Python Selenium Scraping missing image from url

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related