Selenium Unable to locate element (Python) WebScraping

Question

I'm trying to scrape a real estate website for listings. It has an aspx form that has to filled out before submission.

http://www.cbre.us/PropertyListings/Pages/Properties-for-Sale.aspx

All I care about is multifamily properties in Oregon, however. So this was my first attempt:

driver = webdriver.Firefox()

driver.get("http://www.cbre.us/PropertyListings/Pages/Properties-for-Sale.aspx")

#Searching for multifamily residences
selectPropertyType = driver.find_element_by_id("ForSalePropertyType")
selectPropertyType.select_by_value("70")

#In the state of Oregon
selectState = driver.find_element_by_id("ForSaleState_ListBox1")
selectState.select_by_value("OR")

#Submit form
submitBtn = driver.find_element_by_id("ForSaleLooplinkSubmit")
submitBtn.click()

#Wait for results to load
WebDriverWait(driver, 5)

When I run this script it gives an error "Can't locate element "ForSalePropertyType". What am I doing wrong here? Thanks in advance.

alecxe · Accepted Answer · 2015-09-17 17:38:17Z

5

This element located inside an iframe. You have to switch to it's context:

driver.switch_to.frame("ctl00_PlaceHolderMain_IFrameContent_IFrameContent")

# searching for multifamily residences
selectPropertyType = driver.find_element_by_id("ForSalePropertyType")
selectPropertyType.select_by_value("70")

To get back to default context:

driver.switch_to.default_content()

As a side note, be aware of the policies listed in the Disclaimer / Terms of use, specifically:

You agree that you will not: (a) impersonate any person or entity or misrepresent your affiliation with any other person or entity; (b) engage in spamming, flooding, harvesting of e-mail addresses or other personal information, spidering, screen scraping, database scraping, or any other activity with the purpose of obtaining lists of users or any other information, including specifically, property listings available through the site; (c) send chain letters or pyramid schemes via the site; or (d) attempt to gain unauthorized access to other computer systems through the site. You agree that you will not use the site in any manner that could damage, disable, overburden, or impair the site or interfere with any other party's use and enjoyment of the site.

answered Sep 17, 2015 at 17:38

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

John Doe Over a year ago

Duly noted. Thank you for your help.

Collectives™ on Stack Overflow

Selenium Unable to locate element (Python) WebScraping

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related