Timeout exception error scraping with selenium

Question

im trying to scrape table based on dates from this site https://www.bi.go.id/id/statistik/informasi-kurs/transaksi-bi/Default.aspx with this code

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import re
import time
from bs4 import BeautifulSoup
# Import webdriver to initialise a browser
from selenium.webdriver import ActionChains
import csv
import pandas as pd
import requests
driver = webdriver.Chrome()
driver.get("https://www.bi.go.id/id/statistik/informasi-kurs/transaksi-bi/Default.aspx")

wait = WebDriverWait(driver, 10)

# click "usd"



book = wait.until(
    EC.element_to_be_clickable((By.XPATH,' //*[@id="selectPeriod"]/option[range]'))
)
book.click()

book1 = wait.until(
    EC.element_to_be_clickable((By.XPATH,' //*[@id="ctl00_PlaceHolderMain_g_6c89d4ad_107f_437d_bd54_8fda17b556bf_ctl00_ddlmatauang1"]/option[USD]'))
)
book1.click()



start_date = driver.find_element_by_id("ctl00_PlaceHolderMain_g_6c89d4ad_107f_437d_bd54_8fda17b556bf_ctl00_txtFrom")
start_date.send_keys("20-Nov-15")
end_date = driver.find_element_by_id("ctl00_PlaceHolderMain_g_6c89d4ad_107f_437d_bd54_8fda17b556bf_ctl00_txtTo")
end_date.send_keys("20-Nov-20")
submit_button = driver.find_element_by_id("ctl00_PlaceHolderMain_g_6c89d4ad_107f_437d_bd54_8fda17b556bf_ctl00_btnSearch1").click()

but instead i got this error and i dont know why.can anybody help me what goes wrong?

--------------------------------------------------------------------------- TimeoutException Traceback (most recent call last) in 9 10 ---> 11 book = wait.until( 12 EC.element_to_be_clickable((By.XPATH,' //*[@id="selectPeriod"]/option[range]')) 13 )

~\anaconda3\lib\site-packages\selenium\webdriver\support\wait.py in until(self, method, message) 78 if time.time() > end_time: 79 break ---> 80 raise TimeoutException(message, screen, stacktrace) 81 82 def until_not(self, method, message=''):

TimeoutException: Message:

"Selenium will wait for a maximum of 10 seconds for an element matching the given criteria to be found. If no element is found in that time, a TimeoutException is thrown." Try increasing the wait, or make sure the xpath is correct? — goalie1998
– goalie1998, Commented Jan 15, 2021 at 2:02
hm the results still same..and the xpath im sure its right tho.. — adinda aulia
– adinda aulia, Commented Jan 15, 2021 at 2:11
I've never used an option in a wait for clickable, so I'm not sure if it would work or you need to separate it and click it separately — goalie1998
– goalie1998, Commented Jan 15, 2021 at 2:17
If the results are still the same I recommend trying to go through website's html, and try and piece together how you can get there manually by navigating with BeautifulSoup once you use get page source. That's how I used to do it. It does take quite a while to figure out the structure, before you can automate the process in a for loop. — AlphabetsAlphabets
– AlphabetsAlphabets, Commented Jan 15, 2021 at 2:21

Arundeep Chohan · Accepted Answer · 2021-01-15 03:50:38Z

0

This should work since it's a Select tag.

driver.get("https://www.bi.go.id/id/statistik/informasi-kurs/transaksi-bi/Default.aspx")
wait = WebDriverWait(driver, 10)
book = wait.until(EC.element_to_be_clickable((By.ID,"selectPeriod")))
sel = Select(book)
sel.select_by_value("range")

Import

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select

answered Jan 15, 2021 at 3:50

Arundeep Chohan

9,9895 gold badges17 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

adinda aulia Over a year ago

thank you ! but i got 1 more problem,the submit button wont work! submit_button=driver.find_element_by_id("ctl00_PlaceHolderMain_g_6c89d4ad_107f_437d_bd54_8fda17b556bf_ctl00_btnSearch1").click() can you help me?

Arundeep Chohan Over a year ago

The id seems dynamic which means it'll change.//input[@value='Search'] Use the following xpath instead.

Collectives™ on Stack Overflow

Timeout exception error scraping with selenium

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related