I've tried to get data form a web page using python and selenium:
from selenium import webdriver
driver=webdriver.Firefox()
driver.get('https://covid19.min-saude.pt/ponto-de-situacao-atual-em-portugal/')
'RECUPERADOS' in driver.page_source
False
String 'RECUPERADOS' is present in browser window opened by driver, but not in page_source. What prevents selenium form rendering page html? Is there any way to get data from page like the one above?
The element text you are searching for is inside an iframe. You need to switch to iframe first and then wait for element to be visible and then capture the page_source.
Use WebDriverWait() and wait for frame_to_be_available_and_switch_to_it()
Use WebDriverWait() and wait for visibility_of_element_located()
driver.get('https://covid19.min-saude.pt/ponto-de-situacao-atual-em-portugal/')
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//*[@id='main']//iframe[1]")))
WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.XPATH,"//*[name()='text'][text()='RECUPERADOS']")))
if 'RECUPERADOS' in driver.page_source:
print("YES")
else:
print("NO")
Import below libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
To jump out from iframe you need to use below code.
driver.switch_to.default_content()
