scrape and clicking form with selenium

Question

I want to scrape the simulation "Richiedi il tuo prestito online" of a form of this website:

https://www.findomestic.it/

I tried this:

driver = webdriver.PhantomJS()
driver.get("https://www.findomestic.it/")
raison = driver.find_element_by_xpath("//a[@href='javascript:void(0);']")
montant = driver.find_element_by_id('findomestic_simulatore_javascript_importo')
submitButton = driver.find_element_by_id('findomestic_simulatore_javascript_calcola')
actions = ActionChains(driver).click(raison).send_keys('AUTO NUOVA').click(montant).send_keys('2000').send_keys(Keys.RETURN)
actions.perform()
print(driver.find_element_by_tag_name('body').text)
print(driver)
driver.close()

My expected output is the result when you click on the form. I want to find the results of the web page with the interest rate and the amount.

expected outpout But the print is not correct:

The result is just sends me back the session:

<selenium.webdriver.phantomjs.webdriver.WebDriver(session="c4070330-18b2-11e9-81cf-2dbe9dae6b83")>

What is your expected output? Do you want to print simply all the text displayed on page? — Andersson
– Andersson, Commented Jan 15, 2019 at 13:06
My expected output is the result when you click on the form. I want to find the results of the web page with the interest rate and the amount. — user10732315
– user10732315, Commented Jan 15, 2019 at 13:08

Andersson · Accepted Answer · 2019-01-15 14:29:24Z

4

print(driver) returns the string representation of WebDriver instance (driver.__str__()) and it's normal behavior

print(driver.find_element_by_tag_name('body').text) returns nothing as after you submit the form page body is empty - it contains only scripts that are not displayed on page and so text property returns empty string as expected

You need to wait for results to appear on page:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get("https://www.findomestic.it/")

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.select.bh-option"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, 'AUTO NUOVA'))).click()
driver.find_element_by_id("findomestic_simulatore_javascript_importo").send_keys("2000")
driver.find_element_by_id('findomestic_simulatore_javascript_calcola').click()

for item in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'ul.fd-siff-element > li')))[1:]:
    print(item.text.split('\n')[:-1])

The output should be

['56,20 € PER', '42 MESI', '9,54 % TAN FISSO', '9,97 % TAEG FISSO']
['64,10 € PER', '36 MESI', '9,53 % TAN FISSO', '9,96 % TAEG FISSO']
['75,20 € PER', '30 MESI', '9,54 % TAN FISSO', '9,97 % TAEG FISSO']
['91,80 € PER', '24 MESI', '9,46 % TAN FISSO', '9,89 % TAEG FISSO']
['119,70 € PER', '18 MESI', '9,54 % TAN FISSO', '9,97 % TAEG FISSO']

edited Jan 15, 2019 at 14:29

answered Jan 15, 2019 at 13:15

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

18 Comments

user10732315 Over a year ago

Thank you for your answer, but I have an error when I add your code to mine

Andersson Over a year ago

@Turing , which one? share exception

user10732315 Over a year ago

TimeoutException: Message:

Andersson Over a year ago

@Turing , did you successfully pass the main page and navigate to result page?

user10732315 Over a year ago

Yes, I suppose. Can you pass me your entire code for me to test?

|

DisappointedByUnaccountableMod · Accepted Answer · 2021-12-13 09:31:42Z

1

To scrape the simulation "Richiedi il tuo prestito online" of a form of the website you can use the following solution:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.PhantomJS(executable_path=r'C:\\Utility\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe')
driver.get("https://www.findomestic.it/ ")
driver.maximize_window()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.select.bh-option"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//ul[@class='select-list bh-options_list']//li/a[text()='Auto nuova']"))).click()
driver.find_element_by_css_selector("input#findomestic_simulatore_javascript_importo").send_keys("2000")
driver.find_element_by_css_selector("input#findomestic_simulatore_javascript_calcola").submit()

edited Dec 13, 2021 at 9:31

DisappointedByUnaccountableMod

6,8444 gold badges21 silver badges23 bronze badges

answered Jan 15, 2019 at 13:57

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

6 Comments

user10732315 Over a year ago

It work, bu how can I print the result of price and other ? :)

undetected Selenium Over a year ago

As per your code trials you tried to send_keys('AUTO NUOVA') and send_keys('2000') and your expected output was this. Hence was my answer. If your requirement have changed please raise a new question :)

user10732315 Over a year ago

But how could I print the result?

undetected Selenium Over a year ago

You have mentioned your expected output as this which you can easily get now through driver.save_screenshot("filename"). I am sure you tried that hence you commented It work :) Are you looking for something else?

user10732315 Over a year ago

The image is all gray, nothing is displayed! When I say when the code works, there is no error.

|

Collectives™ on Stack Overflow

scrape and clicking form with selenium

2 Answers 2

18 Comments

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

18 Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related