Problem with scraping in Selenium. It does not scrape

Question

I would like to get the number of rounds (14, 15, 16, 17, etc.), then just scrape the numbers, no ".ROUND". I wrote a basic version and a better version, but they don't work. I don't get errors, but I get >>>> (without any text). How to do?

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 20)

Values_PremierLeague = []

driver.get("https://www.betexplorer.com/soccer/england/premier-league/fixtures/")

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "th.h-text-left")))

for PremierLeague in driver.find_elements(By.CLASS_NAME, "th.h-text-left"):
    PremierLeague_text = PremierLeague.text
    Values_PremierLeague.append(tuple([PremierLeague_text]))
    print(PremierLeague_text)
driver.close

or

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import re

wait = WebDriverWait(driver, 20)

Values_PremierLeague = []

driver.get("https://www.betexplorer.com/soccer/england/premier-league/fixtures/")

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".table-main__datetime")))

for PremierLeague in driver.find_elements(By.XPATH, "//*[@class='h-text-left' and contains(text(),'Round')]"):
    PremierLeague_text = PremierLeague.text
    number = re.findall(r'\d+', s)
    Values_PremierLeague.append(tuple([number]))
    print(number)
driver.close

UPDATE

CREATE TABLE "BASE_Giornate" (
    "ID_Round"  INTEGER,
    "Number_Round"  INTEGER,
    "Id_Tournment"  INTEGER,
    PRIMARY KEY("ID_Giornata" AUTOINCREMENT)
);

and the code already working

sqlite_insert_query_PremierLeague = 'INSERT INTO BASE_Giornate (Number_Round) VALUES (?);'
cursor.executemany(sqlite_insert_query_PremierLeague, Values_PremierLeague)
count_squadre_PremierLeague = cursor.rowcount
con.commit()

So the first one you accidentally used class name instead of css selector. s should be premier_League_text in the second. — Arundeep Chohan
– Arundeep Chohan, Commented Nov 23, 2021 at 20:09
@ArundeepChohan Yes. The first one gets the class name wrong and in any case doesn't have the setting to just scrape the number by eliminating ".ROUND". The second one should work, but there is something wrong. Can you help me? Thanks — Erling Olsen
– Erling Olsen, Commented Nov 23, 2021 at 20:16

Arundeep Chohan · Accepted Answer · 2021-11-23 20:16:00Z

1

driver.get("https://www.betexplorer.com/soccer/england/premier-league/fixtures/")

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".table-main__datetime")))

for PremierLeague in driver.find_elements(By.XPATH, "//*[@class='h-text-left' and contains(text(),'Round')]"):
    PremierLeague_text = PremierLeague.text
    number = re.findall(r'\d+', PremierLeague_text)
    Values_PremierLeague.append(tuple([number]))
    print(number)

Not sure what you meant by numbers also not sure why you want tuples. But you had an issue where you never used PremierLeague_text

Outputs:

['13']
['14']
['15']
['16']
['17']
['18']
['19']
['20']
['21']
['22']
['23']
['24']
['25']
['26']
['27']
['28']
['29']
['30']
['31']
['32']
['33']
['34']
['35']
['36']
['37']
['38']

answered Nov 23, 2021 at 20:16

Arundeep Chohan

9,9995 gold badges17 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Erling Olsen Over a year ago

That's exactly what I wanted to achieve. Thanks. By numbers I meant this. As for the tuples, I have to save these numbers in a column of a database, and I have to scrape for other different tournaments as well. At this point you make me doubt: are tuples not necessary?

Arundeep Chohan Over a year ago

If you don't have multiple values in it it's best to just do an array.

Erling Olsen Over a year ago

Ah OK. It is only 1 value. What should I do with the array? (if you want you can not answer). I'll vote for your answer anyway. Thanks

Erling Olsen Over a year ago

Excuse me. I didn't know that if it was closed you couldn't answer. I reopened it momentarily (then I will close it again). Excuse me and thank you

Arundeep Chohan Over a year ago

Oh not sure how your database is set up so cant really comment

|

Collectives™ on Stack Overflow

Problem with scraping in Selenium. It does not scrape

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related