why is my html parser not outputting wanted number

Question

my programming teacher made us program in python a calculator for calculating fuel consummation in L/100KM and i decided to go further and even have it calculate the price per 100km but heres the thing, i tried to make an html parser using beautifulsoup4 (bs4) so it finds the gas price for me and updates it if ever it changes on the website and i found the css selector for the number but im unsure if its wrong or what is wrong in the parser because when i run it, it returns "initial number: none" instead of the number specified by the css selector. heres the code for my parser:

import requests
from bs4 import BeautifulSoup
import time

# URL of the website to monitor
url = 'https://nbeub.ca/index.php?page=current-petroleum-prices-2'

# Function to fetch the number from the website
def fetch_number():
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    
    # Adjust the selector to find the specific number
    number = soup.select_one('body > table > tbody > tr:nth-child(5) > td > table > tbody > tr > td > table > tbody > tr > td:nth-child(3) > table > tbody > tr:nth-child(3) > td:nth-child(2)')
    return str(number)

# Main monitoring function
def monitor():
    last_number = fetch_number()
    print(f"Initial number: {last_number}")

    while True:
        time.sleep(2592000)  # Wait for 30 days before checking again
        current_number = fetch_number()
        
        if current_number != last_number:
            print(f"Number updated: {current_number}")
            last_number = current_number

# Start monitoring
monitor()

my terminal is being normal asking me for inputs but as it gets to parser it just prints initial number: none and then doesnt do the rest of my code

probably because soup.select_one returns None for some reason — Jean-François Fabre
– Jean-François Fabre ♦, Commented Feb 12 at 22:45
@Jean-FrançoisFabre Duh! The question is why it's not finding the selected element. I thought it might be because the page is generated dynamically, but it looks like the table is in the raw HTML. — Barmar
– Barmar, Commented Feb 12 at 22:49
Most of the tables don't have <tbody>. You need to remove those from the selectors. — Barmar
– Barmar, Commented Feb 12 at 22:54

Barmar · Accepted Answer · 2025-02-12 22:58:29Z

0

Most of the tables there don't have <tbody> elements, the <tr> is nested directly inside <table>. Instead of using all those direct child selectors, use descendant selectors without > tbody > so it will match either way.

number = soup.select_one('body > table tr:nth-child(5) > td > table tr > td > table tr > td:nth-child(3) > table tr:nth-child(3) > td:nth-child(2)')

answered Feb 12 at 22:58

Barmar

789k57 gold badges555 silver badges669 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

VXV Feb 12 at 23:04

okay so it did give me the number but instead of only giving me the number only it gave me this "initial number: <td>171.6</td>" how would i only get the number 171.6 and not the whole td stuff

Barmar Feb 12 at 23:06

Use number.string to get the string contents of an element.

VXV Feb 12 at 23:22

i changed this part and it worked def fetch_number(): response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') # Adjust the selector to find the specific number number = soup.select_one('body > table tr:nth-child(5) > td > table tr > td > table tr > td:nth-child(3) > table tr:nth-child(3) > td:nth-child(2)') # Extract the text content from the 'td' element return number.text.strip() if number else 'Element not found'

VXV Feb 12 at 23:23

thank you very much and have a good day or night!

Collectives™ on Stack Overflow

why is my html parser not outputting wanted number

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related