Web scraping on Python

Question

Hey so I need to web scrape this website (don't use beautiful soup) to get the current temperature and I am having trouble. This is what I have so far but I keep getting either a number that isn't the temperature or -1. So any help is greatly appreciated.

def assign4(city_name):
import urllib.request

if city_name == "St. Catharines":
    connection = urllib.request.urlopen("https://weather.gc.ca/city/pages/on-107_metric_e.html")
    condition = str(connection.read(), "utf-8")
    connection.close()

    weather_condition = condition.find("Temperature:</dt>")
    if weather_condition != -1:
        weather_condition_end = condition.find("</dd>",weather_condition)
        if weather_condition_end != -1:
            weather_start = condition.find("metric-hide",0,weather_condition_end)
            if weather_start != -1:
                print(f"Weather Conditions in St. Catharines is {weather_start}")
            else:
                print("'weather_start' not working")
        else:
            print("'weather_condition_end' not working")
    else:
        print("'weather_condition' not working")
assign4("St. Catharines")

Oh okay I see what you mean alex.... I will try different things. — madison grant
– madison grant, Commented Nov 16, 2020 at 16:13

Abhishek Rai · Accepted Answer · 2020-11-16 15:52:50Z

1

There should be a space in between St. and Catherines in the last line. That is where it's wrong.

if city_name == "St. Catharines": assign4("St.Catharines")

When you are calling the function your are not adding the space.

answered Nov 16, 2020 at 15:52

Abhishek Rai

2,2474 gold badges26 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

madison grant Over a year ago

No the space is just that I forgot to put it in when I type that out.

Abhishek Rai Over a year ago

@madisongrant Try and add "expected output" in the question ..so people can know what exactly you want the code to do. As you advance, this would help in getting the right answer sooner. Happy coding.

Diogo Silva · Accepted Answer · 2020-11-16 16:03:42Z

0

You can simplify your code with lxml and requests

import requests
from lxml import html
def assign4(city_name):
  if city_name == "St.Catharines":
      # Get the html page
      resp=requests.get("https://weather.gc.ca/city/pages/on-107_metric_e.html")
      # Build html tree
      html_tree=html.fromstring(resp.text)
      # Get temperature
      temperature=html_tree.xpath("//dd[@class='mrgn-bttm-0 wxo-metric-hide'][(parent::dl[@class='dl-horizontal wxo-conds-col2'])]//text()")[0].replace("Â", "")
      # Print temperature
      print(f"Temperature in {city_name} is {temperature}C")
  

assign4("St.Catharines")

Outputs:

>>> Temperature in St.Catharines is 4.8°C

answered Nov 16, 2020 at 16:03

Diogo Silva

3403 silver badges15 bronze badges

7 Comments

Abhishek Rai Over a year ago

I don't think rewriting the whole program differently is what SO is for. :) ..Kind of skips the learning curve for the OP. We stick to answering the specific issue OP raises.

Diogo Silva Over a year ago

@AbrarAhmed he didn't ask for a specific solution. "Any help" to get the temperature. The only restriction was not using beautifulsoup, which I didn't. But I get your point.

Diogo Silva Over a year ago

@madisongrant If you find it suitable for your problem, please mark it as the solution :)

madison grant Over a year ago

Done so already I just have to make sure it is okay to use but I understand what was done basically done a path where the temperature is between the class and the parent and I greatly appreciated

Diogo Silva Over a year ago

@madisongrant Yes. If you need a deeper understanding of how it works, search about xpath.

|

Collectives™ on Stack Overflow

Web scraping on Python

2 Answers 2

2 Comments

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related