Unable to retrieve temperature value from HTML using beautifulsoup module python

Question

I am using BeautifulSoup4 to parse this HTML (view-source:https://weather.com/en-IN/weather/today/l/17.39,78.49) and I'm trying to retrieve temperature value. But, the value might be stored in attribute obs.temperature. I can see the temperature value "24" in the page Inspect window, but the same can't be found directly in view page source. Below is the snapshot of this problem.

From view source:

<div class="today_nowcard-temp">
<span data-gm-wx-temperature="::todayWxcardVm.obs.temperature" data-text-to-replace="{{ '[[ obs.temperature ]]'.indexOf('\[\[') !== -1 ? '--' : '[[ obs.temperature ]]' }}">[[ obs.temperature ]]</span>
</div>

From Inspect window

<div class="today_nowcard-temp">
<span data-gm-wx-temperature="::todayWxcardVm.obs.temperature" data-text-to-replace="--"><!-- ngIf: tempPrefix --> <!-- ngIf: hasValue --><span data-ng-if="hasValue" class="dir-ltr" data-ng-bind="temp | safeDisplay">24</span><!-- end ngIf: hasValue --><!-- ngIf: hasValue --><sup data-ng-if="hasValue" class="deg dir-ltr">°</sup><!-- end ngIf: hasValue --><!-- ngIf: showTempUnit -->
<!-- ngIf: !hasValue --></span>
</div>

Please let me know how to get temperature value.

Below is my Python code:

import bs4, requests
web = requests.get("https://weather.com/en-IN/weather/today/l/17.39,78.49")
websoup = bs4.BeautifulSoup(web.text, "html.parser")
print(type(websoup))
webtemperature = websoup.select("div .today_nowcard-temp span")
print(webtemperature)
print("from weather.com: "+webtemperature[0].getText()+ "degree celsius\n")

Output:

<class 'bs4.BeautifulSoup'>
[<span data-gm-wx-temperature="::todayWxcardVm.obs.temperature" data-text-to-replace="{{ '[[ obs.temperature ]]'.indexOf('\\[\\[') !== -1 ? '--' : '[[ obs.temperature ]]' }}">[[ obs.temperature ]]</span>]

from weather.com: [[ obs.temperature ]]degree celsius

if it's just to get the api for a city, why don't you use openweathermap.org/api — min2bro
– min2bro, Commented Jul 3, 2016 at 5:50
This page get the temperature with a AJAX request, so beautifulsoup will not work. You will acheive this with Selenium or you can monitor the network traffic and do the same request as this website — Tales Pádua
– Tales Pádua, Commented Jul 4, 2016 at 16:45
I have started learning web scraping using python and found one beautifulsoup article. So, I have used same method what the author has described, I have no idea about Selenium or any other tools. Will read further about Selenium and will try if it can help in fetching the temperature. — Raj
– Raj, Commented Jul 11, 2016 at 9:35

Dennis Shtatnov · Accepted Answer · 2016-07-03 05:44:00Z

1

BS4 is working as expected, as [[ obs.temperature ]] is part of the html templating language they are using the build the page after it is loaded, so you can't grab it straight from the HTML as the full page must be executed. Or:

When you load the page, the temperatures appear to load afterwards as there is a loading spinner. See the network inspector:

The weather.com website uses their internal APIs for retrieving the weather data as JSON. You will need to replicate their request.

If you are using this several times, I'd recommend looking into using an official weather API for developers: just google weather api

answered Jul 3, 2016 at 5:44

Dennis Shtatnov

1,38311 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Unable to retrieve temperature value from HTML using beautifulsoup module python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related