unable to parse a webpage using python

Question

I am trying to parse below webpage to get name of stocks hitting now all time high or low in the exchange.

https://www.bseindia.com/markets/equity/EQReports/HighLow.html?Flag=H#

however, when i download the webpage using beautiful soup and check the data i do not find the stock name or price mentioned in the webpage. I wish to write a function to download the stocks that hit a new all time high each day please help what am i missing?

Please show what you’ve tried and explain what the specific problem was. — mkrieger1
– mkrieger1, Commented Feb 23, 2019 at 13:13

gmds · Accepted Answer · 2019-02-23 13:17:37Z

2

Part of the HTML on the page is generated dynamically by JavaScript. You are most likely using the requests library, which cannot handle HTML generated in this way.

What you can do, instead, is use the Selenium library, which allows you to launch an instance of a web browser controlled by Python, and get the page source from there.

from selenium import webdriver

path = '...' # path to driver here
url = 'https://www.bseindia.com/markets/equity/EQReports/HighLow.html?Flag=H#'

driver = webdriver.Chrome(path)
page_source = driver.get(url).page_source

By parsing page_source with BeautifulSoup, you can get what you want.

answered Feb 23, 2019 at 13:17

gmds

20k4 gold badges37 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Anand Shukla Over a year ago

thank you for the quick response however the page has 2 pages in it so 25 stocks on 1 page and 25 on another using above method i am able to parse first page only , the url is also same if i click on second page, please help how can i solve this ?

gmds Over a year ago

That is a separate question, I think @AnandShukla

Collectives™ on Stack Overflow

unable to parse a webpage using python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related