Call onclick method while parsing a web page

Question

Disclaimer: This is my first attempt at web scraping. Do not mind if its a very simple question.

I get the html source from a web site as below.

def get_soup(site):
"""get the html source for a web site"""
try:
    logging.debug("Getting data from:" + site)
    req = urllib2.Request(site)
    req.add_header('Accept', '*/*')
    req.add_header('User-Agent', 'web-scraping')
    page = urllib2.urlopen(req).read()        
    return BeautifulSoup(page)       
except:
    logging.debug("Check the site: " + str(site))
    raise

And the soup has an onclick method as below. Is there any way to call this method while making the url request(preferably) or afterwards(if at all its possible)?

soup = get_soup(opt_site.replace("<symbol>", symbol).replace("<expiry>", expiry.strftime("%d%b%Y")))
spot_price = soup.findAll('table')[0].findChildren('td')[1].findChildren('span')[0]

<span>As on Dec 23, 2013 15:30:25 IST<a> <img src="/live_market/resources/images/refressbtn.gif" onclick="refresh();" style="cursor: pointer" title="refresh" /></a></span>

falsetru · Accepted Answer · 2013-12-23 14:43:01Z

1

BeautifulSoup does not handle javascript. You should use something that can handle javascript; for instance, selenium.

answered Dec 23, 2013 at 14:43

falsetru

371k69 gold badges769 silver badges659 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Call onclick method while parsing a web page

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related