Instructing Python to click a button using urllib2

Question

I'm writing a web scraper using urllib2 and BeautifulSoup in python and am looking for a way to instruct python to click a button on a page that it reads the HTML source code for.

The following snippet of my script reads in URLs from a csv file and is meant to scrape data from the webpages specified, but an intermediary step is to click a "submit" button that exists on the webpage that is read from the csv's provided URLs.

for line in triplines:
    FromTo = line.split(",")
    From = FromTo[0].strip()
    print(From)
    To = FromTo[1].strip()
    print(To)
    url = KCString1 + From + KCString2 + To + KCString3
    print(url)
    page = urllib2.urlopen(url)
    page_source = page.read()
    soup = BeautifulSoup(page_source)
    print(soup.prettify())

Is there a way to utilize urllib2 functionality in such a way as to say "follow the URL that is obtained from clicking this button"? I imagine I may need to find the JavaScript source to identify the button's identifiers first.

Not sure if you want to use urllib2 for this. Have you looked at Selenium? — Matt
– Matt, Commented Jul 2, 2014 at 19:17

Strikeskids · Accepted Answer · 2014-07-02 19:17:10Z

3

Buttons do not typically have urls attached to them. They normally need javascript interaction, which needs emulation. If you want to click a button, you should use a browser emulator like Ghost instead of a parser like Beautifulsoup

answered Jul 2, 2014 at 19:17

Strikeskids

4,08215 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Instructing Python to click a button using urllib2

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related