Use Mechanize with select field that is not inside a form?

Question

I would like to use Mechanize (with Python) to submit a form, but unfortunately the page has been badly coded and the <select> element is not actually inside <form> tags.

So I can't use the traditional method via the form:

forms = [f for f in br.forms()]
mycontrol = forms[1].controls[0]

What can I do instead?

Here is the page I would like to scrape, and relevant bit of code - I'm interested in the la select item:

    <fieldset class="searchField">
      <label>By region / local authority</label>
      <p id="regp">
        <label>Region</label>
        <select id="region" name="region"><option></option></select>
      </p>
      <p id="lap">
        <label>Local authority</label>
        <select id="la" name="la"><option></option></select>
      </p>
      <input id="byarea" type="submit" value="Go" />
      <img id="regmap" src="/schools/performance/img/map_england.png" alt="Map of regions in England" border="0" usemap="#England" />
    </fieldset>

diwhyyyyy · Accepted Answer · 2012-06-15 15:22:44Z

1

This is actually more complex that you think, but still easy to implement. What is happening is that the webpage you linking is pulling in the local authorities by JSON (which is why the name="la" select element doesn't fill in Mechanize, which lacks Javascript). The easiest way around is to directly ask for this JSON data with Python and use the results to go directly to each data page.

import urllib2
import json

#The URL where we get our array of LA data
GET_LAS = 'http://www.education.gov.uk/cgi-bin/schools/performance/getareas.pl?level=la&code=0'

#The URL which we interpolate the LA ID into to get individual pages
GET_URL = 'http://www.education.gov.uk/schools/performance/geo/la%s_all.html'

def get_performance(la):
    page = urllib2.urlopen(GET_URL % la)
    #print(page.read())

#get the local authority list
las = json.loads(urllib2.urlopen(GET_LAS).read())

for la in las:
    if la != 0:
        print('Processing LA ID #%s (%s)' % (la[0], la[1]))
        get_performance(la[0])

As you can see, you don't even need to load the page you linked or use Mechanize to do it! However, you will still need a way to parse out the school names and then then performance figures.

answered Jun 15, 2012 at 15:22

diwhyyyyy

6,4304 gold badges32 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

flossfan Over a year ago

Thank you very much! I should have realised that if there was no <form> outer element, it couldn't possibly be submitting a form, and therefore must be a link.

Collectives™ on Stack Overflow

Use Mechanize with select field that is not inside a form?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related