Ruby Mechanize not returning Javascript built page correctly

Question

I'm trying to create a script to fill out a multi-page "form" that I have to fill out weekly (unemployment form actually), the 4th page ends up giving you a Checkbox and 2 Radio Buttons, all built by Javascript. When I navigate to this page using Mechanize I get html back without those 3 controls so I can't go any farther in the process.

Is this a common problem?
I'm filling out the form then just calling page = agent.submit(form, form.buttons.first) and it comes back without those controls built.

Jörg W Mittag · Accepted Answer · 2011-01-03 16:23:21Z

2

Mechanize is an HTML parser, not a JavaScript interpreter. If it's not in the HTML, there's nothing it can do. You need a "proper" browser. (By "proper" I mean one which can at least parse HTML, run JavaScript and ideally also interpret CSS.)

There are tools like Selenium & Co. that let you "remote-control" a "real" browser (Firefox, Internet Explorer, …) and there are efforts to build completely scriptable GUI-less browsers for precisely this use case.

Note: Depending on what country you are in, the unemployment agency may be in violation of anti-discrimination laws (especially if it's a government agency), so you could maybe force them to offer a JavaScript-free version of the form, but that's a) not a short-term solution and b) a topic for your lawyer, not StackOverflow.

answered Jan 3, 2011 at 16:23

Jörg W Mittag

371k79 gold badges457 silver badges666 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

cam · Accepted Answer · 2011-01-03 20:20:31Z

1

Are the values of the generated form predictable? I often find it convenient to bypass all the individual form-helpers and just post to the form directly:

browser = Mechanize.new
browser.post(some_url, { field1 => val1, field2 => val2, ... })

answered Jan 3, 2011 at 20:20

cam

14.2k1 gold badge46 silver badges29 bronze badges

Comments

the Tin Man · Accepted Answer · 2011-01-03 18:38:40Z

0

You might want to look into using Watir if you're on Windows, or firewater on Mac/Linux or safariwatir on Mac only. All are basically the same code and are at the same site.

It's more oriented toward testing websites, but you can get at the content of the page using xpath, and from there proceed. Hopefully the browser will have processed the javascript for you and will return that. I've seen some browsers display the JS rendered HTML in their source view, and others don't, so I'm not sure what results you'll have.

edited Jan 3, 2011 at 18:38

answered Jan 3, 2011 at 18:20

the Tin Man

161k44 gold badges222 silver badges308 bronze badges

Comments

Zombies · Accepted Answer · 2011-01-03 22:29:14Z

0

As has been mentioned in other answers, you need to use something which drives a real web browser as there is currently no libraries capable of parsing that level of javascript (some can follow javascript redirects, but that is pretty much it). This would be ideal and easier to maintain.

If you really want to stick with the mechanize approach then you should simply be able to add the post field manually.

If they use a captcha to circumvent automated posting then you can need to resort to a simple decaptcha service (10 dollars for 2000 credits should be enough).

Lastly, it may be prudent to just not go through all this trouble.

answered Jan 3, 2011 at 22:29

Zombies

26k45 gold badges147 silver badges233 bronze badges

Collectives™ on Stack Overflow

Ruby Mechanize not returning Javascript built page correctly

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related