Mechanize getting stuck on Javascript redirect page

Question

I am using Mechanize and Nokogiri in my Ruby on Rails application to scrape our local printers administration panel to retrieve the number of printed pages in the printers lifetime.

I have the following rake task:

# Logs into printer admin page and retrieved counts.
require 'rubygems'
require 'mechanize'
require 'logger'

# Create a new mechanize object
agent = Mechanize.new

# Load the printer admin page
page = agent.get("http://192.168.1.126/index.html?lang=1")

# Select the form with an action of index.cqi
form = agent.page.form_with(:action => "index.cgi")
form.radiobuttons_with(:id => '0x3fdb24153404')[1]

# Submit the form
page = form.submit form.buttons.first

pp page

That returns the following:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<script type="text/javascript">
<!--
window.onload=function(){setTimeout(function(){document.menu_link.submit();},0);}
//-->
</script>
</head>
<body>
<form name="menu_link" action="index.html" method="post" enctype="application/x-www-form-urlencoded">
<input type="hidden" name="lang" value="1">
</form>
</body>
</html>

I don't seem to be able to select the form on the above page, and the script seems to stop at that page and not follow the redirect.

Is there a standard way of working with these kind of redirects? Maybe pausing the script until the redirect has taken place? Would it every allow the redirect to work?

Any pointers would be appreciated!

How are you selecting the form in the second page?

Christian
– Christian

2012-04-22 12:18:47 +00:00
Commented Apr 22, 2012 at 12:18 — Christian
– Christian, Commented Apr 22, 2012 at 12:18
I just selected the form with a name of menu_link.

dannymcc
– dannymcc

2012-04-22 15:17:34 +00:00
Commented Apr 22, 2012 at 15:17 — dannymcc
– dannymcc, Commented Apr 22, 2012 at 15:17

Community · Accepted Answer · 2017-05-23 11:58:01Z

1

You've got two options. Either:

Submit the form manually
Use Watir or Selenium

Basically Mechanise will not run javascript, so you have to manually simulate the javascript running (option 1) or automate a real browser to do it (option 2)

Option 1 should be double if you just do a POST of lang=1 instead of a get as that's all the form is doing.

I'd guess something like this:

page = agent.post('http://192.168.1.126/index.html', {
  "lang" => "1"
})

But I've never actually used Mechanize.

edited May 23, 2017 at 11:58

CommunityBot

11 silver badge

answered Apr 29, 2012 at 20:44

mattmanser

5,8363 gold badges41 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

dannymcc Over a year ago

I'm already requesting lang=1 in the login page page = agent.get("http://192.168.1.126/index.html?lang=1") requesting it again gives a 404 error.

mattmanser Over a year ago

It's not a get, you need to be doing a post.

Jakub Oboza · Accepted Answer · 2012-04-29 20:53:03Z

0

You should try to add follow on redirects like this

agent.follow_meta_refresh = true

Also if this is javascript controlled behaviour then you are in bad position because mechanize can't follow this. He don't executes js. You will have to see in js how he does it and emulate same call in mechanize.

But i think all you need to do is just

agent.post <url>

because he seems to be expecting post method.

There is hardcore alternative :) to use node-crawler in node.js https://github.com/joshfire/node-crawler it can evaluate javascript from client page server side.

answered Apr 29, 2012 at 20:53

Jakub Oboza

5,4211 gold badge21 silver badges10 bronze badges

3 Comments

dannymcc Over a year ago

If I do agent.post "index.html" it returns me to the login page?

mattmanser Over a year ago

There's no meta refresh in that HTML, it's a js form submit, totally different mechanism

dantheta Over a year ago

The link to the node.js alternative is broken. I think it's been moved here github.com/sylvinus/node-crawler

Collectives™ on Stack Overflow

Mechanize getting stuck on Javascript redirect page

2 Answers 2

2 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related