Extend a Ruby Mechanize script to use a proxy

Question

I found a neat Ruby script to sum all purchases done on Amazon.de (no other stores like US) for a given year: https://github.com/pwaldhauer/amazon-account-crawler

After installing Ruby and the necessary Ruby Gems Highline and Mechanize I'm able to run the script. But unfortunately I'm behind a proxy server so the script fails with a "Timeout" error.

I read a lot but didn't found out how to use a proxy server. I tried to "set" a HTTP_PROXY environment variable, but still get errors. I also used the following call:

agent.set_proxy('127.0.0.1', '3128')

But this didn't worked out as well. I have the feeling that the HTTP proxy worked but after the login Amazon uses HTTPS and this fails. Can someone tell me a simple way how to tell Mechanize to use a HTTP and HTTPS proxy server?

You can use github.com/apenwarr/sshuttle to make the whole device you are using runs via ssh tunnle. Really easy to use. Give it a shot. — khelll
– khelll, Commented Feb 3, 2012 at 8:58

pguardiario · Accepted Answer · 2012-02-03 09:30:39Z

2

There is a known issue with mechanize https and proxies, you will need to use an older version (1.0.0) to get it to work. Also the port should be a number not a string.

answered Feb 3, 2012 at 9:30

pguardiario

55.2k21 gold badges130 silver badges169 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

asdrubael Over a year ago

How to install this old version? I tried with gem install mechanize 1.0.0 but there is no such "gem" in the repository.

pguardiario Over a year ago

gem install mechanize -v=1.0.0

asdrubael Over a year ago

Thanks a lot, that worked but now there is a different error:

C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize 1.0.0/lib/mechanize/form.rb:136:in 'method_missing': undefined method 'id'  for #<Mechanize::Form:0x12c51f0> (NoMethodError)         from (eval):4:in 'block (2 levels) in forms_with'

I don't know if this is still a connection problem or if I need to rewrite the script to use mechanize version 1.0.

pguardiario Over a year ago

try using name instead of id, or just forms[0] if it's the first one.

asdrubael Over a year ago

It worked for the login form, thanks. Unfortunately there is a dropdown box, which has no "name" field: select_form = orders_page.form_with(:id => 'order-dropdown-form') This is the HTML source from Amazon:

<form id="order-dropdown-form" action="/gp/css/history/orders/view.html/ref=ohfilter_C" method="get">     <input type="hidden" name="opt" value="ab">

|

Collectives™ on Stack Overflow

Extend a Ruby Mechanize script to use a proxy

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related