Ruby Mechanize issues with http authentication

Question

I'm having issues getting around websites that use http authentication, I have a list of sites which I do some scrapping on but some of these have http authentication on them. I'm not looking to get the content of those sites I want to be able to be able to determine if they are guarded by http auth and then move on. For example in the snippet below agent.get never return so it's impossible for me to handle it. How can I handle a case like this?

require 'mechanize'
agent = Mechanize.new
page = agent.get('http://freyalovesmusic.co.uk')

Justin Ko · Accepted Answer · 2012-10-29 20:28:28Z

2

You could assume that if a page takes too long to load, it is using http authentication. Obviously not 100% accurate, but perhaps good enough for your situation?

You can use the Timeout module to move on after a certain amount of time, even if agent.get never returns:

require 'mechanize'
require 'timeout'

agent = Mechanize.new
begin
    Timeout::timeout(5) do
        page = agent.get('http://freyalovesmusic.co.uk')
    end
rescue Timeout::Error
    puts 'Page likely using http authentication'
end

answered Oct 29, 2012 at 20:28

Justin Ko

46.9k5 gold badges95 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Luis D Urraca Over a year ago

Wow awesome... this is what ended up doing, actually did it before reading it here. Validates my thinking.

pguardiario · Accepted Answer · 2012-10-30 00:56:59Z

1

It should be raising a Mechanize::UnauthorizedError but it's misbehaving for some reason. Maybe you should report it on the mechanize github issues form.

answered Oct 30, 2012 at 0:56

pguardiario

55.2k21 gold badges130 silver badges169 bronze badges

Collectives™ on Stack Overflow

Ruby Mechanize issues with http authentication

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related