Ruby: Detecting broken links without actually crawling the URL?

Question

Is there a Ruby gem, or Ruby-esque way to check a webpage for broken links without crawling the actual links and checking for 404's, etc. Basically, I want a solution that works offline, and I want to detect links that are obviously syntactically broken, not links that point to web pages that don't exist.

So for instance, if a link points to "http//stackoverflow.com", that's a syntactically broken link, and I want to detect that. However if a link points to "http://www.webpagedoesnotexistyet.com" and it returns a 404, I'm OK with not detecting that.

Sounds like you want to use regex. I'd check out this post: stackoverflow.com/questions/4716513/… — Jonathan Bender
– Jonathan Bender, Commented Oct 30, 2013 at 16:50
What logic applies when relative links like /tags or /users are detected? — Anand Shah
– Anand Shah, Commented Oct 30, 2013 at 16:53
You'd still be using regex, first to find all a tags, then checking the href to make sure that it's either a valid full URL or begins with a / and doesn't include any invalid characters before the close quote. See this post: stackoverflow.com/questions/499345/… — Jonathan Bender
– Jonathan Bender, Commented Oct 30, 2013 at 17:25
Not a regex alone, but an XML parser to detect the href attribute for <a> tags, which then get passed through a regex. — PinnyM
– PinnyM, Commented Oct 30, 2013 at 17:31

Philip Hallstrom · Accepted Answer · 2013-10-30 18:26:22Z

0

Use nokogiri to parse the HTML and URI.parse to check for valid URLs. URI will raise an error if it encounters what it considers to be an invalid url.

answered Oct 30, 2013 at 18:26

Philip Hallstrom

19.9k2 gold badges44 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sergio Tulentsev Over a year ago

It will happily eat 'http//stackoverflow.com', though.

Rahul Malhotra · Accepted Answer · 2016-02-12 14:42:16Z

0

Use this : Links below is an array of links

for link in links do
    begin
        url = URI.parse(link)
        req = Net::HTTP.new(url.host, url.port)
        res = req.request_head(url.path)

        if res.code == "200"
            puts "#{res.code} ok - #{link}"
        else
            puts "#{res.code} error - #{link}"
        end
    rescue
        puts "breaking for #{link}"
    end
end

answered Feb 12, 2016 at 14:42

Rahul Malhotra

831 silver badge3 bronze badges

Comments

Sergio Tulentsev · Accepted Answer · 2016-02-12 14:50:07Z

0

You can use URI.regexp. If a string matches it, it is a valid uri.

require 'uri'

def valid_uri?(s)
  !!(s =~ URI.regexp)
end


valid_uri?('http//stackoverflow.com') # => false
valid_uri?('http://www.webpagedoesnotexistyet.com/') # => true

answered Feb 12, 2016 at 14:50

Sergio Tulentsev

231k43 gold badges381 silver badges373 bronze badges

Collectives™ on Stack Overflow

Ruby: Detecting broken links without actually crawling the URL?

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related