0

My code is to search a Link passed in the command prompt, get the HTML code for the webpage at the Link, search the HTML code for links on the webpage, and then repeat these steps for the links found. I hope that is clear.

It should print out any links that cause errors.

Some more needed info:

The max visits it can do is 100. If a website has an error, a None value is returned.

Python3 is what I am using

eg:

s = readwebpage(url)... # This line of code gets the HTML code for the link(url) passed in its argument.... if the link has an error, s = None.

The HTML code for that website has links that end in p2.html, p3.html, p4.html, and p5.html on its webpage. My code reads all of these, but it does not visit these links individually to search for more links. If it did this, it should search through these links and find a link that ends in p10.html, and then it should report that the link ending with p10.html has errors. Obviously it doesn't do that at the moment, and it's giving me a hard time.

My code..

    url = args.url[0]
    url_list = [url]
    checkedURLs = []
    AmountVisited = 0
    while (url_list and AmountVisited<maxhits):
        url = url_list.pop()
        s = readwebpage(url)
        print("testing url: http",url)                  #Print the url being tested, this code is here only for testing..
        AmountVisited = AmountVisited + 1
        if s == None:
            print("* bad reference to http", url)
        else:
            urls_list = re.findall(r'href="http([\s:]?[^\'" >]+)', s) #Creates a list of all links in HTML code starting with...
            while urls_list:                                          #... http or https
                insert = urls_list.pop()            
                while(insert in checkedURLs and urls_list):
                    insert = urls_list.pop()
                url_list.append(insert)
                checkedURLs = insert

Please help :)

Improve this question
14
  • 2
    Hi Shawn, why don’t you take a look at the stackoverflow.com/tour first :) Commented Jun 28, 2015 at 4:14
  • I did Rishav, I can't seem to understand why my code doesn't search links found in the HTML... Commented Jun 28, 2015 at 4:22
  • 1
    Shawn, not trying to be rude, but your question looks like a mess, and unless you clean it up, nobody will want to help you. Use the formatting tools. Code should be inside backticks ` like this is code. All your code should be here and not on OneDrive. Clean up your question, I’ll help you. Commented Jun 28, 2015 at 4:24
  • 2
    You are attempting to parse HTML with regexes. That is an unpardonable sin. Commented Jun 28, 2015 at 4:29
  • 1
    BeautifulSoup is not permitted, unfortunately Commented Jun 28, 2015 at 4:37

3 Answers 3

Reset to default
1

Here is the code you wanted. However, please, stop using regexes for parsing HTML. BeautifulSoup is the way to go for that.

import re
from urllib import urlopen

def readwebpage(url):
  print "testing ",current     
  return urlopen(url).read()

url = 'http://xrisk.esy.es' #put starting url here

yet_to_visit= [url]
visited_urls = []

AmountVisited = 0
maxhits = 10

while (yet_to_visit and AmountVisited<maxhits):

    print yet_to_visit
    current = yet_to_visit.pop()
    AmountVisited = AmountVisited + 1
    html = readwebpage(current)


    if html == None:
        print "* bad reference to http", current
    else:
        r = re.compile('(?<=href=").*?(?=")')
        links = re.findall(r,html) #Creates a list of all links in HTML code starting with...
        for u in links:

          if u in visited_urls: 
            continue
          elif u.find('http')!=-1:
            yet_to_visit.append(u)
        print links
    visited_urls.append(current)
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Tried it out, it doesn't find the link ending in p10.html unfortunately. I suppose that could be due to a difference in our readwebpage, i'm not sure. Other than BeautifulSoup and regex, could you suggest another way I find the links in the HTML? I appreciate your help
It should print the URL whenever it goes to a page
@Shawn if you’re there, then the output produced by my code will help me debug it :D
0

Not Python but since you mentioned you aren't tied strictly to regex, I think you might find some use in using wget for this.

wget --spider -o C:\wget.log -e robots=off -w 1 -r -l 10 http://www.stackoverflow.com

Broken down:

--spider: When invoked with this option, Wget will behave as a Web spider, which means that it will not download the pages, just check that they are there.
-o C:\wget.log: Log all messages to C:\wget.log.
-e robots=off: Ignore robots.txt
-w 1: set a wait time of 1 second
-r: set recursive search on
-l 10: sets the recursive depth to 10, meaning wget will only go as deep as 10 levels in, this may need to change depending on your max requests
http://www.stackoverflow.com: the URL you want to start with

Once complete, you can review the wget.log entries to determine which links had errors by searching for something like HTTP status codes 404, etc.

Improve this answer

Comments

0
  1. I suspect your regex is part of your problem. Right now, you have http outside your capture group, and [\s:] matches "some sort of whitespace (ie \s) or :"

I'd change the regex to: urls_list = re.findall(r'href="(.*)"',s). Also known as "match anything in quotes, after href=". If you absolutely need to ensure the http[s]://, use r'href="(https?://.*)"' (s? => one or zero s)

EDIT: And with actually working regex, using a non-greedly glom: href=(?P<q>[\'"])(https?://.*?)(?P=q)'

(Also, uh, while it's not technically necessary in your case because re caches, I think it's good practice to get into the habit of using re.compile.)

  1. I think it's awfully nice that all of your URLs are full URLs. Do you have to deal with relative URLs at all? `
Improve this answer

1 Comment

This regex will unfortunately fail. Try running it on google.com

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.