3

Ok, so i'm working on a regular expression to search out all the header information in a site.

I've compiled the regular expression:

regex = re.compile(r'''
    <h[0-9]>\s?
    (<a[ ]href="[A-Za-z0-9.]*">)?\s?
    [A-Za-z0-9.,:'"=/?;\s]*\s?
    [A-Za-z0-9.,:'"=/?;\s]?
''',  re.X)

When i run this in python reg ex. tester, it works out wonderfully.

Sample data:

<body>
    <h1>Dog </h1>
    <h2>Cat </h2>
    <h3>Fancy </h3>
    <h1>Tall cup of lemons</h1>
    <h1><a href="dog.com">Dog thing</a></h1>
</body>

Now, in the REDemo, it works wonderfully.

When i put it in my python code, however, it only prints <a href="dog.com">

Here's my python code, I'm not sure if i'm doing something wrong or if something is lost in translation. I appreciate your help.

stories=[]
response = urllib2.urlopen('http://apricotclub.org/duh.html')
html = response.read().lower()
p = re.compile('<h[0-9]>\\s?(<a href=\"[A-Za-z0-9.]*\">)?\\s?[A-Za-z0-9.,:\'\"=/?;\\s]*\\s?[A-Za-z0-9.,:\'\"=/?;\\s]?')
stories=re.findall(p, html)
for i in stories:
    if len(i) >= 5:
        print i

I should also note, that when i take out the (<a href=\"[A-Za-z0-9.]*\">)? from the regular expression it works fine for non-link <hN> lines.

Improve this question

6 Answers 6

Reset to default
23

This question has been asked in several forms over the last few days, so I'm going to say this very clearly.

Q: How do I parse HTML with Regular Expressions?

A: Please Don't.

Use BeautifulSoup, html5lib or lxml.html. Please.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Don't forget that html5lib can parse non-conforming documents into a reasonable structure as well.
@Jerub: I agree with your answer but it doesn't answer the OP question. Imagine you can't use BeautifulSoup, html5lib, lxml and your html files are very simple then a trivial regex would suffice.
4

Parsing things with regular expressions works for regular languages. HTML is not a regular language, and the stuff you find on web pages these days is absolute crap. BeautifulSoup deals with tag-soup HTML with browser-like heuristics so you get parsed HTML that resembles what a browser would display.

The downside is it's not very fast. There's lxml for parsing well-formed html, but you should really use BeautifulSoup if you're not 100% certain that your input will always be well-formed.

Improve this answer

Comments

3

Building on the answers so far:

It's best to use a parsing engine. It can cover a lot of cases and in an elegant way. I've tried BeautifulSoup and I like it very much. Also easy to use, with a great tutorial.

If sometimes it feels like shooting flies with a cannon you can use a regular expression for quick parsing. If that's what you need here is the modified code that will catch all the headers (even those over multiple lines):

p = re.compile(r'<(h[0-9])>(.+?)</\1>', re.IGNORECASE | re.DOTALL)
stories = re.findall(p, html)
for i in stories:
    print i
Improve this answer

1 Comment

i in your cases is a tuple e.g. ('h1', '<a href="dog.com">Dog thing</a>')
2

Because of the braces around the anchor tag, that part is interpreted as a capture group. This causes only the capture group to be returned, and not the whole regex match.

Put the entire regex in braces and you'll see the right matches showing up as the first element in the returned tuples.

But indeed, you should use a real parser.

Improve this answer

Comments

2

I have used beautifulsoup to parse your desired HTML. I have the above HTML code in a file called foo.html and later read as a file object.

from BeautifulSoup import BeautifulSoup


H_TAGS = ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']

def extract_data():
   """Extract the data from all headers
   in a HTML page."""
   f = open('foo.html', 'r+')
   html = f.read()
   soup = BeautifulSoup(html)
   headers = [soup.findAll(h) for h in H_TAGS if soup.findAll(h)]
   lst = []
   for x in headers:
      for y in x:
         if y.string:
            lst.append(y.string)
         else:
            lst.append(y.contents[0].string)
   return lst

The above function returns:

>>> [u'Dog ', u'Tall cup of lemons', u'Dog thing', u'Cat ', u'Fancy ']

You can add any number of header tags in h_tags list. I have assumed all the headers. If you can solve things easily using BeautifulSoup then its better to use it. :)

Improve this answer

Comments

1

As has been mentioned, you should use a parser instead of a regex.

This is how you could do it with a regex though:

import re

html = '''
<body>

<h1>Dog </h1>
<h2>Cat </h2>
<h3>Fancy </h3>
<h1>Tall cup of lemons</h1>
<h1><a href="dog.com">Dog thing</a></h1>
</body>
'''

p = re.compile(r'''
    <(?P<header>h[0-9])>             # store header tag for later use
    \s*                              # zero or more whitespace
    (<a\shref="(?P<href>.*?)">)?     # optional link tag. store href portion
    \s*
    (?P<title>.*?)                   # title
    \s*
    (</a>)?                          # optional closing link tag
    \s*
    </(?P=header)>                   # must match opening header tag
''', re.IGNORECASE + re.VERBOSE)

stories = p.finditer(html)

for match in stories:
    print '%(title)s [%(href)s]' % match.groupdict()

Here are a couple of good regular expression resources:

Improve this answer

1 Comment

The regex doesn't work for headers with links. And OP asks specifically about headers with links.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.