How to get a html elements with python lxml

Question

I have this html code:

<table>
 <tr>
  <td class="test"><b><a href="">aaa</a></b></td>
  <td class="test">bbb</td>
  <td class="test">ccc</td>
  <td class="test"><small>ddd</small></td>
 </tr>
 <tr>
  <td class="test"><b><a href="">eee</a></b></td>
  <td class="test">fff</td>
  <td class="test">ggg</td>
  <td class="test"><small>hhh</small></td>
 </tr>
</table>

I use this Python code to extract all <td class="test"> with lxml module.

import urllib2
import lxml.html

code   = urllib.urlopen("http://www.example.com/page.html").read()
html   = lxml.html.fromstring(code)
result = html.xpath('//td[@class="test"][position() = 1 or position() = 4]')

It works good! The result is:

<td class="test"><b><a href="">aaa</a></b></td>
<td class="test"><small>ddd</small></td>


<td class="test"><b><a href="">eee</a></b></td>
<td class="test"><small>hhh</small></td>

(so the first and the fourth column of each <tr>) Now, I have to extract:

aaa (the title of the link)

ddd (text between <small> tag)

eee (the title of the link)

hhh (text between <small> tag)

How could I extract these values?

(the problem is that I have to remove <b> tag and get the title of the anchor on the first column and remove <small> tag on the forth column)

Thank you!

Ian Bicking · Accepted Answer · 2010-05-11 02:13:07Z

8

If you do el.text_content() you'll strip all the tag stuff from each element, i.e.:

result = [el.text_content() for el in result]

answered May 11, 2010 at 2:13

Ian Bicking

9,9406 gold badges36 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jochen Ritzel · Accepted Answer · 2010-05-11 01:20:10Z

4

Why dont you just fetch what you want in each step?

links = [el.text for el in html.xpath('//td[@class="test"][position() = 1]/b/a')]
smalls = [el.text for el in html.xpath('//td[@class="test"][position() = 4]/small')]
print zip(links, smalls) 
# => [('aaa', 'ddd'), ('eee', 'hhh')]

answered May 11, 2010 at 1:20

Jochen Ritzel

108k33 gold badges205 silver badges196 bronze badges

Collectives™ on Stack Overflow

How to get a html elements with python lxml

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related