Scrape address using BeautifulSoup for Python

Question

I am having difficulties scraping the address from the following weblink, please help me scrape the address.

http://www.salatomatic.com/d/Revesby+17154+Ahlus-Sunnah-Wal-Jamaah-Revesby

the source code for the weblink above is as follow

<td width="100%"><div class="titleBM">Bankstown Masjid </div>Meredith Street, Bankstown, New South Wales 2200</td>

I am trying to scrape the value immediatly after </div>

my current code is not completed but looks like follow

content1 = urllib2.urlopen(url1).read()
soup1 = BeautifulSoup(content1)
div1 = soup1.find('div', {'class':'titleBM'}) #get the div where it's located
span1 = div1.find('</div>')
pos1 = span1.text       

print datetime.datetime.now(), 'street address:  ' , pos1)

Birei · Accepted Answer · 2013-12-03 12:09:16Z

1

The text is the next sibling of the <div> element, so use next_sibling:

from bs4 import BeautifulSoup
import urllib2
import datetime

url1 = 'http://www.salatomatic.com/d/Revesby+17154+Ahlus-Sunnah-Wal-Jamaah-Revesby'

content1 = urllib2.urlopen(url1).read()
soup1 = BeautifulSoup(content1)
div1 = soup1.find('div', {'class':'titleBM'}) #get the div where it's located
pos1 = div1.next_sibling

print datetime.datetime.now(), 'street address:  ' , pos1

Run it like:

python2 script.py

It yields:

2013-12-03 12:55:41.306271 street address:   9-11 Mavis Street, Revesby, New South Wales 2212

edited Dec 3, 2013 at 12:09

answered Dec 3, 2013 at 11:56

Birei

36.4k3 gold badges80 silver badges84 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 11:43:58Z

-1

This happening because of JavaScript, you should use selenium webdriver to solve this issue:

from selenium.webdriver import Firefox

Find more here Link

edited May 23, 2017 at 11:43

CommunityBot

11 silver badge

answered Dec 3, 2013 at 13:43

Yogesh dwivedi Geitpl

4,4622 gold badges22 silver badges34 bronze badges

1 Comment

Adam Williamson Over a year ago

I think you were a little quick to jump to selenium on this one. The accepted answer shows how to complete without

Collectives™ on Stack Overflow

Scrape address using BeautifulSoup for Python

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related