How to use webscraping in Python using username and password for website

Question

Hi I am using this code to get the data from the timeanddate.com site. But I need to do the same for my facebook accout using my username and password & grab the comments on my wall and write it to text file. Can I do this with python? How to do that?

import urllib2
from BeautifulSoup import BeautifulSoup


soup = BeautifulSoup(urllib2.urlopen('http://www.timeanddate.com/worldclock/astronomy.html?n=78').read())

for row in soup('table', {'class' : 'spad'})[0].tbody('tr'):
tds = row('td')
print tds[0].string, tds[1].string

Thanks....

Yes, you can do it in Python. I generally use the requests module and BeautifulSoup for screen scraping. I've never used authentication so far, but if requests is not enough, you could always emulate a browser, e.g. with mechanize, or use the API as said. — Ignacio Contreras Pinilla
– Ignacio Contreras Pinilla, Commented Nov 2, 2012 at 12:34
with facebook and the likes, I wouldn't try to screenscrape anything zdnet.com/plaxo-screen-scraper-tool-offends-facebook-1339284842 — Uku Loskit
– Uku Loskit, Commented Nov 2, 2012 at 12:36

Sushant Gupta · Accepted Answer · 2012-11-02 12:31:02Z

7

Use Requests API. It handles authentication as well.

answered Nov 2, 2012 at 12:31

Sushant Gupta

9,5185 gold badges46 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to use webscraping in Python using username and password for website

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related