Twitter scraping in Python

Question

I have to scrape tweets from Twitter for a specific user (@salvinimi), from January 2018. The issue is that there are a lot of tweets in this range of time, and so I am not able to scrape all the ones I need! I tried multiple solutions:

1)

pip install twitterscraper
from twitterscraper import query_tweets_from_user as qtfu
tweets = qtfu(user='matteosalvinimi')

With this method, I get only a few teets (500~600 more or less), instead of all the tweets... Do you know why?

2)

!pip install twitter_scraper
from twitter_scraper import get_tweets
tweets = []
for i in get_tweets('matteosalvinimi', pages=100):
    tweets.append(i)

With this method I get an error -> "ParserError: Document is empty"... If I set "pages=40", I get the tweets without errors, but not all the ones. Do you know why?

bglbrt · Accepted Answer · 2019-11-22 14:42:18Z

2

Three things for the first issue you encounter:

first of all, every API has its limits and one like Twitter would be expected to monitor its use and eventually stop a user from retrieving data if the user is asking for more than the limits. Trying to overcome the limitations of the API might not be the best idea and might result in being banned from accessing the site or other things (I'm taking guesses here as I don't know what's the policy of Twitter on the matter). That said, the documentation on the library you're using states :

With Twitter's Search API you can only sent 180 Requests every 15 minutes. With a maximum number of 100 tweets per Request this means you can mine for 4 x 180 x 100 = 72.000 tweets per hour. By using TwitterScraper you are not limited by this number but by your internet speed/bandwith and the number of instances of TwitterScraper you are willing to start.
then, the function you're using, query_tweets_from_user() has a limit argument which you can set to an integer. One thing you can try is changing that argument and seeing whether you get what you want or not.
finally, if the above does not work, you could be subsetting your time range in two, three ore more subsets if needed, collect the data separately and merge them together afterwards.

The second issue you mention might be due to many different things so I'll just take a broad guess here. For me, either setting pages=100 is too high and by one way or another the program or the API is unable to retrieve the data, or you're trying to look at a hundred pages when there is less than a hundred in pages to look for reality, which results in the program trying to parse an empty document.

edited Nov 22, 2019 at 14:42

answered Nov 22, 2019 at 11:47

bglbrt

2,10811 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Luca Over a year ago

Thanks for your reply. I will try as you suggested.

Luca Over a year ago

Anyway, for future issues like this, I just found another useful library (getOldTweets3) that performs the tweets scraping very good, without returning the aforementioned errors!

Collectives™ on Stack Overflow

Twitter scraping in Python

1)

2)

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1)

2)

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related