How to add username and name columns to pandas dataframe with search_all_tweets lookup in python

Question

I am trying to collect tweets from 2022 using Twitter API. I can record the tweet_fields for the tweets, but I can't figure out how to add columns for the username and name (the user_fields) for each tweet.

I'm running the following code:

import requests
import os
import json
import tweepy
import pandas as pd
from datetime import timedelta
import datetime

bearer_token = "my_bearer_token_here"

keyword = "#WomeninSTEM"
start_time = "2022-01-01T12:01:00Z"
end_time = "2023-01-01T12:01:00Z"

client = tweepy.Client(bearer_token=bearer_token)

responses = client.search_all_tweets(query = "#WomeninSTEM", max_results= 500, start_time=start_time, end_time = end_time, 
user_fields = ["username", "name"],
tweet_fields =["in_reply_to_user_id", "author_id", "lang",
"public_metrics", "created_at", "conversation_id"])

**##I can't get the username or name columns to work here.** 
column = []
for i in range(len(responses.data)) :
    row = []
    Username = responses.data[i]["username"]
    row.append(Username)
    name = responses.data[i]["name"]
    row.append(name)
    text = responses.data[i].text
    row.append(text)
    favoriteCount = responses.data[i].public_metrics["like_count"]
    row.append(favoriteCount)
    retweet_count = responses.data[i].public_metrics["retweet_count"]
    row.append(retweet_count)
    reply_count = responses.data[i].public_metrics["reply_count"]
    row.append(reply_count)
    quote_count = responses.data[i].public_metrics["quote_count"]
    row.append(quote_count)
    created = responses.data[i].created_at
    row.append(created)
    ReplyTo = responses.data[i].text.split(" ")[0]
    row.append(ReplyTo)
    ReplyToUID = responses.data[i].in_reply_to_user_id
    row.append(ReplyToUID)
    ConversationID = responses.data[i]["conversation_id"]
    row.append(ConversationID)
    column.append(row)

data = pd.DataFrame(column)

Whenever I try and include username and name, I get this error:KeyError Traceback (most recent call last)

e-motta · Accepted Answer · 2023-01-11 16:31:35Z

0

Assuming you're querying at https://api.twitter.com/2/tweets/[...], the response does not have a 'username' or a 'name' parameter, that's why you're getting a KeyError when trying to access them.

It does have an 'author_id' parameter, which you can use to perform an additional query at https://api.twitter.com/2/users/:id and retrieve 'username' and 'name'.

More info here and here.

answered Jan 11, 2023 at 16:31

e-motta

7,5953 gold badges10 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

aoden Over a year ago

Is there no way to pull the username or name of the people who shared the tweet? I can pull the author_id, but it doesn't give me screen names or any useful information.

e-motta Over a year ago

If you query the api.twitter.com/2/users/:id endpoint with the author_id you will get the author's username and name.

Collectives™ on Stack Overflow

How to add username and name columns to pandas dataframe with search_all_tweets lookup in python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related