Combine DataFrame in Pandas

Question

I have to connect and read data from 10 databases and save data as pandas data frame. After combining dataframes, I have an empty dataframe:

df1 = pd.DataFrame(columns={'name', 'ip'})

# in a loop I connect to db and read sql data and combine data 

for db in database_list:
  db_df = pd.read_sql_query(sql, con)
  df1 = df1.append(db_df)
  df1 = df1.drop_duplicates(subset='name', keep='last')

print df1

df1 is an empty dataframe. What is the correct way for combine dataframes in this situation?

jezrael · Accepted Answer · 2017-10-08 09:19:33Z

2

I think you can append each DataFrame to list and last concat all to one big df:

dfs = []
for db in database_list:
  db_df = pd.read_sql_query(sql, con)
  print db_df
  db_df = db_df.drop_duplicates(subset='name', keep='last')
  dfs.append(db_df)

df1 = pd.concat(dfs, ignore_index=True)
print df1

edited Oct 8, 2017 at 9:19

answered Oct 8, 2017 at 8:58

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

TheNone Over a year ago

with list I have ` 'NoneType' object has no attribute 'append' ` error

jezrael Over a year ago

Hmmmm, db_df = pd.read_sql_query(sql, con) return for you DataFrame? Can you check it?

TheNone Over a year ago

yes it is a dataframe, I think I do the wrong thing in drop duplicates

Collectives™ on Stack Overflow

Combine DataFrame in Pandas

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related