Ignore empty dataframe when merging

Question

I have four df (df1,df2,df3,df4)

Sometimes df1 is null, sometimes df2 is null, sometimes df3 and df4 accordingly.

How can I do an outer merge so that the df which is empty is automatically ignored? I am using the below code to merge as of now:-

df = f1.result().merge(f2.result(), how='left', left_on='time', right_on='time').merge(f3.result(), how='left', left_on='time', right_on='time').merge(f4.result(), how='left', left_on='time', right_on='time')

and

df = reduce(lambda x,y: pd.merge(x,y, on='time', how='outer'), [f1.result(),f2.result(),f3.result(),f4.result()])

Shubham Sharma · Accepted Answer · 2020-04-01 05:48:45Z

1

You can use df.empty attribute or len(df) > 0 to check whether the dataframe is empty or not.

Try this:

dfs = [df1, df2, df3, df4]
non_empty_dfs = [df for df in dfs if not df.empty]

df_final = reduce(lambda left,right: pd.merge(left,right, on='time', how='outer'), non_empty_dfs)

Or, you could also filter empty dataframe as,

non_empty_dfs = [df for df in dfs if len(df) > 0]

edited Apr 1, 2020 at 5:48

answered Apr 1, 2020 at 5:37

Shubham Sharma

71.8k6 gold badges26 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

sammywemmy · Accepted Answer · 2020-04-01 05:43:46Z

1

use pandas' dataframe empty method, to filter out the empty dataframe, then you can concatenate or run whatever merge operation you have in mind:

df4 = pd.DataFrame({'A':[]}) #empty dataframe
df1 = pd.DataFrame({'B':[2]})
df2 = pd.DataFrame({'C':[3]})
df3 = pd.DataFrame({'D':[4]})

dfs = [df1,df2,df3,df4]

#concat
#u can do other operations since u have gotten rid of the empty dataframe

pd.concat([df for df in dfs if not df.empty],axis=1)

    B   C   D
0   2   3   4

answered Apr 1, 2020 at 5:43

sammywemmy

28.9k4 gold badges21 silver badges35 bronze badges

2 Comments

abhi Over a year ago

This works but supposes if one of the df has some missing DateTime index rows then it shifts the values. and do not return null for nonmatching values. Suppose none of them are empty df but if df2 has four missing values then it shifts the values and always returns the last four rows as null.

abhi Over a year ago

yes, but that's why I specified merging. Thanks a lot for your answer!

Collectives™ on Stack Overflow

Ignore empty dataframe when merging

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related