Removing Common data from two dataframes

Question

I have two dataframes, Dataframe 1 have 4 columns (A,B,C,D) with 20 rows and Dataframe 2 have same column headers as Dataframe 1 with 25 rows where first 20 rows are same as dataframe 1's 20 rows.

I want to Fetch the Dataframe 2's Unique 5 rows. and the fetching of those 5 rows should be done on index column.

Things that i've tried :

I have tried the left outer join merge (it works but isn't efficient much)
also tried the concatenation of two dataframes and then dropping the duplicates (it works only for 50% cases)
also researched about the subtraction method but didn't understand anything

Kindly Help me through this problem, Thank you.

Edit:

Dataframe 1 :

Date	Col1	Col2	Col3
10-2-2020	rowdata	row data	row_data
11-2-2020	rowdata	row data	row_data
...	...	...	...
20-2-2020	rowdata	row data	row_data

Dataframe 2 :

Date	Col1	Col2	Col3
10-2-2020	rowdata	row data	row_data
11-2-2020	rowdata	row data	row_data
...	...	...	...
20-2-2020	rowdata	row data	row_data
21-2-2020	rowdata	row data	row_data
...	...	...	...
25-2-2020	rowdata	row data	row_data

I want the data which is unique which means this data : Result Dataframe :

Date	Col1	Col2	Col3
21-2-2020	rowdata	row data	row_data
...	...	...	...
25-2-2020	rowdata	row data	row_data

and sometimes some data can be different in rows of both dataframe with same date index, but i don't care about that data all i want is above result dataframe

Please provide enough code so others can better understand or reproduce the problem. — Community
– Community Bot, Commented Sep 8, 2022 at 16:22

vtasca · Accepted Answer · 2022-09-08 13:40:11Z

0

You can run the following for a complete diff between the two dataframes:

# complete diff between two dfs
df = pd.concat([one_df, two_df])
df = df.reset_index(drop=True)
df_gpby = df.groupby(list(df.columns))
idx = [x[0] for x in df_gpby.groups.values() if len(x) == 1]
df.reindex(idx)

answered Sep 8, 2022 at 13:40

vtasca

1,78014 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Removing Common data from two dataframes

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related