5

I need to normalize the rows of a dataframe containing rows populated with all zero. For example: 

df= pd.DataFrame({"ID": ['1', '2', '3', '4'], "A": [1, 0, 10, 0], "B": [4, 0, 30, 0]})

ID  A   B
1   1   4
2   0   0
3   10  30
4   0   0

My approach is to first exclude the zero-value rows followed by normalizing the non-zero subset using:

df1 = df[df.sum(axis=1) != 0]
df2 = df[df.sum(axis=1) == 0]
sum_row = df1.sum(axis=1)
df1.div(sum_row, axis=0)

and then concatenate the two dataframes as follows:

pd.concat([df1, df2]).reset_index()

However, I end up with the following error while applying df1.div(sum_row, axis=0)

ValueError: operands could not be broadcast together with shapes (6,) (2,)

I wonder how to fix the error and if there exists a more efficient approach. Thanks!

Edit: The resulting dataframe is expected to look like as:

ID  A     B
1   0.2   0.8 
2   0     0
3   0.25  0.75
4   0     0
Improve this question
2
  • 1
    Could you add the excepted results, please? Commented Aug 24, 2018 at 15:03
  • @AnnaIliukovich-Strakovskaia Done! Commented Aug 24, 2018 at 15:09

3 Answers 3

Reset to default
7

You can use Normalizer in scikit-learn

df= pd.DataFrame({"ID": ['1', '2', '3', '4'], "A": [1, 0, 10, 0], "B": [4, 0, 30, 0]})
df = df.set_index('ID')

from sklearn.preprocessing import Normalizer
df.iloc[:,:] = Normalizer(norm='l1').fit_transform(df)

print(df)

       A     B
ID            
1   0.20  0.80
2   0.00  0.00
3   0.25  0.75
4   0.00  0.00
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

This is great because you can easily change the norm used. In my case, I see so many examples of the l1 norm on SO, whereas I need to normalize each row according to l2 for my current needs.
4

Use div: 

df= pd.DataFrame({"ID": ['1', '2', '3', '4'], "A": [1, 0, 10, 0], "B": [4, 0, 30, 0]})
df.set_index("ID", inplace=True)
df.div(df.sum(axis=1), axis=0).fillna(0)
Improve this answer

Comments

1

Using melt with crosstab

newdf=df.melt('ID')
pd.crosstab(index=newdf.ID,columns=newdf.variable,values=newdf.value,normalize='index',aggfunc='mean')
Out[447]: 
variable     A     B
ID                  
1         0.20  0.80
2         0.00  0.00
3         0.25  0.75
4         0.00  0.00
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.