Why I am getting Error while using Lambda within Apply

Question

Request help on why the following is giving error?:

import numpy as np
from pydataset import data
mtcars = data('mtcars')

mtcars.apply(['mean', lambda x: max(x)-min(x), lambda x: np.percentile(x, 0.15)])

I am trying to create a data frame with the mean, max-min and 15th Percentile for all columns of the dataset mtcars.

Error Message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/aggregation.py in agg_list_like(obj, arg, _axis)
    674     try:
--> 675         return concat(results, keys=keys, axis=1, sort=False)
    676     except TypeError as err:

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/reshape/concat.py in concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)
    284     """
--> 285     op = _Concatenator(
    286         objs,

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/reshape/concat.py in __init__(self, objs, axis, join, keys, levels, names, ignore_index, verify_integrity, copy, sort)
    369                 )
--> 370                 raise TypeError(msg)
    371 

TypeError: cannot concatenate object of type '<class 'float'>'; only Series and DataFrame objs are valid

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-645-51b8f1de1855> in <module>
----> 1 mtcars.apply(['mean', lambda x: max(x)-min(x), lambda x: np.percentile(x, 0.15)])

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/frame.py in apply(self, func, axis, raw, result_type, args, **kwds)
   7766             kwds=kwds,
   7767         )
-> 7768         return op.get_result()
   7769 
   7770     def applymap(self, func, na_action: Optional[str] = None) -> DataFrame:

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/apply.py in get_result(self)
    145             # pandas\core\apply.py:144: error: "aggregate" of "DataFrame" gets
    146             # multiple values for keyword argument "axis"
--> 147             return self.obj.aggregate(  # type: ignore[misc]
    148                 self.f, axis=self.axis, *self.args, **self.kwds
    149             )

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/frame.py in aggregate(self, func, axis, *args, **kwargs)
   7576         result = None
   7577         try:
-> 7578             result, how = self._aggregate(func, axis, *args, **kwargs)
   7579         except TypeError as err:
   7580             exc = TypeError(

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/frame.py in _aggregate(self, arg, axis, *args, **kwargs)
   7607             result = result.T if result is not None else result
   7608             return result, how
-> 7609         return aggregate(self, arg, *args, **kwargs)
   7610 
   7611     agg = aggregate

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/aggregation.py in aggregate(obj, arg, *args, **kwargs)
    584         # we require a list, but not an 'str'
    585         arg = cast(List[AggFuncTypeBase], arg)
--> 586         return agg_list_like(obj, arg, _axis=_axis), None
    587     else:
    588         result = None

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/aggregation.py in agg_list_like(obj, arg, _axis)
    651             colg = obj._gotitem(col, ndim=1, subset=selected_obj.iloc[:, index])
    652             try:
--> 653                 new_res = colg.aggregate(arg)
    654             except (TypeError, DataError):
    655                 pass

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/series.py in aggregate(self, func, axis, *args, **kwargs)
   3972             func = dict(kwargs.items())
   3973 
-> 3974         result, how = aggregate(self, func, *args, **kwargs)
   3975         if result is None:
   3976 

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/aggregation.py in aggregate(obj, arg, *args, **kwargs)
    584         # we require a list, but not an 'str'
    585         arg = cast(List[AggFuncTypeBase], arg)
--> 586         return agg_list_like(obj, arg, _axis=_axis), None
    587     else:
    588         result = None

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/aggregation.py in agg_list_like(obj, arg, _axis)
    683         result = Series(results, index=keys, name=obj.name)
    684         if is_nested_object(result):
--> 685             raise ValueError(
    686                 "cannot combine transform and aggregation operations"
    687             ) from err

ValueError: cannot combine transform and aggregation operations

But, the following works:

mtcars.apply(['mean', lambda x: max(x)-min(x)])

Both type(mtcars.apply(lambda x: np.percentile(x, 0.15))) and type(mtcars.apply(lambda x: max(x)-min(x))) gives Pandas Series. Then why the problem is happening with only the Percentile?

Thanks

What is mtcars? Is it a list, an instance of a custom class, a Pandas dataframe, etc.? If it's Pandas, you'll probably want to tag this question with pandas — Silvio Mayolo
– Silvio Mayolo, Commented Jul 26, 2021 at 18:49

StupidWolf · Accepted Answer · 2021-07-26 19:59:34Z

1

Reading the answer by @James my guess is that you need to write the custom function such that the function is applied on the series and not over each element. Maybe someone else who is more familiar with the underlying pandas code can chip in:

def min_max(x):
    return max(x)-min(x)
def perc(x):
    return x.quantile(0.15)

mtcars.agg(['mean',min_max,perc])

               mpg     cyl        disp        hp      drat       wt      qsec      vs       am    gear    carb
mean     20.090625  6.1875  230.721875  146.6875  3.596563  3.21725  17.84875  0.4375  0.40625  3.6875  2.8125
min_max  23.500000  4.0000  400.900000  283.0000  2.170000  3.91100   8.40000  1.0000  1.00000  2.0000  7.0000
perc     14.895000  4.0000  103.485000   82.2500  3.070000  2.17900  16.24300  0.0000  0.00000  3.0000  1.0000

answered Jul 26, 2021 at 19:59

StupidWolf

47.1k17 gold badges50 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Samit Over a year ago

Thank you for your answer. But I am still not clear as to why the Lambda expressions are not working.

StupidWolf Over a year ago

I think if you do lambda x: x.quantile(0.15) it should work. I did the above step of defining the function so that the end product is more readable

Collectives™ on Stack Overflow

Why I am getting Error while using Lambda within Apply

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related