How can I convert Spark dataframe column to Numpy array efficiently?

Question

I have a Spark dataframe with around 1 million rows. I am using pyspark and have to apply box-cox transformation from scipy library on each column of the dataframe. But the box-cox function allows only 1-d numpy array as input. How can I do this efficiently?

Is numpy array distributed on spark or it collects all the elements to single node on which driver program is running?

suppose df is my dataframe with column as C1 then, I want to perform the operation similar to this

stats.boxcox(df.select("C1"))

There is pretty much no case when you can benefit from having Spark DataFrame and be able process individual columns using Numpy. Basically either your data is small enough (cleaned, aggregated) that you can process it locally by converting to Pandas for example or you need a method that can work on distributed data which is not something that can be typically done with Numpy alone. — zero323
– zero323, Commented Jul 11, 2016 at 23:04

Community · Accepted Answer · 2017-05-23 11:51:27Z

0

The dataframes/RDD in Spark allow abstracting from how the processing is distributed.

To do what you require, I think a UDF can be very useful. Here you can see an example of its use:

Functions from Python packages for udf() of Spark dataframe

edited May 23, 2017 at 11:51

CommunityBot

11 silver badge

answered Jul 10, 2016 at 18:42

Josemy

8481 gold badge13 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sandeep Veerlapati Over a year ago

Thanks for the reply. I have to apply the following function from scipy library, which accepts only ndarray as input not the single element. stats.boxcox(x) where x is 1-d numpy array

moufkir · Accepted Answer · 2018-02-27 06:25:27Z

0

I have a workaround that solve the issue but not sure is the optimal solution in term of performance as you are switching between pyspark and pandas dataframes:

dfpd = df.toPandas()
colName = 'YOUR_COLUMN_NAME'
colBCT_Name = colName + '_BCT'
print colBCT_Name
maxVal = dfpd[colName][dfpd[colName].idxmax()]
minVal = dfpd[colName][dfpd[colName].idxmin()]
print maxVal
print minVal

col_bct, l = stats.boxcox(dfpd[colName]- minVal +1)
col_bct = col_bct*l/((maxVal +1)**l-1)
col_bct =pd.Series(col_bct)
dfpd[colBCT_Name] = col_bct
df = sqlContext.createDataFrame(dfpd)
df.show(2)

answered Feb 27, 2018 at 6:25

moufkir

818 bronze badges

Collectives™ on Stack Overflow

How can I convert Spark dataframe column to Numpy array efficiently?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related