How to convert DataFrame columns from string to float/double in PySpark 1.6?

Question

In PySpark 1.6 DataFrame currently there is no Spark builtin function to convert from string to float/double.

Assume, we have a RDD with ('house_name', 'price') with both values as string. You would like to convert, price from string to float. In PySpark, we can apply map and python float function to achieve this.

New_RDD = RawDataRDD.map(lambda (house_name, price): (house_name, float(x.price))    # this works

In PySpark 1.6 Dataframe, it does not work:

New_DF = rawdataDF.select('house name', float('price')) # did not work

Until a built in Pyspark function available, how to do achieve this conversion with a UDF? I developed this conversion UDF as follows:

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

def string_to_float(x):
    return float(x)

udfstring_to_float = udf(string_to_float, StringType())
rawdata.withColumn("house name", udfstring_to_float("price"))

Is there a better and much simpler way to achieve the same?

Hristo Iliev · Accepted Answer · 2020-05-09 16:58:20Z

11

According to the documentation, you can use the cast function on a column like this:

rawdata.withColumn("house name", rawdata["price"].cast(DoubleType()).alias("price"))

edited May 9, 2020 at 16:58

Hristo Iliev

74.8k12 gold badges148 silver badges197 bronze badges

answered Feb 28, 2016 at 22:40

Alex

21.9k11 gold badges68 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

makansij Over a year ago

This does not work for me @Jaco. The OP says he is using pyspark 1.6 and the documentation you linked to is 1.3. When i try this on 1.6 i get AttributeError: 'DoubleType' object has no attribute 'alias'

Alex Over a year ago

Do you have the import from pyspark.sql.types import DoubleType ? I am sure I tested this on PySpark 1.6 before posting.

AntiPawn79 Over a year ago

FIX: Should be rawdata.withColumn("house name",rawdata["price"].cast(DoubleType()).alias("price") instead

Egemen Yiğit Kömürcü · Accepted Answer · 2020-05-09 16:52:24Z

4

The answer should be as follows:

>>> rawdata.printSchema()
root
 |-- house name: string (nullable = true)
 |-- price: string (nullable = true)

>>> rawdata=rawdata.withColumn('price',rawdata['price'].cast("float").alias('price'))

>>> rawdata.printSchema()
root
 |-- house name: string (nullable = true)
 |-- price: float (nullable = true)

As it is the shortest one-line code without using any user-defined function. You can see whether it worked correctly by using printSchema() function.

edited May 9, 2020 at 16:52

answered May 9, 2020 at 13:32

Egemen Yiğit Kömürcü

1211 silver badge2 bronze badges

Collectives™ on Stack Overflow

How to convert DataFrame columns from string to float/double in PySpark 1.6?

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related