How to cast String float to Float in PySpark?

Question

I have the following PySpark dataframe:

df = spark.createDataFrame(
    [
        ('31,2', 'foo'),
        ('33,1', 'bar'),
    ],
    ['cost', 'label']
)

I need to cast the ´cost´ column to float. I do it as follows:

df = df.withColumn('cost', df.cost.cast('float'))

However, as I result I get null values instead of numbers in the cost column.

How can I convert cost to float numbers?

"," is not a valid char for the float. you need to replace with "." — Emma
– Emma, Commented Nov 17, 2022 at 19:39
take a look at regex_replace function. spark.apache.org/docs/3.1.1/api/python/reference/api/… — Emma
– Emma, Commented Nov 17, 2022 at 19:47

CharlieBONS · Accepted Answer · 2022-11-17 19:54:09Z

2

This should work for you.

df = (df.withColumn('cost', F.regexp_replace(df.cost, ',', '.')
        .withColumn('cost', df.cost.cast('float')))

answered Nov 17, 2022 at 19:54

CharlieBONS

1812 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

CharlieBONS Over a year ago

I'm not sure 'float will work'. Best to try DecimalType() form spark.sql.types

Ben Y · Accepted Answer · 2022-11-17 23:08:56Z

1

I think a simple lambda expression should take care of most things.

    df.loc[:, 'cost'] = df.cost.apply(lambda x: float(x.replace(',', '.')))

answered Nov 17, 2022 at 23:08

Ben Y

1,0438 silver badges19 bronze badges

3 Comments

Fluxy Over a year ago

Hmm, I use PySpark, not pandas. Is it also applicable to PySpark?

Ben Y Over a year ago

I haven't used PySpark in the past, but a quick glance at the description on Pypi.org shows that it makes use of numpy and Pandas, so I believe it's applicable. More importantly, have you tried the snippet I provided? Thanks!

Ben Y Over a year ago

Just for reference, I tend to avoid adding additional dependencies, so that's why my answer only uses what I know exists in the language itself.

Collectives™ on Stack Overflow

How to cast String float to Float in PySpark?

2 Answers 2

1 Comment

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related