Pyspark Dataframe Column - Convert Decimal values represented as string in column

Question

I have the below Pyspark dataframe with Column_1 as string

S.No.      Column_1
1          1.0
2          1.0
3          2.0
4          N
5          N
6          3.0
7          N

I have to convert the numbers in column 1 from 1.0, 2.0 etc to just 1 or 2 or 3. I have to truncate the decimal places. I cannot cast it into int because it has string values too.

Expected Output:

S.No.      Column_1
1          1
2          1
3          2
4          N
5          N
6          3
7          N

anky · Accepted Answer · 2020-05-16 15:58:54Z

3

Seems like you can use regexp_replace here:

df.withColumn("test",F.regexp_replace(F.col("Column_1"),'[.].?','')).show()

+-----+--------+----+
|S.No.|Column_1|test|
+-----+--------+----+
|    1|     1.0|   1|
|    2|     1.0|   1|
|    3|     2.0|   2|
|    4|       N|   N|
|    5|       N|   N|
|    6|     3.0|   3|
|    7|       N|   N|
+-----+--------+----+

answered May 16, 2020 at 15:58

anky

75.3k11 gold badges46 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark Dataframe Column - Convert Decimal values represented as string in column

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related