Converting String to Integer Returns null in PySpark

Question

I am trying to convert a string to integer in my PySpark code.

input = 1670900472389, where 1670900472389 is a string

I am doing this but it's returning null.

df = df.withColumn("lastupdatedtime_new",col("lastupdatedtime").cast(IntegerType()))

I have read the posts on Stack Overflow. They have quotes or commas in their input string causing this. However that's not the case with my input string. Any ideas what's happening?

Azhar Khan · Accepted Answer · 2022-12-13 08:42:16Z

5

The max value that a Java integer can hold is 2147483647 i.e. 32-bits or 2³¹-1.

Use LongType instead:

import pyspark.sql.functions as F
from pyspark.sql.types import LongType

df = spark.createDataFrame(data=[["1670900472389"]], schema=["lastupdatedtime"])

df = df.withColumn("lastupdatedtime_new", F.col("lastupdatedtime").cast(LongType()))

Output:

+---------------+-------------------+
|lastupdatedtime|lastupdatedtime_new|
+---------------+-------------------+
|1670900472389  |1670900472389      |
+---------------+-------------------+

Schema:

root
 |-- lastupdatedtime: string (nullable = true)
 |-- lastupdatedtime_new: long (nullable = true)

edited Dec 13, 2022 at 8:42

answered Dec 13, 2022 at 8:24

Azhar Khan

4,15811 gold badges31 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Converting String to Integer Returns null in PySpark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related