How to change column types in Spark SQL's(In java) DataFrame?

Question

I am trying to convert a column data type from long to int in spark sql using java, I have seen some of example in scala and trying out but, Its not wotking

df.withColumn("is_open",  toInt("is_open"));

where do I need to change. thanks in advance

Not working, error is showing "The method col(String) is undefined for the type" @philantrovert — Bhagesh Arora
– Bhagesh Arora, Commented Jan 22, 2019 at 10:21

Constantine · Accepted Answer · 2019-01-22 10:20:04Z

3

You can make use of the cast function.

scala> val df = spark.range(10)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> df.withColumn("new_col", col("id").cast("string"))
res4: org.apache.spark.sql.DataFrame = [id: bigint, new_col: string]

answered Jan 22, 2019 at 10:20

Constantine

1,41615 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Bhagesh Arora Over a year ago

Not working in java, error is showing "The method col(String) is undefined for the type" @Constantine,

Constantine Over a year ago

Can you check the spark documentation ? You can either use col or column. Please import it if not.

Constantine Over a year ago

You should be able to use it after import static org.apache.spark.sql.functions.col;

Bhagesh Arora Over a year ago

yes, now error is gone by importing static function.col. But In console Info is showing CatalystSqlParser: Parsing command: int, and when we see using df.printschema(), that datatype is not converted.

pheeleeppoo Over a year ago

This because you have to remember to update the Object: df = df.withColumn("new_col", col("id").cast("string"))

|

Volodymyr Zubariev · Accepted Answer · 2019-02-28 18:56:04Z

2

df = df.withColumn("is_open", df.col("is_open").cast("int"));

Please be aware, this cast is applying after the data is computed on previous step. If the previous step is select or something similar, it will compute into original type first, and then convert to new type on the next step. This will not solve an issue with selecting into original type.

answered Feb 28, 2019 at 18:56

Volodymyr Zubariev

4434 silver badges9 bronze badges

Collectives™ on Stack Overflow

How to change column types in Spark SQL's(In java) DataFrame?

2 Answers 2

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related