Filling nulls values from a CSV file issue-spark

Question

I'm using Scala and Apache Spark 2.3.0 with a CSV file. I'm doing this because when I try to use the csv for k means it tells me that I have null values but it keeps appearing the same issue even if I try to fill those nulls

scala>val df = sqlContext.read.format("com.databricks.spark.csv")
    .option("header", "true")
    .option("delimiter",";")
    .schema(schema).load("33.csv")

scala> df.na.fill(df.columns.zip(
  df.select(df.columns.map(mean(_)): _*).first.toSeq
).toMap)

scala> val featuresCols = Array("LONGITUD","LATITUD")
featuresCols: Array[String] = Array(LONGITUD, LATITUD)

scala> val featureCols = Array("LONGITUD","LATITUD")
featureCols: Array[String] = Array(LONGITUD, LATITUD)

scala> val assembler = new VectorAssembler().setInputCols(featureCols).setOutputCol("features")
assembler: org.apache.spark.ml.feature.VectorAssembler = vecAssembler_440117601217

scala> val df2 = assembler.transform(df)
df2: org.apache.spark.sql.DataFrame = [ID_CALLE: int, TIPO: int ... 6 more fields]

scala> df2.show

Caused by: org.apache.spark.SparkException: Values to assemble cannot be null

Michael West · Accepted Answer · 2018-10-10 15:34:56Z

1

Looks like you did na.fill() but didn't assign it to a DataFrame.

Try val nonullDF = df.na.fill(...)

answered Oct 10, 2018 at 15:34

Michael West

1,7161 gold badge16 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

D.per Over a year ago

I alredy try it but, when I try to do the VectorAssembler to transform and return a new dataframe I still have the same issue val nonullDF = df.na.fill(df.columns.zip(df.select(df.columns.map(mean()):*).first.toSeq).toMap)

Michael West Over a year ago

I am unable to replicate your issue. Can you provide runnable code and data that creates the issue so that I can investigate it?

Collectives™ on Stack Overflow

Filling nulls values from a CSV file issue-spark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related