Scala: Creating a dataframe from a series of lists

Question

i am trying to create a dataframe based on 4 lists i have. i have to use scala only (we can't use SQL for various reasons).

all lists have 3 values, and the column_head list is a list of the column names.

column_head =["a","b","c"]

master_in =[1,2,"dog"]

master_out =[3,4,"cat"]

master_max = [5,6,"llama"]

ive tried:

val values 
=Seq(columns_head,master_in,master_out,master_maxweight)

val master_df= values.toDF()

but i get an exception saying: java.lang.ClassNotFoundException: scala.Any

This is likely because the last value of each list is a STRING value, whereas the first two for each list are INTEGERS.

How do I solve this problem?

I can't import any other libraries outside of:

import org.apache.spark.sql.functions.desc

import org.apache.spark.sql.functions._

case class edges(Source: String, Target: String, Weight: Int)

import spark.implicits._

how do i make a df from the lists i have?

I saw you have first list is the header , in spark you could do write to csv for example with spark.sqlContext.createDataFrame(rdd, schema) and option("header", "true") //Write the header — tree em
– tree em, Commented Mar 17, 2019 at 5:34

Sim · Accepted Answer · 2019-03-17 21:46:04Z

1

The issue you are having stems from the type of data in the different columns.

Because there are both integers and a string in the data, you can't think of it as a list of lists. Well, you can, but then "inner" list will have an element type of Any, which is the closest shared ancestor of Int and String. Of course, Spark can't work with Any. It's too general.

The solution is simple: describe the type of data explicitly using a case class.

case class Data(a: Int, b: Int, c: String)

spark.createDataFrame(Seq(
  Data(1,2,"dog"), Data(3,4,"cat"), Data(5,6,"llama")
))

answered Mar 17, 2019 at 21:46

Sim

13.6k11 gold badges69 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scala: Creating a dataframe from a series of lists

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related