Cannot convert a list of int + array(int) into a pyspark dataframe

Question

I'm trying to convert an int ID, and an array of 3 ints into a dataframe with 2 columns to then union with another dataframe in pyspark;

However I'm just getting error after error related to schema and nothing seems to work. I'm not sure why this is.

emp_rdd = spark.sparkContext.emptyRDD()
schema = StructType([
    StructField("id", IntegerType(), True),
    StructField("data", ArrayType(IntegerType()), True),])
df = spark.createDataFrame(data=emp_rdd, schema=schema)

columns = ['id','data']
for i in range(10):     
  data = [id, data1]
  newRows = spark.createDataFrame(data,columns) 
  df= df.union(newRows)

This is giving me this error;

Can not infer schema for type: <class 'int'>

Any help would be appreciated

blackbishop · Accepted Answer · 2021-11-14 20:13:39Z

1

The reason you get this error is that in your for-loop, you're passing the param data as a simple list while spark.createDataFrame expects an iterable of lists or tuples.

Try changing it to :

data = [(id, data1)]

Example:

for i in range(5):
    data = [(i, [i + 1, i + 2, i + 3])]
    newRows = spark.createDataFrame(data, columns)
    df = df.union(newRows)

df.show()

#+---+---------+
#| id|     data|
#+---+---------+
#|  0|[1, 2, 3]|
#|  1|[2, 3, 4]|
#|  2|[3, 4, 5]|
#|  3|[4, 5, 6]|
#|  4|[5, 6, 7]|
#+---+---------+

answered Nov 14, 2021 at 20:13

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Cannot convert a list of int + array(int) into a pyspark dataframe

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related