Generate a PySpark DataFrame using list comprehension

Question

I'm trying to generate a DataFrame with the following list of column names:

cols = [
    'name',
    'age',
    'team',
    'column1',
    'column2',
    'column3',
    'column4',
    'column5',
    'column6',
]

rows = [Row(**{k: '1776-07-04'}) for k in cols]
df = spark.createDataFrame(rows)

If I run df.columns, the columns from the list above are returned as expected. But when I run df.show, I get the following error:

Caused by: java.lang.IllegalStateException: Input row doesn't have expected number of values required by the schema. 9 fields are required while 1 values are provided.

And so ultimatley I (somewhat) understand why i'm getting this error, but I was under the impression that 1776-07-04 would just be assigned to any/all values. What am I missing here?

rogue-one · Accepted Answer · 2017-06-23 23:14:18Z

2

This is because you are creating a Dataframe with 9 Rows but each row has only data for one of the columns.

the right way to create a single row with all 9 columns assigned a value of '1776-07-04' is

>>> df = spark.createDataFrame([Row(**{k:'1776-07-04' for k in cols})], cols)
>>> df.show()
+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|      name|       age|      team|   column1|   column2|   column3|   column4|   column5|   column6|
+----------+----------+----------+----------+----------+----------+----------+----------+----------+
|1776-07-04|1776-07-04|1776-07-04|1776-07-04|1776-07-04|1776-07-04|1776-07-04|1776-07-04|1776-07-04|
+----------+----------+----------+----------+----------+----------+----------+----------+----------+

answered Jun 23, 2017 at 23:14

rogue-one

11.6k8 gold badges56 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

too_many_questions Over a year ago

Thanks for the explanation.

Collectives™ on Stack Overflow

Generate a PySpark DataFrame using list comprehension

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related