I have a value in a variable - ID as 1 and a list of ten values say
LIST1 = [1,2,3,4,5,6,7,8,9,10].
Now I wanted to create a pyspark data frame as below:
ID LIST
1 1
1 2
1 3
1 4
1 5
1 6
1 7
1 8
1 9
1 10
NOTE: The List1 length is dynamic, based on its length we need to have the rows accordingly.
It depends on whether the ID is constant or you will even have List2 with ID 2 and then want to union for both into one DataFrame.
As far as the constant is concerned, there are two options:
ID = 1
LIST1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
source = list(map(lambda x: (ID, x), LIST1))
# source: [(1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (1, 7), (1, 8), (1, 9), (1, 10)]
df = spark.createDataFrame(source, ['ID', 'LIST'])
df.show()
# +---+----+
# | ID|LIST|
# +---+----+
# | 1| 1|
# | 1| 2|
# | 1| 3|
# | 1| 4|
# | 1| 5|
# | 1| 6|
# | 1| 7|
# | 1| 8|
# | 1| 9|
# | 1| 10|
# +---+----+
or
from pyspark.sql.functions import lit
ID = 1
LIST1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
source = list(map(lambda x: (x,), LIST1))
# createDataFrame needs iter of iters -> list/tuple of lists/tuples
df = spark.createDataFrame(source, ['LIST'])
df.withColumn('ID', lit(ID)).show()
+----+---+
|LIST| ID|
+----+---+
| 1| 1|
| 2| 1|
| 3| 1|
| 4| 1|
| 5| 1|
| 6| 1|
| 7| 1|
| 8| 1|
| 9| 1|
| 10| 1|
+----+---+
