How to create a PySpark DataFrame from a Python loop

Question

I am looping through multiple webservices which works fine

customers= json.loads(GetCustomers())

for o in customers["result"]:
  if o["customerId"] is not None:
    custRoles = GetCustomersRoles(o["customerId"])
    custRolesObj = json.loads(custRoles)

    if custRolesObj["result"] is not None:
      for l in custRolesObj["result"]:
        print str(l["custId"]) + ", " + str(o["salesAmount"])

This works, and my output from print is also correct. But, now I need to create a DataFrame out of this. I read, we cannot "create a DataFrame with two columns and add row by row while looping".

But how would I solve this?

Update

I hope this is the correct way to create a list?

customers= json.loads(GetCustomers())
result = []

for o in customers["result"]:
  if o["customerId"] is not None:
    custRoles = GetCustomersRoles(o["customerId"])
    custRolesObj = json.loads(custRoles)

    if custRolesObj["result"] is not None:
      for l in custRolesObj["result"]:
          result.append(make_opportunity(str(l["customerId"]), str(o["salesAmount"])))

When this is correct, how to create a Dataframe out of it?

Store your results in a list of tuples (or lists) and then create the spark DataFrame at the end. You can add a row inside a loop but it would be terribly inefficient — pault
– pault, Commented Oct 11, 2018 at 18:57
As @pault stated, I would definitely not add (or append) rows to a dataframe inside of a for loop. It will be terribly inefficient. Much more performant to create the dataframe all at once outside of the loop after assembling your data. On that note, you should include a sample of your data in your OP. — rahlf23
– rahlf23, Commented Oct 11, 2018 at 19:01
@pault: Could you give me a sample for a two column scenario please? — STORM
– STORM, Commented Oct 11, 2018 at 19:10
df = spark.createDataFrame([('a', 1), ('b', 2), ('c', 3)], ["letter", "number"]). Also take a look at this post. — pault
– pault, Commented Oct 11, 2018 at 19:31
Thats what i know now and know how to imlement in theory. But how to do this practically eith a small code snippet. This would be the answer and what i am looking for. — STORM
– STORM, Commented Oct 12, 2018 at 5:46

STORM · Accepted Answer · 2018-10-12 17:06:10Z

2

I solved my problem by using the following code

customers= json.loads(GetCustomers())
result = []

for o in customers["result"]:
  if o["customerId"] is not None:
    custRoles = GetCustomersRoles(o["customerId"])
    custRolesObj = json.loads(custRoles)

    if custRolesObj["result"] is not None:
      for l in custRolesObj["result"]:
          result.append([str(l["customerId"]), str(o["salesAmount"])])

from pyspark.sql import *

df = spark.createDataFrame(result,['customerId', 'salesAmount'])

answered Oct 12, 2018 at 17:06

STORM

4,32614 gold badges65 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to create a PySpark DataFrame from a Python loop

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related