Running command in parallel in pyspark

Question

I have small snippet of code in pyspark.

# save data frame as temp table 
df.createOrReplaceTempView("my_table")

# save data frame as csv
df.write.format("csv").save("my_csv")

Now the 1st command executes then only 2nd will execute.

Is there a way where I can trigger both these commands in parallel i.e I want both commands to execute at the same time

blackbishop · Accepted Answer · 2021-02-04 22:31:08Z

2

You can use python multiprocessing

from multiprocessing import Process

def create_temp_view(df):
    df.createOrReplaceTempView("my_table")

def write_df(df):
    df.write.mode("overwrite").format("csv").save("my_csv")


p1 = Process(create_temp_view(df))
p1.start()
p2 = Process(write_df(df))
p2.start()
p1.join()
p2.join()

answered Feb 4, 2021 at 22:31

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Running command in parallel in pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related