How to convert pandas dataframe to pyspark dataframe which has attribute to rdd?

Question

Now I am doing a project for my course, and find a problem to convert pandas dataframe to pyspark dataframe . I have produce a pandas dataframe named data_org as follows. enter image description here

And I want to covert it into pyspark dataframe to adjust it into libsvm format. So my code is

from pyspark.sql import SQLContext  
spark_df = SQLContext.createDataFrame(data_org)

However, it went wrong.

TypeError: createDataFrame() missing 1 required positional argument: 'data'

I really do not know how to do. And my python version is 3.5.2 and pyspark version is 2.0.1. I am looking forward to your reply.

Stefan · Accepted Answer · 2020-11-08 02:54:16Z

4

First pass sparkContext to SQLContext:

from pyspark import SparkContext
from pyspark.sql import SQLContext 
sc = SparkContext("local", "App Name")
sql = SQLContext(sc)

then use createDataFrame like below:

spark_df = sql.createDataFrame(data_org)

edited Nov 8, 2020 at 2:54

Stefan

1,96222 silver badges38 bronze badges

answered Mar 29, 2018 at 11:55

Sociopath

13.4k22 gold badges53 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Carmelo Smith Over a year ago

what does sc mean? I fisrt attain my data by the format of pandas dataframe.

Sociopath Over a year ago

sc means sparkContext. if you are running the script using spark-submit, it is initialized by spark

Carmelo Smith Over a year ago

I use the anaconda spyder to run to pyspark code. How to solve the problem in this situation?

Carmelo Smith Over a year ago

Thanks a million !

Carmelo Smith Over a year ago

Can you help me solve my another question?stackoverflow.com/questions/49559331/…

|

Collectives™ on Stack Overflow

How to convert pandas dataframe to pyspark dataframe which has attribute to rdd?

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related