pyspark reading csv using pandas, how to keep header

Question

I am reading in a csv using pandas chunks functionality. It works, except for I am not able to retain headers. Is there a way/option to do this? here is sample code:

import pyspark
import pandas as pd
sc = pyspark.SparkContext(appName="myAppName")
spark_rdd = sc.emptyRDD()

# filename: csv file
chunks = pd.read_csv(filename, chunksize=10000)
for chunk in chunks:
    spark_rdd +=  sc.parallelize(chunk.values.tolist())

    #print(chunk.head())
    #print(spark_rdd.toDF().show())
    #break

spark_df = spark_rdd.toDF()
spark_df.show()

Gaurav Dhama · Accepted Answer · 2017-02-13 22:55:41Z

1

Try this :

import pyspark
import pandas as pd
sc = pyspark.SparkContext(appName="myAppName")
spark_rdd = sc.emptyRDD()

# Read ten rows to get column names
x = pd.read_csv(filename,nrows=10)
mycolumns = list(x)

# filename: csv file
chunks = pd.read_csv(filename, chunksize=10000)
for chunk in chunks:
    spark_rdd +=  sc.parallelize(chunk.values.tolist())

spark_df = spark_rdd.map(lambda x:tuple(x)).toDF(mycolumns)
spark_df.show()

answered Feb 13, 2017 at 22:55

Gaurav Dhama

1,3469 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

muon Over a year ago

for reading headers, x = pd.read_csv(filename,nrows=1) should suffice?

Gaurav Dhama Over a year ago

I agree its arbitrary, won't matter practically if you take 1,5 or 10 rows as long as you take atleast one.

muon · Accepted Answer · 2017-02-13 21:56:23Z

0

I ended up using databricks' spark-csv

sc = pyspark.SparkContext()
sql = pyspark.SQLContext(sc)

df = sql.read.load(filename, 
                 format='com.databricks.spark.csv', 
                 header='true', 
                 inferSchema='true')

answered Feb 13, 2017 at 21:56

muon

14.2k13 gold badges74 silver badges94 bronze badges

Collectives™ on Stack Overflow

pyspark reading csv using pandas, how to keep header

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related