how to read csv file in pyspark?

Question

I am trying to read csv file using pyspark but its showing some error. Can you tell what is the correct process to read csv file?

python code:

from pyspark.sql import *
df = spark.read.csv("D:\Users\SPate233\Downloads\iMedical\query1.csv", inferSchema = True, header = True)

i tried also below one:

sqlContext = SQLContext
df = sqlContext.load(source="com.databricks.spark.csv", header="true", path = "D:\Users\SPate233\Downloads\iMedical\query1.csv")

error:

Traceback (most recent call last):
  File "<pyshell#18>", line 1, in <module>
    df = spark.read.csv("D:\Users\SPate233\Downloads\iMedical\query1.csv", inferSchema = True, header = True)
NameError: name 'spark' is not defined

and

Traceback (most recent call last):
      File "<pyshell#26>", line 1, in <module>
        df = sqlContext.load(source="com.databricks.spark.csv", header="true", path = "D:\Users\SPate233\Downloads\iMedical\query1.csv")
    AttributeError: type object 'SQLContext' has no attribute 'load'

Sagar · Accepted Answer · 2019-11-11 14:38:27Z

1

First you need to create a SparkSession like below

from pyspark.sql import SparkSession


spark = SparkSession.builder.master("yarn").appName("MyApp").getOrCreate()

and your csv needs to be on hdfs then you can use spark.csv

df = spark.read.csv('/tmp/data.csv', header=True)

where /tmp/data.csv is on hdfs

edited Nov 11, 2019 at 14:38

answered Nov 11, 2019 at 13:05

Sagar

4282 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Grigory Skvortsov · Accepted Answer · 2019-11-11 12:30:39Z

0

The simplest to read csv in pyspark - use Databrick's spark-csv module.

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('file.csv')

Also you can read by string and parse to your separator.

reader = sc.textFile("file.csv").map(lambda line: line.split(","))

answered Nov 11, 2019 at 12:30

Grigory Skvortsov

4734 silver badges23 bronze badges

2 Comments

Grigory Skvortsov Over a year ago

Also there are any problem with imports. See this

abhishek Over a year ago

i'm getting this error - AttributeError: 'property' object has no attribute 'format'

Collectives™ on Stack Overflow

how to read csv file in pyspark?

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related