Spark-csv data source: infer data types

Question

I'm experimenting with Spark-CSV package (https://github.com/databricks/spark-csv) for reading csv files into Spark DataFrames.

Everything works but all columns are assumed to be of StringType.

As shown in Spark SQL documentation (https://spark.apache.org/docs/latest/sql-programming-guide.html), for built-in sources such as JSON, the schema with data types can be inferred automatically.

Can the types of columns in CSV file be inferred automatically?

1. StringTypes are a field type in SparkSQL. 2. What you are asking is not very clear, can you be more specific about what you are trying to achieve — eliasah
– eliasah, Commented Apr 19, 2015 at 8:28
I'm asking about automatic type inference, which is available in built-in data sources such as JSON. I.e. if one creates df using sqlContext.jsonFile("...") from json file having say one integer and one string field - these types would be defined in schema. Is this possible with CSV data source format? — Oleg Shirokikh
– Oleg Shirokikh, Commented Apr 19, 2015 at 8:56

Olga · Accepted Answer · 2017-04-14 08:41:34Z

7

Starting from Spark 2 we can use option 'inferSchema' like this: getSparkSession().read().option("inferSchema", "true").csv("YOUR_CSV_PATH")

answered Apr 14, 2017 at 8:41

Olga

911 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

dpeacock · Accepted Answer · 2015-04-19 13:55:35Z

3

Unfortunately this is not currently supported but it would be a very useful feature. Currently they must be declared in DLL. From the documentation we have:

header: when set to true the first line of files will be used to name columns and will not be included in data. All types will be assumed string. Default value is false.

which is what you are seeing.

Note that it is possible to infer schema at query time, e.g.

select sum(mystringfield) from mytable

answered Apr 19, 2015 at 13:55

dpeacock

2,76716 silver badges17 bronze badges

1 Comment

Oleg Shirokikh Over a year ago

Thanks, that's what I thought, thanks for confirming. I agree - this would be extremely nice functionality. Exact type matching may be infeasible, but various heuristics may be possible: type based on the first N records in column, based on random N records, based on N/size fraction, etc

Collectives™ on Stack Overflow

Spark-csv data source: infer data types

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related