How do I use Python Spark API to specify a dataframe schema by hand?

Question

I would like to specify a schema for spark dataframes in python. After I load the data once, I can print the Schema, I might see something like

df = spark.read.json(datapath)
df.schema

StructType(List(StructField(fldname,StringType,true)))

Having created this python object: df.schema by reading the data, I can now use it to read more. However I think I will wait less if I don't have to first read the data to get the schema - I'd like to persist the schema, even just typing in the schema in my script. For typing it in, I've tried

from pyspark.sql.types import StructType, StructField, StringType

schema = StructType([ StructField('fldname', StringType, True)])

but I get the message

AssertionError: dataType should be DataType

This is spark 2.0.2

Instead of StringType, use StringType().

Tom Lynch
– Tom Lynch

2017-06-19 17:06:48 +00:00
Commented Jun 19, 2017 at 17:06 — Tom Lynch
– Tom Lynch, Commented Jun 19, 2017 at 17:06

koiralo · Accepted Answer · 2017-06-19 17:06:46Z

2

While creating the schema you missed () parenthesis

schema = StructType([ StructField('fldname', StringType(), True)])

In python, you need to construct as StringType() instead of using a singleton.

Hope this solved the issue.

answered Jun 19, 2017 at 17:06

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How do I use Python Spark API to specify a dataframe schema by hand?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related