Ways to Plot Spark Dataframe without Converting it to Pandas

Question

Is there any way to plot information from Spark dataframe without converting the dataframe to pandas?

Did some online research but can't seem to find a way. I need to automatically save these plots as .pdf, so using the built-in visualization tool from databricks would not work.

Right now, this is what I'm doing (as an example):

# df = some Spark data frame 
df = df.toPandas()
df.plot()
display(plt.show())

I want to produce line graphs, histograms, bar charts and scatter plots without converting my dataframe to pandas dataframe. Thank you!

Gravity · Accepted Answer · 2020-01-16 10:53:10Z

19

The display function is only available in databricks kernel notebook, not in spark

answered Jan 16, 2020 at 10:53

Gravity

2292 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

sunil karki · Accepted Answer · 2022-12-09 11:35:50Z

If the spark dataframe 'df' (as asked in question) is of type 'pyspark.pandas.frame.DataFrame', then try the following:

# Plot spark dataframe
df.column_name.plot.pie()

where column_name is one of the columns in the spark dataframe 'df'.

You can try finding the type of 'df' by

type(df)

There are other functions like

pyspark.pandas.DataFrame.plot.line

pyspark.pandas.DataFrame.plot.bar

pyspark.pandas.DataFrame.plot.scatter

This can be found on the apache spark docs: https://spark.apache.org/docs/3.2.1/api/python/reference/pyspark.pandas/api/pyspark.pandas.DataFrame.plot.bar.html

If the spark dataframe 'df' is of type 'pyspark.sql.dataframe.DataFrame', then try the following:

# Import pyspark.pandas
import pyspark.pandas as ps

# Convert pyspark.sql.dataframe.DataFrame to pyspark.pandas.frame.DataFrame
temp_df = ps.DataFrame( df ).set_index('column_name')

# Plot spark dataframe
temp_df.column_name.plot.pie()

Note: There could be other better ways to do it as well. If there are kindly suggest them in the comment.

Peter Pan · Accepted Answer · 2019-07-30 07:27:54Z

1

Just to use display(<dataframe-name>) function with a Spark dataframe as the offical document Visualizations said as below.

Then, to select the plot type and change its options as the figure below to show a chart with spark dataframe directly.

If you want to show the same chart as the pandas dataframe plot of yours, your current way is the only way.

answered Jul 30, 2019 at 7:27

Peter Pan

24.2k4 gold badges31 silver badges47 bronze badges

2 Comments

DieterDP Over a year ago

This does not seem to work for me in Jupyter notebooks. Is this answer specifically for Databricks notebooks?

Sander Vanden Hautte Over a year ago

Yes, it is for Databricks only.

CHEEKATLAPRADEEP · Accepted Answer · 2025-08-27 14:35:49Z

0

Azure Databricks introduce native plotting in PySpark with Databricks Runtime 17.0

Note: With Spark native plotting, you no longer need to convert a data frame to a pandas object to create charts in code.

Example: using PySpark Plotting df.plot.line(), df.plot.bar() and df.plot.pie()

For more details, refer to PySpark Native Plotting

edited Aug 27 at 14:35

answered Aug 27 at 13:52

CHEEKATLAPRADEEP

12.8k1 gold badge22 silver badges42 bronze badges

Collectives™ on Stack Overflow

Ways to Plot Spark Dataframe without Converting it to Pandas

4 Answers 4

Comments

Comments

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related