Databricks- Convert Python DataFrame to Scala DataFrame

Question

I have a dataframe in python, df, that i want to pass to be able to use in % scala.

I have tried -

%python
pyDf.createOrReplaceTempView("testDF") // error message

Venu A Positive · Accepted Answer · 2022-04-26 12:03:03Z

1

it's not too difficult. I am sharing a sample code pls try. It's working in Pycharm or databricks.

from pyspark.sql import *

import pandas as pd

spark = SparkSession.builder.master("local").appName("testing").getOrCreate()

data = [['venu', 50], ['renu', 45], ['anu', 54],['bhanu',14]]

Create the pandas DataFrame

pdf= pd.DataFrame(data, columns = ['Name', 'Age'])

print(pdf)

sparkDF=spark.createDataFrame(pdf)

sparkDF.printSchema()

sparkDF.show()

answered Apr 26, 2022 at 12:03

Venu A Positive

3,0722 gold badges31 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

How can you access python object from scala ?

Alex Ott · Accepted Answer · 2022-04-26 06:39:38Z

1

Just query it with spark.sql:

val scalaDf = spark.sql("select * from testDF")

answered Apr 26, 2022 at 6:39

Alex Ott

88.1k10 gold badges110 silver badges157 bronze badges