Cannot access to the table with pyspark in azure

Question

I created a Databse with a table inside it in azure synaps which was an external table of sample azure data.

I run the code below to get access to the data in this table through pyspark

df = spark.sql("select * FROM greentaxidb.dbo.taxitable")

when I run the code above I get:

pyspark.sql.utils.AnalysisException: The namespace in session catalog must have exactly one name part: greentaxidb.dbo.taxitable

I checked also the answeres here github.com/apache/iceberg/issues/1144 but there were no usefull information for this case — Amir
– Amir, Commented Jan 10, 2022 at 14:25

NiharikaMoola · Accepted Answer · 2022-01-11 12:14:22Z

1

Load the data into dataframe with saveAsTable to run it in pyspark notebook.

%%pyspark
df = spark.read.load('abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<filename>', format='parquet')
df.write.mode("overwrite").saveAsTable("testdb.test1")

--

%%pyspark
df = spark.sql("select * from testdb.test1")
display(df)

You can go through the links from here and here for more details.

answered Jan 11, 2022 at 12:14

NiharikaMoola

5,1241 gold badge5 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Cannot access to the table with pyspark in azure

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related