Read multiple json files from blob storage to dataframe using pyspark in databricks

Question

I am trying to get all the json files stored in a single container in a subfolder in blob storage. I have setup the environment in databricks and have the connection linked. Currently I am using this code

df = spark.read.json("wasbs://container_name@blob_storage_account.blob.core.windows.net/sub_folder/*.json")

but I am getting just the first file and not all the json files present in the subfolder even after including the wildcard /*.json.

I am trying to get all the files from the subfolder in a single dataframe and store as a table in sql database.

Can someone assist on what I am missing.

@pltc because it's showing just the first file data when I use df.display(). Is there better way to check if I have the data for all the files? — StupendousEnzio
– StupendousEnzio, Commented Nov 3, 2021 at 2:21
huh, display only shows an limited amount of data. Did you try querying the data? — SakuraFreak
– SakuraFreak, Commented Nov 3, 2021 at 2:29
databricks only display first 1000 records. You should counting instead — pltc
– pltc, Commented Nov 3, 2021 at 6:15

RamaraoAdapa · Accepted Answer · 2021-11-03 10:33:58Z

1

I have tested in my environment.

I have 3 json blob files inside the subfolder of my container in storage account. I am able to read all the blob json files in a single data frame

You can use the below code to display all json the files from the subfolder in a single data frame

df = spark.read.json("wasbs://container_name@blob_storage_account.blob.core.windows.net/sub_folder/*.json")
df.show()

answered Nov 3, 2021 at 10:33

RamaraoAdapa

3,1772 gold badges9 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Read multiple json files from blob storage to dataframe using pyspark in databricks

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related