Write Data as Json from DataFrame to Azure Blob Storage

Question

I have some data in dataframe which i have to convert to json and store it into Azure Blob Storage. Is there any way to achieve this? Below are the steps which i have tried. I am trying it from spark-shell.

val df = spark.sql("select * from historic_data.all_historic_data").show()

spark.conf.set("fs.azure.account.key.<STORAGE_ACCOUNT_NAME>.blob.core.windows.net","STORAGE_ACCOUNT_KEY")

df.write.mode(SaveMode.Append).json("wasbs://BlobStorageContainer@<STORAGE_ACCOUNT_NAME>.blob.core.windows.net/<FOLDER_PATH_OF BLOB>/")

While running the write command i am getting below error

org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: The specifed resource name contains invalid characters.
  at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:2208)
  at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatusInternal(NativeAzureFileSystem.java:2673)
  at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2618)
  at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1448)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:92

Is there anything i have missed while providing blob details? below is screen shot of my storage account :

I havent seen any similiar kind of question here which will WRITE from dataframe as Json into Azure Blob.

Peter Bons · Accepted Answer · 2019-07-24 12:28:54Z

1

The specifed resource name contains invalid characters.

Message is clear about the problem. You are using uppercase characters in the container name, which is not allowed, so change BlobStorageContainer to blobstoragecontainer.

A full list of rules can be found here.

answered Jul 24, 2019 at 12:28

Peter Bons

30k4 gold badges64 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Antony Over a year ago

Thank you..it worked.. correct as

df.write.mode(SaveMode.Append).json("wasbs://<container_name@<storage_account_name>.blob.core.windows.net/<folder_path_of blob>/")

and worked fine.

Collectives™ on Stack Overflow

Write Data as Json from DataFrame to Azure Blob Storage

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related