Using Scala to Write a DF to a SQL Server Table

Question

I'm trying to use the code below.

import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.connect._

// Aquire a DataFrame collection (val collection)

val config = Config(Map(
  "url"          -> "mysqlserver.database.windows.net",
  "databaseName" -> "MyDatabase",
  "dbTable"      -> "dbo.Clients"
  "user"         -> "username",
  "password"     -> "*********"
))

import org.apache.spark.sql.SaveMode
collection.write.mode(SaveMode.Append).sqlDB(config)

The script is from this link.

https://github.com/Azure/azure-sqldb-spark

I'm running this in a databricks environment. I'm getting these errors:

command-836397363127942:5: error: object sqlDB is not a member of package com.microsoft.azure
import com.microsoft.azure.sqlDB.spark.connect._
                           ^
command-836397363127942:4: error: object sqlDB is not a member of package com.microsoft.azure
import com.microsoft.azure.sqlDB.spark.config.Config
                           ^
command-836397363127942:7: error: not found: value Config
val bulkCopyConfig = Config(Map(
                     ^
command-836397363127942:18: error: value sqlDB is not a member of org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row]
df.write.mode(SaveMode.Append).sqlDB(bulkCopyConfig)

I'm guessing that some kind of library is not installed correctly. I Googled for an answer, but didn't find anything useful. Any idea how to make this work? Thanks.

Have you included the connector dependency azure-sqldb-spark ? It seems it's missing — blackbishop
– blackbishop, Commented Dec 19, 2019 at 21:09
add this and try  <dependency> <groupId>com.microsoft.azure</groupId> <artifactId>azure-sqldb-spark</artifactId> <version>1.0.2</version> </dependency> — Rajasekhar
– Rajasekhar, Commented Dec 20, 2019 at 5:29

CHEEKATLAPRADEEP · Accepted Answer · 2022-04-05 03:59:54Z

1

If you are getting the sqldb error means all other support libraries already imported to your notebook and only the latest JAR with dependencies are missing.

As per the repro, I got the same error message as shown above:

After bit of research, I had found that you will experience this error due to missing JAR with dependencies.

To resolve this issue, you need to download the JAR file from here: https://search.maven.org/artifact/com.microsoft.azure/azure-sqldb-spark/1.0.2/jar

After downloaded the jar file, upload the JAR library into the cluster and install it.

Note: After installing both the libraries, make sure to restart the cluster.

Now, you will be able to run the command successfully.

answered Apr 5, 2022 at 3:59

CHEEKATLAPRADEEP

12.8k1 gold badge22 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Rajasekhar · Accepted Answer · 2019-12-20 05:34:08Z

1

I think you have missed the library

If your using Maven Build Add the following library in pom.xml

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-sqldb-spark</artifactId>
    <version>1.0.2</version>
</dependency>

If your using SBT Build Add the following library in build.sbt

libraryDependencies += "com.microsoft.azure" % "azure-sqldb-spark" % "1.0.2"

answered Dec 20, 2019 at 5:34

Rajasekhar

2,3551 gold badge15 silver badges20 bronze badges

Comments

Stian · Accepted Answer · 2021-02-24 13:22:49Z

0

Have you imported and installed the library in DataBricks?

I found it easiest to import the library using Maven. See this answer: How to install a library on a databricks cluster using some command in the notebook?

Note: You need to install the Library on your cluster and then restart the cluster before you can use it.

answered Feb 24, 2021 at 13:22

Stian

1,4314 gold badges21 silver badges30 bronze badges

Collectives™ on Stack Overflow

Using Scala to Write a DF to a SQL Server Table

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related