0

I'm trying to read data from GCP kafka through azure databricks but getting below warning and notebook is simply not completing. Any suggestion please?

  WARN NetworkClient: Consumer groupId Bootstrap broker rack disconnected

  Please note I've properly configured SSL properties of truststore etc and I'm able to read data from simple JAVA spring app locally.

Improve this question
4
  • 1
    Have you allowed Azure to connect to the GCP VPC? Please share code as a minimal reproducible example Commented Sep 17, 2022 at 15:18
  • How to allow Azure to connect to GCP VPC? Commented Sep 17, 2022 at 18:02
  • Code is df = spark.readStream \ .format("kafka") \ .option("kafka.bootstrap.servers", ...) \ .option("kafka.security.protocol", "SSL") \ .option("kafka.ssl.truststore.location", <dbfs-truststore-location>) \ .option("kafka.ssl.keystore.location", <dbfs-keystore-location>) \ .option("kafka.ssl.keystore.password", dbutils.secrets.get(scope=<certificate-scope-name>,key=<keystore-password-key-name>)) \ .option("kafka.ssl.truststore.password","") Commented Sep 17, 2022 at 18:09
  • Please edit your question and format the code. You go to your GCP console and edit the network configs to allow Azure connections. Or, you run Databricks in Google Cloud instead since that's where your kafka is Commented Sep 17, 2022 at 21:57
Related questions
4
Databricks Connect: can't connect to remote cluster on azure, command: 'databricks-connect test' stops
2
Can't connect to Azure Data Lake Gen2 using PySpark and Databricks Connect
2
Azure databricks: KafkaUtils createDirectStream causes Py4JNetworkError("Answer from Java side is empty") error
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.