How to create connectors for Kafka-connect on Kubernetes?

Question

I am deploying Kafka-connect on Google Kubernetes Engine (GKE) using cp-kafka-connect Helm chart in distributed mode.

A working Kafka cluster with broker and zookeeper is already running on the same GKE cluster. I understand I can create connectors by sending post requests to http://localhost:8083/connectors endpoint once it is available. However, Kafka-connect container goes into RUNNING state and then starts loading the jar files and till all the jar files are loaded the endpoint mentioned above is unreachable.

I am looking for a way to automate the step of manually exec the pod, check if the endpoint is ready and then send the post requests. I have a shell script that has a bunch of curl -X POST requests to this endpoint to create the connectors and also have config files for these connectors which work fine with standalone mode (using Confluent platform show in this confluent blog).

Now there are only two ways to create the connector:

Somehow identify when the container is actually ready (when the endpoint has started listening) and then run the shell script containing the curl requests
OR use the configuration files as we do in standalone mode (Example: $ <path/to/CLI>/confluent local load connector_name -- -d /connector-config.json)

Which of the above approach is better?

Is the second approach (config files) even doable with distributed mode?

If YES: How to do that?
If NO: How to successfully do what is explained in the first approach?

EDIT: With reference to his github issue(thanks to @cricket_007's answer below) I added the following as the container command and connectors got created after the endpoint gets ready:

...
command:
  - /bin/bash
  - -c
  - |
    /etc/confluent/docker/run &
    echo "Waiting for Kafka Connect to start listening on kafka-connect  "
    while : ; do
      curl_status=`curl -s -o /dev/null -w %{http_code} http://localhost:8083/connectors`
      echo -e `date` " Kafka Connect listener HTTP state: " $curl_status " (waiting for 200)"
      if [ $curl_status -eq 200 ] ; then
        break
      fi
      sleep 5
    done
    echo -e "\n--\n+> Creating Kafka Connector(s)"
    /tmp/scripts/create-connectors.sh
    sleep infinity
...

/tmp/scripts/create-connectors.sh is a script mounted externally containing a bunch of POST requests using CURL to the Kafka-connect API.

OneCricketeer · Accepted Answer · 2020-02-01 15:27:26Z

1

confluent local doesn't interact with a remote Connect cluster, such as one in Kubernetes.

Please refer to the Kafka Connect REST API

You'd connect to it like any other RESTful api running in the cluster (via a Nodeport, or an Ingress/API Gateway for example)

the endpoint mentioned above is unreachable.

Localhost is the physical machine you're typing the commands into, not the remote GKE cluster

Somehow identify when the container is actually ready

Kubernetes health checks are responsible for that

kubectl get services

there are only two ways to create the connector

That's not true. You could additional run Landoop's Kafka Connect UI or Confluent Control Center in your cluster to point and click.

But if you have local config files, you could also write code to interact with the API

Or try and see if you can make a PR for this issue

https://github.com/confluentinc/cp-docker-images/issues/467

edited Feb 1, 2020 at 15:27

answered Feb 1, 2020 at 14:04

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Amit Yadav Over a year ago

Most of the points mentioned in this answer are already there in the question itself. I understand the use of confluent local but as mentioned in the question, I am searching for a way to create the connectors using config files (in distributed mode) or maybe automate the step to 1. Check if the endpoint is reachable (because it takes some time to load the configurations), 2. If yes, then make the post requests. Would you mind elaborating the if you have local config files, you could also write code to interact with the API point please?

OneCricketeer Over a year ago

Bash? Something like find configs/ -name "*.json" -exec curl -X POST -d@{} http://connect-k8s-svc:8083 \;

OneCricketeer Over a year ago

Or write python, Java, etc. It's just http calls

Amit Yadav Over a year ago

Thank you for pointing me to the issue. I think to change the command var of the YAML manifest to wait for the endpoint to be ready infinitely and then start will do the trick. let me give that a try and let you know the result. Also, I am now interested to know what kind (format) of messages I would have to put inside the config topic in order to create the connectors. Would you mind shedding some light on that (or point to a documentation/issue)?

OneCricketeer Over a year ago

They're in JSON format. I've not actually done it before, but you're welcome to consume those internal topics at any point to inspect them

|

Collectives™ on Stack Overflow

How to create connectors for Kafka-connect on Kubernetes?

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related