Custom Environment Variables for Kafka Connect via Docker

Question

Is there a way to provide custom variables via Docker-Compose that can be referenced within a Kafka Connector config?

I have the following setup in my docker-compose.yml:

- "sql_server=1.2.3.4"
- "sql_database=db_name"
- "sql_username=some_user"
- "sql_password=nahman"
- "sql_applicationname=kafka_connect"

Here is my .json configuration file:

{
  "name": "vwInv_Tran_Amounts",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "src.consumer.interceptor.classes": "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor",
    "tasks.max": 2,
    "connection.url": "jdbc:sqlserver://${sql_server};database=${sql_database};user=${sql_username};password={sql_password};applicationname={sql_applicationname}",
    "query": "SELECT * FROM vwInv_Tran_Amounts",
    "mode": "timestamp",
    "topic.prefix": "inv_tran_amounts",
    "timestamp.column.name": "timestamp",
    "incrementing.column.name": "Inv_Tran_ID"
  }
}

I was able to reference the environment variables using this method with Elastic Logstash, but it doesn't appear to work here.

Whenever loading it via curl I receive:

The connection string contains a badly formed name or value. for configuration Couldn't open connection to jdbc:sqlserver://${sql_server};database=${sql_database};user=${sql_username};password={sql_password};applicationname={sql_applicationname}\nInvalid value com.microsoft.sqlserver.jdbc.SQLServerException: The connection string contains a badly formed name or value.

EDIT///////// I tried prefixing environment varibles like CONNECT_SQL_SERVER and that didn't work.

On second thought, I suppose this really isn't a docker or kafka issue.. as I'm trying to inject the variable into JSON, and then using cURL. Obviously the variable wouldn't be injected there.. back to the drawing board. — Jeff Beagley
– Jeff Beagley, Commented Nov 2, 2018 at 15:22
Coming back to this years later, I feel like the trick here anytime a program/service can't reference environment variables, the trick is to add envsubstr and use that to build out the new json file with applied env vars. — Jeff Beagley
– Jeff Beagley, Commented Jul 28, 2022 at 15:35

OneCricketeer · Accepted Answer · 2018-11-02 20:54:30Z

1

I feel like you are looking for Externalizing Kafka Connect secrets, but that would require mounting a file, not using env vars.

JSON Connector config files aren't loaded on Docker container startup. I made this issue to see if this would be possible.

You would have to template out the JSON file externally, then HTTP-POST them to the port exposed by the container.

Tried prefixing environment varibles like CONNECT_SQL_SERVER

Those values would go into the Kafka Connect Worker properties, not the properties that need to be loaded by a specific connector task.

edited Nov 2, 2018 at 20:54

answered Nov 2, 2018 at 20:48

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Custom Environment Variables for Kafka Connect via Docker

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related