The Prometheus jmx_exporter project gives an official example configuration for Kafka here:
https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-0-8-2.yml
Is there any similar configuration for Kafka Connect to export Kafka Connect JMX metrics to Prometheus?
The current solution that I'm using is the official Confluent Helm Charts and Docker image which has built-in Prometheus metric export:
https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka-connect
For reference, if you want a non-Docker, non-Kubernetes solution, a previous approach that works 100% is as follows:
1) Define this file kafka-connect-jmx.yaml. I am storing this config file in /usr/share/, you can put it elsewhere, but you will need to change the path reference below.
whitelistObjectNames: ["kafka.connect:*"]
lowercaseOutputName: false
lowercaseOutputLabelNames: false
rules:
- pattern: 'status: running'
value: 1
- pattern: 'status: *'
value: 0
- pattern: ".*"
2) Download https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar. I put it in /usr/share/, you can put it elsewhere, but you will need to update the path in step 3.
3) When you run Kafka Connect connect-distributed make sure this environment variable is set and the paths match those used in steps 1+2
KAFKA_OPTS=-javaagent:/usr/share/jmx_prometheus_javaagent-0.12.0.jar=7072:/usr/share/kafka-connect-jmx.yaml
I was looking for the same and found below. Hope that helps :)
https://raw.githubusercontent.com/zenreach/docker-kafka-connect/master/jmx_exporter.yaml
lowercaseOutputName: true
rules:
- pattern: kafka.connect<type=connect-node-metrics, client-id=(.+), node-id=(.+)><>([a-z-]+)
name: kafka_connect_node_$3
labels:
clientId: "$1"
nodeId: "$2"
- pattern: kafka.connect<type=connect-coordinator-metrics, client-id=(.+)><>([a-z-]+)thu
name: kafka_connect_coordinator_$2
labels:
clientId: "$1"
- pattern: kafka.consumer<type=consumer-node-metrics, client-id=(.+), node-id=(.+)><>([a-z-]+)
name: kafka_connect_consumer_node_$3
labels:
clientId: "$1"
nodeId: "$2"
- pattern: kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+), topic=(.+)><>([a-z-]+)
name: kafka_connect_consumer_fetch_manager_$3
labels:
clientId: "$1"
topic: "$2"
- pattern: kafka.producer<type=producer-topic-metrics, client-id=(.+), topic=(.+)><>([a-z-]+)
name: kafka_connect_producer_topic_$3
labels:
clientId: "$1"
topic: "$2"
- pattern: kafka.producer<type=producer-node-metrics, client-id=(.+), node-id=(.+)><>([a-z-]+)
name: kafka_connect_producer_node_$3
labels:
clientId: "$1"
nodeId: "$2"
