Kafka Connect Dynamic Routing without Kafka Stream

Question

I am working on a Integration layer, where I want to use Kafka Connect between Kafka Topics to achieve dynamic routing.

I want to know if it is possible to use Kafka Connect to connect between 2 Topics (where Kafka Connect act consumer and then producer) and achieve dynamic routing between topics using Message header without Kafka Stream.

As Kafka Stream also act as a consumer and needs to run on the client application (outside Kafka Broker and Kafka Connect). While I want Integration Layer have all the routing internally (Kafka Connect + Kafka Routing).

OneCricketeer · Accepted Answer · 2023-09-25 13:01:09Z

1

This is exactly what MirrorMaker2 is, however it is not recommended to use it within the same cluster without disabling its internal state management of heartbeats and offset syncs

Apache Camel connector with Kafka source and sink tasks may work better.

If you just don't want to write Kafka Steams code, ksqlDB is another option that abstracts that away

answered Sep 25, 2023 at 13:01

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Kafka Connect Dynamic Routing without Kafka Stream

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related