0

I have an architectural question. I have an application which is subscribed to a log compacted Kafka topic. I have to process each event and store into a persistent datastore. I am planning to run the app in 4 instances with the same group-id so that the partitions are distributed with all the nodes.

During the lifetime of the application, there will be a need for

  • Resetting consumer to start from 0th offset
  • Pausing the consumer for a while
  • Resuming a paused consumer

The best way to achieve this was to implement an API which can be called from a script. The API will have an access to the KafkaConsumer object and will call the pause operation.

Now, the issue I see with this approach is there isn't a way I know which can ensure all of the 4 instances. How can I make sure that one API (or N API where N = number of instances) call will do this operation?

Any help on this would be appreciated.

Improve this question

1 Answer 1

Reset to default
1

If your problem is to basically control all of the instances of Kafka processing nodes, well, you already have Kafka. Why not introduce a "command" topic which all nodes subscribe to? You can send one command from your "api" that all the nodes will receive (as long as they have different group ids).

Improve this answer
2
  • Thank you Robert for suggesting an approach. Would you be able to point to some reference or example of a command topic? Commented May 23, 2020 at 12:15
  • I think you were referring to : kafka.apache.org/documentation/#kafka_mq Commented May 23, 2020 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.