Installing kafka cluster

Question

I want to install 2 node Kafka cluster on Amazon EC2.

I follow the steps from this link: https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04

Also, I want to have zookeeper on both nodes, because If I have it only on one node, if that node dies, my kafka cluster dies.

In step 9 (Installing multi-node cluster), they say that I need to modify zookeeper.connect in kafka server properties, so that it has comma separated list of ip:port for each node where zookeeper is installed.

On the other hand, when I want to create a topic, in the script I only specify 1 zookeeper!

1) Will the other zookeeper node know that the topic has been created?

2) In case that 1 zookeeper node fails, will the other one takeover?

3) `When the failed node goes up again, will it take again the information about topics from the node that stayed alive?

Regards,

Srdjan

If you have only 2 Zookeeper nodes, your Kafka cluster will die anyway if one Zookeeper instance dies. Zookeeper requires a quorum to work (which is n/2 + 1 instances). Thus, if you have 2 Zookeeper nodes and one dies, the one that's left won't work too. The recommended setup is to always have an odd number of ZK nodes. — serejja
– serejja, Commented Jun 16, 2016 at 14:38
Thank you for the comment. Do you happen to know answers to other questions? — Srdjan Nikitovic
– Srdjan Nikitovic, Commented Jun 17, 2016 at 12:55

Matthew McCants · Accepted Answer · 2016-06-17 20:05:53Z

3

You should create a cluster with no less than three nodes. Like Serejja mentioned, it should be odd-numbered for fault-tolerance.

3,5,7,9 etc.

For Kafka, you should specify a --replication-factor when creating the topic. In a three node cluster, it's recommended to set it to two or three.

In this scenario if one of the brokers goes down, the data will get replicated across the available nodes, and then once the unavailable node comes back online, the data will propagate to it.

The Kafka Documentation is fantastic, and I recommend further reading of the Replication topic.

answered Jun 17, 2016 at 20:05

Matthew McCants

2301 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Installing kafka cluster

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related