Migrating Kafka setup from Single node cluster to multi node Cluster

Question

Currently, we don't have much load on Kafka. But we expect some decent load in the future. Hence for the cost-cutting reasons, we are planning to have Kafka stack with only one node (One Zookeeper and one Kafka Broker) for the time being and later based on the load we would like to make it a proper cluster (with 3 nodes at least).

Is it ok to start this way (starting with one node and then adding more nodes).

I read somewhere that it can't be done in a seamless fashion. Can someone explain what can go wrong if not?

OneCricketeer · Accepted Answer · 2019-04-23 13:43:50Z

2

You will need downtime to stop Zookeeper + Kafka, edit the configurations to include more servers, then start them back up. In those terms, it is not seamless, as there will be service outages for apps reading/writing. But once everything is started again, those should be okay.

Outside of a local environment, it is definitely recommended to start with at least 3 Zookeepers, even if you have only one broker. A failure of just that one Zookeeper server would be really bad for Kafka... And it is also recommended to have at least 3 brokers for replication and fault-tolerance, despite how much load you have. Having at least one copy of the data is better than none at all.

You do not need to scale Zookeeper evenly with Kafka; e.g. 3 ZK + 1 brokers, then add more brokers later, and don't go more than 5-7 Zookeepers for large clusters.

edited Apr 23, 2019 at 13:43

answered Apr 22, 2019 at 18:37

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

iwekesi Over a year ago

If we go with 3ZK and 1 Broker and later introduce one more broker, what happens to the data on the already existing broker. will it be distributed evenly? can we change the replication factor after adding the new broker?

OneCricketeer Over a year ago

All existing data will stay on the first broker. Both operations of rebalancing and adding replicas must be performed manually. kafka.apache.org/documentation/#basic_ops_cluster_expansion Or you can find external tools to make that more automated/scriptable

Collectives™ on Stack Overflow

Migrating Kafka setup from Single node cluster to multi node Cluster

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related