kafka different topics set different partitions

Question

As I know, 'num.partitions' in kafka server.properties will be worked for all topics. Now I want to set partitionNumber=1 for topicA and partitionNumber=2 for topicB. Is that possible to implementation with high level api?

So you're trying to create the topics through the API rather than through the command line scripts as shown in the quickstart? kafka.apache.org/documentation.html#quickstart — Morgan Kenyon
– Morgan Kenyon, Commented Nov 20, 2015 at 13:46
yes.I want to set the partitions for topic with api. Any idea for that? — fcbflying
– fcbflying, Commented Nov 21, 2015 at 13:10

Heejin · Accepted Answer · 2015-11-21 06:45:46Z

2

num.partitions is a value used when a topic is generated automatically. If you generate a topic yourself, you can set any number of partitions as you want.

You can generate a topic yourself with the following command. (replication factor 3 and the number of partitions 2. Capital words are what you have to replace.)

bin/kafka-topics.sh --create --zookeeper ZOOKEEPER_HOSTNAME:ZOOKEEPER_PORT \
--replication-factor 3 --partitions 2 --topic TOPIC_NAME

edited Nov 21, 2015 at 6:45

answered Nov 21, 2015 at 6:40

Heejin

4,5913 gold badges29 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Morgan Kenyon · Accepted Answer · 2015-11-21 17:47:07Z

0

There a configuration value that can be set on a Kafka Broker.

auto.create.topics.enable=true

True is actually the default setting,

Enable auto creation of topic on the server. If this is set to true then attempts to produce data or fetch metadata for a non-existent topic will automatically create it with the default replication factor and number of partitions.

So if you read or write from a non-existent partition as if it existed, if will automatically create one for you. I've never heard of using the high level api to automatically create one.

Looking over the Kafka Protocol Documentation, there doesn't seem to be a provided way to create topics.

answered Nov 21, 2015 at 17:47

Morgan Kenyon

3,1821 gold badge29 silver badges38 bronze badges

3 Comments

fcbflying Over a year ago

Thanks.How about the low level api? Create topic with partitions manually?

Morgan Kenyon Over a year ago

So both the high level and low level api implement the Protocol Documentation that I listed above. So there doesn't seem a way to create a topic through either of the APIs. You've would either have to explicitly create it with a command like Heejin's answer, or setup your default configuration and enable auto topic creation like I said in my answer.

fcbflying Over a year ago

Thanks. That means if I want to create two topics with different partitions.I have to use command line script.

Collectives™ on Stack Overflow

kafka different topics set different partitions

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related