How to programmatically create topics using kafka-python?

Question

I am getting started with Kafka and fairly new to Python. I am using this library named kafka-python to communicate with my Kafka broker. Now I need to dynamically create a topic from my code, from the docs what I see is I can call create_topics() method to do so, however I am not sure, how will I get an instance of this class. I am not able to understand this from the docs.

Can some one help me with this?

Giorgos Myrianthous · Accepted Answer · 2020-07-13 09:00:26Z

8

You first need to create an instance of KafkaAdminClient. The following should do the trick for you:

from kafka.admin import KafkaAdminClient, NewTopic


admin_client = KafkaAdminClient(
    bootstrap_servers="localhost:9092", 
    client_id='test'
)

topic_list = [NewTopic(name="example_topic", num_partitions=1, replication_factor=1)]
admin_client.create_topics(new_topics=topic_list, validate_only=False)

Alternatively, you can use confluent_kafka client which is a lightweight wrapper around librdkafka:

from confluent_kafka.admin import AdminClient, NewTopic


admin_client = AdminClient({"bootstrap_servers": "localhost:9092"})
topic_list = [NewTopic("example_topic", 1, 1)]
admin_client.create_topics(topic_list)

edited Jul 13, 2020 at 9:00

answered Apr 4, 2019 at 10:34

Giorgos Myrianthous

40.4k21 gold badges156 silver badges175 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Joel Mata Over a year ago

You may need to "read" the topic before making sure its created: ``` res = admin_client.create_topics(topic_list) for topic, f in res.items(): f.result() ```

Collectives™ on Stack Overflow

How to programmatically create topics using kafka-python?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related