This is a series of Kafka 101 tricks that our team see during our engineer work. Blog posts will be very small and objective.

This first post is about sending messages to topics that don’t exists and what happens to that messages.

 

Here’s an example of one topic that we created:

root@kafka1:~# kafka-topics –zookeeper 172.19.0.2:2181 –list

redglue

A description on the topic will show that it has 3 partitions with replication factor of 1. Simple and easy.

root@kafka1:~# kafka-topics –zookeeper 172.19.0.2:2181 –topic redglue –describe

Topic: redglue PartitionCount:3 ReplicationFactor:1 Configs:

Topic: redglue Partition: 0 Leader: 1 Replicas: 1 Isr: 1

Topic: redglue Partition: 1 Leader: 1 Replicas: 1 Isr: 1

Topic: redglue Partition: 2 Leader: 1 Replicas: 1 Isr: 1

Now let’s produce messages to a non-existent topic called redglue_nonexistent:

root@kafka1:~# kafka-console-producer –broker-list 127.0.0.1:9092 –topic redglue_nonexistent
I maybe don’t exists
[2018-11-28 14:22:12,454] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {redglue_nonexistent=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Obviously there was a WARNING saying that the topic doesn’t exists, but it allows you to “send” messages to that specific topic, so instead of you having all messages lost to this topic, it will be created automatically…with the defaults (and that can be a problem)

root@kafka1:~# kafka-topics –zookeeper 172.19.0.2:2181 –topic redglue_nonexistent –describe
Topic:redglue_nonexistent PartitionCount:1 ReplicationFactor:1 Configs:
Topic: redglue_nonexistent Partition: 0 Leader: 1 Replicas: 1 Isr: 1

That shows that the topic was created with 1 partition and a replication factor of 1 (defaults) and your message is not lost.

 

What you have to take in mind here is:

1) Make sure that you create the topics BEFORE sending messages to them
2) If you don’t follow 1), make sure that your server properties have the right defaults when creating a new topic (eg: num.partitions=1)
3) Control automatic topic creation with parameter auto.create.topics.enable

 

That all for today Kafka 101.