Apache Kafka – Producer – Send messages to Kafka using kafka-python

Tram Ho

Overview

Understand how to produce message and send to the Kafka topic

Architecture

Producer has many types and sources: message from Credit Card transactions, message from Facebook, Email or any systems

When the producer sends the message to kafka, kafka after receiving the message and randomly distributes that message to each partition. So the Producer only needs to care about:

  • Bootstrap Server
  • Topic
  • Value_serializer : the way, the format that the message is sent to
  • client_id :is the id that the client is given and the producer has this id to send a message to kafka topic
  • acks : there are 3 types (0, 1, ‘all’), when sending a message to kafka, the procedure asks kafka to confirm itself to proceed with the process to continue other messages. defaults to charges=1
  1. acks = 0: Producer will not wait for Kafka to confirm that the data has been received. Every time there is a message, the Producer will automatically add the message to the memory. Therefore, in some cases data will be lost and Kafka does not guarantee this.
  2. acks = 1: Producer will just wait for the message to be written to the leader and not care if the message is replicated to other followers. Soon, the Producer will continue to send another message. With this mechanism, in some cases the message will be lost in the consumer, due to the error of the leader
  3. acks = all. The producer will wait for the entire leader and follower process to be written down before processing another message. So Kafka will ensure that the message will not lose data. But there will be cases where the data is blocked at the Producer

Code example

Requirement

  • Python 3.6 , 3.7, 3.8
  • pip install kafka-python – required
  • pip install Faker – optional : this Lib to randomly create dummy data

Code example

  1. Please visit the previous document to know how to set up kafka, kafka CLI, Kafka UI.
  2. Produce the dummy data

  1. Send data to Kafka

As above we know that 3 points must have to send the message to kafka is:

  • boostrap server or broker: the ip/host of broker
  • topic name
  • value_serializer : message and message type

Code for sending message:

Focus on only one partition

Set up sending messages to only 1 given partition in list partition of kafka

Share the news now