Deploying Confluent Kafka (Enterprise) for Kubernetes

Tram Ho

2 years ago

Confluent Kafka makes it easy to connect your applications and systems to data streams and process them in real time. Confluent Platform provides a simpler solution to deploying Kafka clusters on Docker and Kubernetes than Apache Kafka

Prerequisites

Kubectl and Helm 3 already installed
For this tutorial, your Kubernetes cluster is assumed to already have a default dynamic storage provisioner . Use the kubectl get sc command to check if the Kubernetes cluster has a default dynamic storage provisioner.

Step 1: Create a namespace

kubectl create namespace confluent

1 2	kubectl create namespace confluent

Set the confluent namespace to the default namespace

kubectl config set-context --current --namespace confluent

1 2	kubectl config set-context --current --namespace confluent

Step 2: Install Confluent Platform for Kubernetes

Add Confluent to Helm repository.

helm repo add confluentinc https://packages.confluent.io/helm
helm repo update

helm repo add confluentinc https://packages.confluent.io/helm

helm repo update

Install Confluent Platform using helm

helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes

1 2	helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes

Make sure the Operator is running and ready before going to the next step

kubectl get pods

1 2	kubectl get pods

Step 3: Install Confluent Platform components

Confluent Platform components include:

Zookeepers
Kafka Brokers
Kafka Connectors
KsqlDB
Control Center (Kafka Web UI)
Schema Registry
Kafka Rest Proxy

kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/confluent-platform.yaml

1 2	kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/confluent-platform.yaml

Install a producer app and create a topic for testing.

kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/producer-app-data.yaml

1 2	kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/producer-app-data.yaml

Check if everything is Running and Ready:

kubectl get pods

1 2	kubectl get pods

Step 4: Create External Load balancer for Control Center

Create a file control-center.yaml with the following content:

apiVersion: v1
kind: Service
metadata:
  name: kafka-ui-lb
spec:
  type: LoadBalancer
  sessionAffinity: None
  selector:
    app: controlcenter
  ports:
    - name: kafka-ui-lb
      port: 9021
      protocol: TCP
      targetPort: 9021

apiVersion: v1

kind: Service

metadata:

name: kafka-ui-lb

spec:

type: LoadBalancer

sessionAffinity: None

selector:

app: controlcenter

ports:

- name: kafka-ui-lb

port: 9021

protocol: TCP

targetPort: 9021

Use kubectl to apply the configuration file and check if the Load balancer has been created successfully

kubectl apply -f control-center.yaml

kubectl get svc

kubectl apply -f control-center.yaml

kubectl get svc

Open the Web Browser and go to the address <External-IP>:9021

Brief introduction about Control Center:

Here is the main interface:

Brokers: Monitor the status of Brokers nodes

Topics: Review created Topics, see msg inside each Topic, Produce msg

Connect: Manage sources, sink connectors

KsqlDB: Handling Kafka Stream related tasks

Share the news now