Confluent Kafka makes it easy to connect your applications and systems to data streams and process them in real time. Confluent Platform provides a simpler solution to deploying Kafka clusters on Docker and Kubernetes than Apache Kafka
Prerequisites
KubectlandHelm 3already installed- For this tutorial, your Kubernetes cluster is assumed to already have a default dynamic storage provisioner . Use the
kubectl get sccommand to check if the Kubernetes cluster has a default dynamic storage provisioner.
Step 1: Create a namespace
1 2 | kubectl create namespace confluent |
Set the confluent namespace to the default namespace
1 2 | kubectl config set-context --current --namespace confluent |
Step 2: Install Confluent Platform for Kubernetes
Add Confluent to Helm repository.
1 2 3 | helm repo add confluentinc https://packages.confluent.io/helm helm repo update |
Install Confluent Platform using helm
1 2 | helm upgrade --install confluent-operator confluentinc/confluent-for-kubernetes |
Make sure the Operator is running and ready before going to the next step
1 2 | kubectl get pods |
Step 3: Install Confluent Platform components
Confluent Platform components include:
- Zookeepers
- Kafka Brokers
- Kafka Connectors
- KsqlDB
- Control Center (Kafka Web UI)
- Schema Registry
- Kafka Rest Proxy
1 2 | kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/confluent-platform.yaml |
Install a producer app and create a topic for testing.
1 2 | kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/producer-app-data.yaml |
Check if everything is Running and Ready:
1 2 | kubectl get pods |
Step 4: Create External Load balancer for Control Center
Create a file control-center.yaml with the following content:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | apiVersion: v1 kind: Service metadata: name: kafka-ui-lb spec: type: LoadBalancer sessionAffinity: None selector: app: controlcenter ports: - name: kafka-ui-lb port: 9021 protocol: TCP targetPort: 9021 |
Use kubectl to apply the configuration file and check if the Load balancer has been created successfully
1 2 3 4 | kubectl apply -f control-center.yaml kubectl get svc |
Open the Web Browser and go to the address <External-IP>:9021
Brief introduction about Control Center:
Here is the main interface:
Brokers: Monitor the status of Brokers nodes
Topics: Review created Topics, see msg inside each Topic, Produce msg
Connect: Manage sources, sink connectors
KsqlDB: Handling Kafka Stream related tasks