I am trying to run kafka zooker, broker, shema registry and restproxy in kubernetes deployment.
I am sharing below yaml file for schema registry and broker where theese pods are giving error;
I am planning to run kafka broker, producer (N replicas of it whihc will be run in a FAST API endpoint ) and consumer.
# zookeeper-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: zookeeper
spec:
replicas: 1
selector:
matchLabels:
app: zookeeper
template:
metadata:
labels:
app: zookeeper
spec:
containers:
- name: zookeeper
image: confluentinc/cp-zookeeper:latest
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_CLIENT_PORT
value: "2181" # Example port number
# schema-registry-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: schema-registry
spec:
replicas: 1
selector:
matchLabels:
app: schema-registry
template:
metadata:
labels:
app: schema-registry
spec:
containers:
- name: schema-registry
image: confluentinc/cp-schema-registry:7.6.0
ports:
- containerPort: 8081
env:
- name: SCHEMA_REGISTRY_HOST_NAME
value: schema-registry
- name: SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL
value: "zookeeper:2181"
- name: SCHEMA_REGISTRY_LISTENERS
value: "http://0.0.0.0:8081"
# broker1-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kafka
spec:
replicas: 3
selector:
matchLabels:
app: kafka
template:
metadata:
labels:
app: kafka
spec:
hostname: kafka
containers:
- name: kafka
image: confluentinc/cp-kafka:7.6.0
ports:
- containerPort: 39092
env:
- name: KAFKA_BROKER_ID
valueFrom:
fieldRef:
fieldPath: metadata.labels['broker-id']
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: 'PLAINTEXT:PLAINTEXT'
- name: KAFKA_ADVERTISED_LISTENERS
value: "PLAINTEXT://kafka:39092" # Adjust this value based on your deployment configuration
#rest-proxy-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: rest-proxy
spec:
replicas: 1
selector:
matchLabels:
app: rest-proxy
template:
metadata:
labels:
app: rest-proxy
spec:
containers:
- name: rest-proxy
image: confluentinc/cp-kafka-rest:7.6.0
ports:
- containerPort: 8082
env:
- name: KAFKA_REST_BOOTSTRAP_SERVERS
value: "kafka:9092"
# service.yaml
apiVersion: v1
kind: Service
metadata:
name: kafka
spec:
type: NodePort
selector:
app: kafka
ports:
- protocol: TCP
port: 9092
targetPort: 9092
nodePort: 30007
---
apiVersion: v1
kind: Service
metadata:
name: zookeeper
spec:
type: NodePort
selector:
app: zookeeper
ports:
- protocol: TCP
port: 2181
targetPort: 2181
nodePort: 30008
---
apiVersion: v1
kind: Service
metadata:
name: schema-registry
spec:
type: NodePort
selector:
app: schema-registry
ports:
- protocol: TCP
port: 8081
targetPort: 8081
nodePort: 30009
---
apiVersion: v1
kind: Service
metadata:
name: rest-proxy
spec:
type: NodePort
selector:
app: rest-proxy
ports:
- protocol: TCP
port: 8082
targetPort: 8082
nodePort: 30010
nitin@nitin-safepro:~/kubernetes_exp/kafka-api-server$ kubectl logs -f schema-registry-f5f5d54-vvrj6
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
PORT is deprecated. Please use SCHEMA_REGISTRY_LISTENERS instead.
nitin@nitin-safepro:~/kubernetes_exp/kafka-api-server$ kubectl logs -f kafka-7bcbc6d69c-brvc8
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
Running in Zookeeper mode...
port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.
nitin@nitin-safepro:~/kubernetes_exp/kafka-api-server$
I have set servive type as nodePort which also did not help?