Im trying to connect to the kubernetes minikube kafka pod from outside the kubernetes cluster.
The server starts without any problem but I cant manage to connect a local kafka producer/consumer to the external kafka pod.
On the kafka server image inside the cluster I set the bootstrap-server as:
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic "test"
The service for the kafka server pod is set as:
apiVersion: v1
kind: Service
metadata:
name: {{ .Chart.Name }}-service
spec:
type: NodePort
ports:
- name: {{ .Chart.Name }}
port: 9092
nodePort: 30092
selector:
app: {{ .Chart.Name }}
The only kafka config that I set in the deployment are below for the listeners.
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Chart.Name }}-deployment
labels:
app: {{ .Chart.Name }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.deployment.replicaCount }}
{{- end }}
selector:
matchLabels:
app: {{ .Chart.Name }}
template:
metadata:
labels:
app: {{ .Chart.Name }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.deployment.image }}"
imagePullPolicy: {{ .Values.deployment.pullPolicy }}
command: {{ .Values.deployment.initCommand | toYaml | nindent 12 }}
ports:
- containerPort: 9092
name: internal
- containerPort: 9093
name: controller
- containerPort: 30092
name: nodeport
env:
- name: KRAFT_TOPIC_ML
value: {{ .Values.deployment.topic.nameML }}
- name: KRAFT_TOPIC_ML_REPLICATION_FACTOR
value: "{{ .Values.deployment.topic.replicationFactorML }}"
- name: KRAFT_TOPIC_ML_PARTITIONS
value: "{{ .Values.deployment.topic.partitionsML }}"
- name: KRAFT_TOPIC_MILP
value: {{ .Values.deployment.topic.nameMILP }}
- name: KRAFT_TOPIC_MILP_REPLICATION_FACTOR
value: "{{ .Values.deployment.topic.replicationFactorMILP }}"
- name: KRAFT_TOPIC_MILP_PARTITIONS
value: "{{ .Values.deployment.topic.partitionsMILP }}"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL:PLAINTEXT,NODEPORT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL"
- name: KAFKA_LISTENERS
value: "INTERNAL://0.0.0.0:9092,NODEPORT://0.0.0.0:30092,CONTROLLER://0.0.0.0:9093"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL://localhost:9092,NODEPORT://localhost:30092"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL"
- name: KAFKA_CONTROLLER_LISTENER
value: "CONTROLLER"
When trying to connect to the kafka server locally on my machine with the local IP and PORT as 30092 I can find the broker and connect but when producing messages it doesnt show up on the consumer python side on the same IP and PORT.
I tried changing the kafka listeners config without any success.