I have a linux box in GCP with docker installed and Prometheus running as a container.
I also have 2 GKE clusters (A & B) with istio installed (just as an ingress controller only) – service mesh is not enabled.
All 3 are in seperate projects. The linux box project is peered to project with cluster ‘A’ and the project with cluster ‘A’ is peered to project containing cluster ‘B’.
Basically the GKE ‘A’ cluster sits in between the Linux box and the GKE ‘B’ cluster like a proxy.
I am trying to scrape the metrics from the Prometheus server in GKE ‘B’.
Inside GKE ‘A’ I have the following gateway, serviceEntry and Virtual Service
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: istio-ilb
namespace: istio-ingress
spec:
selector:
istio: ingress
servers:
---rest ommited
- hosts:
- '*'
port:
name: http-web
number: 9090
protocol: HTTP
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
spec:
endpoints:
- address: 192.XX.XX.XX # istio loadbalancer IP of GKE B
hosts:
- prometheus.B.infra.internal
location: MESH_EXTERNAL
ports:
- name: http-web
number: 9090
protocol: HTTP
resolution: DNS
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
external-dns.alpha.kubernetes.io/target: ilb.A.infra.internal
labels:
app.kubernetes.io/instance: kube-prometheus
name: kube-prometheus
namespace: monitoring
spec:
gateways:
- istio-ingress/istio-ilb
hosts:
- prometheus.A.infra.internal
http:
- match:
- uri:
exact: /metrics
route:
- destination:
host: prometheus.B.infra.internal
port:
number: 9090
- match:
- uri:
prefix: /
route:
- destination:
host: prometheus.B.infra.internal
port:
number: 9090
In GKE ‘B; I have the following config:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: istio-ilb
namespace: istio-ingress
spec:
selector:
istio: ingress
servers:
---rest ommited
- hosts:
- '*'
port:
name: http-web
number: 9090
protocol: HTTP
piVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
external-dns.alpha.kubernetes.io/target: ilb.B.infra.internal
labels:
app.kubernetes.io/instance: kube-prometheus
name: kube-prometheus
namespace: monitoring
spec:
gateways:
- istio-ingress/istio-ilb
hosts:
- prometheus.B.infra.internal
http:
- match:
- uri:
exact: /metrics
route:
- destination:
host: kube-prometheus-kube-prome-prometheus.monitoring.svc.cluster.local
port:
number: 9090
- match:
- uri:
prefix: /
route:
- destination:
host: kube-prometheus-kube-prome-prometheus.monitoring.svc.cluster.local
port:
number: 9090
The endpoint in cluster B
kube-prometheus-kube-prome-prometheus 10.31.0.89:9090,10.31.0.89:8080
I get the following errors in the istio loadbalancers of each cluster
GKE ‘A’
[2024-07-25T14:32:53.435Z] "GET /metrics HTTP/1.1" 404 - via_upstream - "-" 0 0 2 2 "192.168.31.94" "Prometheus/2.52.0" "a4cfdd83-9e23-47e7-8f │
│ 35-7e0f5d6e1943" "prometheus.A.infra.internal:9090" "192.<cluster B IP>:9090" outbound|9090||prometheus.B.infra.internal 10.33.2.167:42320 10.33.2.167:9090 192.<cluster A IP>:42 │
│ 890 - -
GKE ‘B’
[2024-07-25T14:38:27.043Z] "GET /metrics HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "192.<cluster A IP>,192.<cluster B IP>" "Prometheus/2.52.0" "367 │
│ 5e168-e5c8-435c-8e18-43b07162df76" "prometheus.A.infra.internal:9090" "-" - - 10.31.0.31:9090 192.168.30.83:21654 - -
A curl and nslookup from linux box to GKE ‘A’
curl -I http://prometheus.A.internal:9090/metrics
HTTP/1.1 404 Not Found
date: Thu, 25 Jul 2024 14:41:45 GMT
server: istio-envoy
x-envoy-upstream-service-time: 2
transfer-encoding: chunked
nslookup prometheus.A.infra.internal
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
prometheus.A.infra.internal canonical name = ilb.A.infra.internal.
Name: ilb.A.infra.internal
Address: 192.<cluster A IP>
From a pod in GKE ‘A’
kubectl run curlpod-proxy --image=radial/busyboxplus:curl -i --tty --rm
root@curlpod-proxy:/ ]$ nslookup prometheus.B.infra.internal
Server: 10.32.240.10
Address 1: 10.32.240.10 kube-dns.kube-system.svc.cluster.local
Name: prometheus.B.infra.internal
Address 1: 192.<cluster B IP>
curl -s http://prometheus.B.infra.internal:9090/metrics | head -n 5
# HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime.
# TYPE go_gc_cycles_automatic_gc_cycles_total counter
go_gc_cycles_automatic_gc_cycles_total 16205
# HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application.
# TYPE go_gc_cycles_forced_gc_cycles_total counter
istioctl pc route comd
NAME VHOST NAME DOMAINS MATCH VIRTUAL SERVICE
http.80 prometheus.A.infra.internal:80 prometheus.A.infra.internal /metrics kube-prometheus.monitoring
http.80 prometheus.A.infra.internal:80 prometheus.A.infra.internal /* kube-prometheus.monitoring
http.9090 prometheus.B.infra.internal:9090 prometheus.B.infra.internal /metrics kube-prometheus.monitoring
http.9090 prometheus.B.infra.internal:9090 prometheus.B.infra.internal /* kube-prometheus.monitoring
What am I doing wrong? – is it possible what I’m trying to do?