I am doing a Kubernetes Gateway API travel in my own ,airgapped,non cloud based kubernetes platform
Following the Get Started Section https://istio.io/latest/zh/docs/setup/additional-setup/getting-started/,
All the deployment and gateway ,and istioctl install are same as the url before.
but, the sidecar pod reported a StartupProbe failed ,so the gateway API is not able to work.
Entering the pod log, it just tried to dial 169.254.169.254,
( HTTP request failed: Get “http://169.254.169.254/metadata/instance?api-version=2019-08-15”: context deadline exceeded (Client.Timeout exceeded while awaiting headers))
which seems to be a cloud platform based IP addr for metadata storing,the sidecar pod proxyv2 did the http request ,
it seems to be a istioctl issue, because a single proxyv2 pod runs successfully.
following the
set environment CLOUD_PLATFORM=’none’ did not work in a air gap and non cloud k8s platform
once install istioctl in demo profile, the exactly same error occurs in istio-ingress/egress gateway pod.
Version:
k8s: 1.30.2
istio 1.22.3
k8s gateway api version: v1
istioctl profile: minimal
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: bookinfo-gateway
spec:
gatewayClassName: istio
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Same
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: bookinfo
spec:
parentRefs:
- name: bookinfo-gateway
rules:
- matches:
- path:
type: Exact
value: /productpage
- path:
type: PathPrefix
value: /static
- path:
type: Exact
value: /login
- path:
type: Exact
value: /logout
- path:
type: PathPrefix
value: /api/v1/products
backendRefs:
- name: productpage
port: 9080
and the kubectl describe pod show a startupProbe error:
kubectl describe pod bookinfo-gateway-istio-6ccbf9b4f-qt8wm
Name: bookinfo-gateway-istio-6ccbf9b4f-qt8wm
Namespace: default
Priority: 0
Service Account: bookinfo-gateway-istio
Node: master/192.168.1.100
Start Time: Sat, 27 Jul 2024 23:25:52 +0800
Labels: gateway.networking.k8s.io/gateway-name=bookinfo-gateway
istio.io/gateway-name=bookinfo-gateway
pod-template-hash=6ccbf9b4f
service.istio.io/canonical-name=bookinfo-gateway-istio
service.istio.io/canonical-revision=latest
sidecar.istio.io/inject=false
Annotations: cni.projectcalico.org/containerID: ee2de49455458dca810dfda21a11819a3299556b4aeff4be0cab0a9bf70f71d9
cni.projectcalico.org/podIP: 172.27.205.163/32
cni.projectcalico.org/podIPs: 172.27.205.163/32
istio.io/rev: default
prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true
Status: Running
IP: 172.27.205.163
IPs:
IP: 172.27.205.163
Controlled By: ReplicaSet/bookinfo-gateway-istio-6ccbf9b4f
Containers:
istio-proxy:
Container ID: containerd://faafbba7c3c90e80643ebf60b519bed077d3f7eac89c86842ea6846f9eb87f83
Image: docker.io/istio/proxyv2:1.22.3
Image ID: docker.io/istio/proxyv2@sha256:f43ddace4b6ab7265b1f6f8fe74a11fa86300add81be338333993e4e8205d07e
Ports: 15021/TCP, 15090/TCP
Host Ports: 0/TCP, 0/TCP
Args:
proxy
router
--domain
$(POD_NAMESPACE).svc.cluster.local
--proxyLogLevel
warning
--proxyComponentLogLevel
misc:error
--log_output_level
default:info
State: Running
Started: Sat, 27 Jul 2024 23:25:56 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Readiness: http-get http://:15021/healthz/ready delay=0s timeout=1s period=15s #success=1 #failure=4
Startup: http-get http://:15021/healthz/ready delay=1s timeout=1s period=1s #success=1 #failure=30
Environment:
PILOT_CERT_PROVIDER: istiod
CA_ADDR: istiod.istio-system.svc:15012
POD_NAME: bookinfo-gateway-istio-6ccbf9b4f-qt8wm (v1:metadata.name)
POD_NAMESPACE: default (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
HOST_IP: (v1:status.hostIP)
ISTIO_CPU_LIMIT: 2 (limits.cpu)
PROXY_CONFIG: {}
ISTIO_META_POD_PORTS: []
ISTIO_META_APP_CONTAINERS:
GOMEMLIMIT: 1073741824 (limits.memory)
GOMAXPROCS: 2 (limits.cpu)
ISTIO_META_CLUSTER_ID: Kubernetes
ISTIO_META_NODE_NAME: (v1:spec.nodeName)
ISTIO_META_INTERCEPTION_MODE: REDIRECT
ISTIO_META_WORKLOAD_NAME: bookinfo-gateway-istio
ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/default/deployments/bookinfo-gateway-istio
ISTIO_META_MESH_ID: cluster.local
TRUST_DOMAIN: cluster.local
Mounts:
/etc/istio/pod from istio-podinfo (rw)
/etc/istio/proxy from istio-envoy (rw)
/var/lib/istio/data from istio-data (rw)
/var/run/secrets/credential-uds from credential-socket (rw)
/var/run/secrets/istio from istiod-ca-cert (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-njmgq (ro)
/var/run/secrets/tokens from istio-token (rw)
/var/run/secrets/workload-spiffe-credentials from workload-certs (rw)
/var/run/secrets/workload-spiffe-uds from workload-socket (rw)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
workload-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
credential-socket:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
workload-certs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
istio-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-podinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
istio-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 43200
istiod-ca-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-ca-root-cert
Optional: false
kube-api-access-njmgq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 33m default-scheduler Successfully assigned default/bookinfo-gateway-istio-6ccbf9b4f-qt8wm to master
Normal Pulled 33m kubelet Container image "docker.io/istio/proxyv2:1.22.3" already present on machine
Normal Created 33m kubelet Created container istio-proxy
Normal Started 33m kubelet Started container istio-proxy
Warning Unhealthy 33m (x4 over 33m) kubelet Startup probe failed: Get "http://172.27.205.163:15021/healthz/ready": dial tcp 172.27.205.163:15021: connect: connection refused
2024-07-27T15:26:01.115128Z info Workload SDS socket not found. Starting Istio SDS Server
2024-07-27T15:26:01.115136Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2024-07-27T15:26:01.115138Z info Opening status port 15020
2024-07-27T15:26:01.115182Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2024-07-27T15:26:01.121325Z info ads All caches have been synced up in 5.038009041s, marking server ready
2024-07-27T15:26:01.121473Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2024-07-27T15:26:01.122145Z info Pilot SAN: [istiod.istio-system.svc]
2024-07-27T15:26:01.122514Z info Starting proxy agent
2024-07-27T15:26:01.122559Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields -l warning --component-log-level misc:error --concurrency 2]
2024-07-27T15:26:01.135657Z info sds Starting SDS grpc server
2024-07-27T15:26:01.136019Z info sds Starting SDS server for workload certificates, will listen on "./var/run/secrets/workload-spiffe-uds/socket"
2024-07-27T15:26:01.135907Z info starting Http service at 127.0.0.1:15004
2024-07-27T15:26:01.154136Z warning envoy main external/envoy/source/server/server.cc:835 Usage of the deprecated runtime key overload.global_downstream_max_connections, consider switching to `envoy.resource_monitors.downstream_connections` instead.This runtime key will be removed in future. thread=15
2024-07-27T15:26:01.154362Z warning envoy main external/envoy/source/server/server.cc:928 There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor. thread=15
2024-07-27T15:26:01.159721Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=1
2024-07-27T15:26:01.170371Z info ads ADS: new connection for node:bookinfo-gateway-istio-6ccbf9b4f-qt8wm.default-1
2024-07-27T15:26:01.170735Z info ads ADS: new connection for node:bookinfo-gateway-istio-6ccbf9b4f-qt8wm.default-2
2024-07-27T15:26:01.182882Z info cache generated new workload certificate latency=61.367763ms ttl=23h59m59.817119728s
2024-07-27T15:26:01.182901Z info cache Root cert has changed, start rotating root cert
2024-07-27T15:26:01.182910Z info ads XDS: Incremental Pushing ConnectedEndpoints:2 Version:
2024-07-27T15:26:01.182930Z info cache returned workload trust anchor from cache ttl=23h59m59.817069728s
2024-07-27T15:26:01.182965Z info cache returned workload certificate from cache ttl=23h59m59.817035327s
2024-07-27T15:26:01.183062Z info cache returned workload trust anchor from cache ttl=23h59m59.816938526s
2024-07-27T15:26:01.183107Z info ads SDS: PUSH request for node:bookinfo-gateway-istio-6ccbf9b4f-qt8wm.default resources:1 size:4.0kB resource:default
2024-07-27T15:26:01.183163Z info ads SDS: PUSH request for node:bookinfo-gateway-istio-6ccbf9b4f-qt8wm.default resources:1 size:1.1kB resource:ROOTCA
2024-07-27T15:26:01.183200Z info cache returned workload trust anchor from cache ttl=23h59m59.816799725s
2024-07-27T15:26:01.305696Z info Readiness succeeded in 5.224696454s
2024-07-27T15:26:01.305827Z info Envoy proxy is ready
2024-07-27T15:26:06.114973Z warn HTTP request failed: Get "http://169.254.169.254/metadata/instance?api-version=2019-08-15": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-07-27T15:26:06.114986Z warn Could not unmarshal response: unexpected end of JSON input:
2024-07-27T15:56:14.159162Z info xdsproxy connected to delta upstream XDS server: istiod.istio-system.svc:15012 id=2
Version
istioctl version
client version: 1.22.3
control plane version: 1.22.3
data plane version: 1.22.3 (8 proxies)
kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.2