Every morning I see the same problem. Graphs of CPU, Disk and Memory of Kubernetes nodes start from 6 AM/8:30 AM but I can’t see previous data.
Graphs of CPU, Disk and Memory of Kubernetes nodes start from 6 AM/8:30 AM but I can’t see previous data.
I create the environment with Terraform and Helm.
This is my configuration for prometheus-values.yaml
:
prometheus:
serviceMonitorSelector: {}
podMonitorSelector: {}
additionalScrapeConfigs:
- job_name: 'kubernetes-nodes'
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name]
action: replace
target_label: __metrics_path__
replacement: /api/v1/namespaces/${1}/pods/${2}/proxy/metrics
And these are my queries:
CPU:
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle", instance=~"$node"}[5m])) * 100)
Disk usage
sum by (instance) (node_filesystem_size_bytes{fstype=~"ext4|xfs", instance=~"$node"} - node_filesystem_free_bytes{fstype=~"ext4|xfs", instance=~"$node"})
Available memory
sum by (instance) (node_memory_MemAvailable_bytes{instance=~"$node"})
Memory usage
sum by (instance) (node_memory_MemTotal_bytes{instance=~"$node"} - node_memory_MemAvailable_bytes{instance=~"$node"})
To solve I tried adding to prometheus-values.yaml
:
prometheusSpec:
retention: 7d
storageSpec:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "8Gi"
cpu: "4"
but there is still the same problem.