Every morning I see the same problem. Graphs of CPU, Disk and Memory of Kubernetes nodes start from 6 AM/8:30 AM but I can’t see previous data.

Graphs of CPU, Disk and Memory of Kubernetes nodes start from 6 AM/8:30 AM but I can’t see previous data.

I create the environment with Terraform and Helm.
This is my configuration for prometheus-values.yaml:

prometheus:
    serviceMonitorSelector: {}
    podMonitorSelector: {}
    additionalScrapeConfigs:
      - job_name: 'kubernetes-nodes'
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics
      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name]
            action: replace
            target_label: __metrics_path__
            replacement: /api/v1/namespaces/${1}/pods/${2}/proxy/metrics

And these are my queries:

CPU:
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle", instance=~"$node"}[5m])) * 100)

Disk usage
sum by (instance) (node_filesystem_size_bytes{fstype=~"ext4|xfs", instance=~"$node"} - node_filesystem_free_bytes{fstype=~"ext4|xfs", instance=~"$node"})

Available memory
sum by (instance) (node_memory_MemAvailable_bytes{instance=~"$node"})

Memory usage
sum by (instance) (node_memory_MemTotal_bytes{instance=~"$node"} - node_memory_MemAvailable_bytes{instance=~"$node"})

To solve I tried adding to prometheus-values.yaml:

prometheusSpec:
    retention: 7d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
    resources:
      requests:
        memory: "4Gi"
        cpu: "2"
      limits:
        memory: "8Gi"
        cpu: "4"

but there is still the same problem.