We are trying to run node-exporter as a side-car container in kubernetes deployment.
We have multiple filesystem devices, but in metrics we are getting only for single device filesystem metrics
- name: node-exporter
image: quay.io/prometheus/node-exporter:latest
args:
- --collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc|tmpfs|shm)($|/)"
volumeMounts:
- mountPath: /etc/app
name: app-lib
- name: main-application
image:
volumeMounts:
- mountPath: /etc/app
name: app-lib
/ # df -h
Filesystem Size Used Available Use% Mounted on
overlay 50.0G 14.7G 35.3G 29% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 31.1G 0 31.1G 0% /sys/fs/cgroup
/dev/nvme1n1 195.8G 675.8M 195.1G 0% /etc/app
/dev/nvme0n1p1 50.0G 14.7G 35.3G 29% /etc/hosts
/dev/nvme0n1p1 50.0G 14.7G 35.3G 29% /dev/termination-log
/dev/nvme0n1p1 50.0G 14.7G 35.3G 29% /etc/hostname
/dev/nvme0n1p1 50.0G 14.7G 35.3G 29% /etc/resolv.conf
shm 64.0M 96.0K 63.9M 0% /dev/shm
/dev/nvme2n1 48.9G 9.8G 39.1G 20% /etc/local/app1
tmpfs 61.2G 12.0K 61.2G 0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs 31.1G 0 31.1G 0% /proc/acpi
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/latency_stats
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 31.1G 0 31.1G 0% /sys/firmware
We are unable to find the root cause for this issue.
We are running node exporter as root user and we are getting only this device metrics
node_filesystem_device_error{device="/dev/nvme0n1p1", fstype="xfs", instance="10.2.1.66:9100", job="xxxxx-node", kubernetes_namespace="xxxxxxxx", mountpoint="/etc/resolv.conf"}
Node exporter logs:
ts=2024-07-08T09:29:19.757Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.1, branch=HEAD, revision=400c3979931613db930ea035f39ce7b377cdbb5b)"
ts=2024-07-08T09:29:19.757Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.3, platform=linux/amd64, user=root@7afbff271a3f, date=20240521-18:36:22, tags=unknown)"
ts=2024-07-08T09:29:19.757Z caller=node_exporter.go:196 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
ts=2024-07-08T09:29:19.758Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2024-07-08T09:29:19.759Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-07-08T09:29:19.759Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvmed+nd+p)d+$
ts=2024-07-08T09:29:19.759Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=arp
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=bcache
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=bonding
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=btrfs
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=conntrack
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=cpu
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=cpufreq
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=diskstats
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=dmi
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=edac
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=entropy
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=fibrechannel
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=filefd
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=filesystem
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=hwmon
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=infiniband
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=ipvs
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=loadavg
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=mdadm
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=meminfo
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=netclass
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=netdev
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=netstat
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=nfs
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=nfsd
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=nvme
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=os
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=powersupplyclass
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=pressure
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=rapl
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=schedstat
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=selinux
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=sockstat
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=softnet
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=stat
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=tapestats
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=textfile
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=thermal_zone
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=time
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=timex
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=udp_queues
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=uname
ts=2024-07-08T09:29:19.759Z caller=node_exporter.go:118 level=info collector=vmstat
ts=2024-07-08T09:29:19.760Z caller=node_exporter.go:118 level=info collector=watchdog
ts=2024-07-08T09:29:19.760Z caller=node_exporter.go:118 level=info collector=xfs
ts=2024-07-08T09:29:19.760Z caller=node_exporter.go:118 level=info collector=zfs
ts=2024-07-08T09:29:19.760Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
ts=2024-07-08T09:29:19.760Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100