I am trying to spin-up the Mimir docker sample available here but I am facing a number of issues starting with the Prometheus container:
ts=2024-05-10T07:00:42.970Z caller=manager.go:1009 level=info component="rule manager" msg="Starting rule manager..."
ts=2024-05-10T07:00:48.979Z caller=compact.go:523 level=info component=tsdb msg="write block" mint=1714629600000 maxt=1714636800000 ulid=01HXGMVW5FMTRTS49PFRB59NQ6 duration=36.562625ms
ts=2024-05-10T07:00:48.982Z caller=head.go:1298 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=1.66275ms
ts=2024-05-10T07:00:48.982Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=33 to_segment=34 mint=1714636800000
ts=2024-05-10T07:00:49.019Z caller=head.go:1266 level=info component=tsdb msg="WAL checkpoint complete" first=33 last=34 duration=37.182875ms
ts=2024-05-10T07:01:37.410Z caller=scrape.go:1344 level=error component="scrape manager" scrape_pool=prometheus target=http://localhost:9090/metrics msg="Scrape commit failed" err="write to WAL: log samples: write data/wal/00000037: no space left on device"
ts=2024-05-10T07:01:38.938Z caller=scrape.go:1344 level=error component="scrape manager" scrape_pool=app-a target=http://app-a:8000/metrics msg="Scrape commit failed" err="write to WAL: log samples: write data/wal/00000037: no space left on device"
ts=2024-05-10T07:01:42.410Z caller=scrape.go:1344 level=error component="scrape manager" scrape_pool=prometheus target=http://localhost:9090/metrics msg="Scrape commit failed" err="write to WAL: log samples: write data/wal/00000037: no space left on device"
ts=2024-05-10T07:01:43.934Z caller=scrape.go:1344 level=error component="scrape manager" scrape_pool=app-a target=http://app-a:8000/metrics msg="Scrape commit failed" err="write to WAL: log samples: write data/wal/00000037: no space left on device"
ts=2024-05-10T09:00:50.310Z caller=db.go:984 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
ts=2024-05-10T09:00:53.786Z caller=db.go:984 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"```
Mimir container logs:
```ts=2024-05-10T08:56:53.197459588Z caller=featurecontrol.go:97 level=warn msg="Classic mode enabled"
ts=2024-05-10T10:51:41.838524708Z caller=basic_lifecycler_delegates.go:147 level=warn msg="auto-forgetting instance from the ring because it is unhealthy for a long time" instance=mimir-3 last_heartbeat="2024-05-10 10:48:11 +0000 UTC" forget_period=20s
ts=2024-05-10T10:51:41.843039625Z caller=basic_lifecycler_delegates.go:147 level=warn msg="auto-forgetting instance from the ring because it is unhealthy for a long time" instance=mimir-1 last_heartbeat="2024-05-10 10:48:11 +0000 UTC" forget_period=20s
ts=2024-05-10T10:51:41.843081292Z caller=basic_lifecycler_delegates.go:147 level=warn msg="auto-forgetting instance from the ring because it is unhealthy for a long time" instance=mimir-2 last_heartbeat="2024-05-10 10:48:11 +0000 UTC" forget_period=20s
ts=2024-05-10T10:51:43.892482626Z caller=basic_lifecycler.go:440 level=warn msg="instance is missing in the ring (e.g. the ring backend storage has been reset), registering the instance with an updated registration timestamp" ring=ruler
ts=2024-05-10T10:51:47.080631377Z caller=gateway.go:311 level=warn msg="failed to synchronize TSDB blocks" reason=ring-change err="unable to check tenants owned by this store-gateway instance: store-gateway is unhealthy in the ring"```
I am running on a Mac system and I have checked the space:
```df -hi
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk3s3s1 460Gi 9.5Gi 295Gi 4% 404k 3.1G 0% /
devfs 209Ki 209Ki 0Bi 100% 722 0 100% /dev
/dev/disk3s6 460Gi 20Ki 295Gi 1% 0 3.1G 0% /System/Volumes/VM
/dev/disk3s4 460Gi 5.7Gi 295Gi 2% 1.1k 3.1G 0% /System/Volumes/Preboot
/dev/disk3s2 460Gi 41Mi 295Gi 1% 50 3.1G 0% /System/Volumes/Update
/dev/disk1s2 500Mi 6.0Mi 481Mi 2% 1 4.9M 0% /System/Volumes/xarts
/dev/disk1s1 500Mi 6.1Mi 481Mi 2% 33 4.9M 0% /System/Volumes/iSCPreboot
/dev/disk1s3 500Mi 2.4Mi 481Mi 1% 49 4.9M 0% /System/Volumes/Hardware
/dev/disk3s1 460Gi 149Gi 295Gi 34% 367k 3.1G 0% /System/Volumes/Data
map auto_home 0Bi 0Bi 0Bi 100% 0 0 - /System/Volumes/Data/home
This is how my docker-compose file looks:
services:
minio:
image: minio/minio
entrypoint: [""]
command: ["sh", "-c", "mkdir -p /data/mimir && minio server --quiet /data"]
environment:
- MINIO_ROOT_USER=mimir
- MINIO_ROOT_PASSWORD=supersecret
volumes:
- minio-data:/data
grafana:
image: grafana/grafana:latest
pull_policy: always
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_USERS_DEFAULT_THEME=light
- GF_LOG_MODE=console
- GF_LOG_LEVEL=critical
volumes:
- ./config/grafana-provisioning-dashboards.yaml:/etc/grafana/provisioning/dashboards/provisioning-dashboards.yaml:ro
- ./config/grafana-provisioning-datasources.yaml:/etc/grafana/provisioning/datasources/provisioning-datasources.yaml:ro
# Explicitly list the dashboards we want to show in the demo. We intentionally exclude dashboards that require
# Kubernetes metrics (eg. resources or networking) and other services not available in the demo (eg. Grafana Loki).
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-alertmanager.json:/var/lib/grafana/dashboards/mimir-alertmanager.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-compactor.json:/var/lib/grafana/dashboards/mimir-compactor.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-object-store.json:/var/lib/grafana/dashboards/mimir-object-store.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-overrides.json:/var/lib/grafana/dashboards/mimir-overrides.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-queries.json:/var/lib/grafana/dashboards/mimir-queries.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-reads.json:/var/lib/grafana/dashboards/mimir-reads.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-ruler.json:/var/lib/grafana/dashboards/mimir-ruler.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-tenants.json:/var/lib/grafana/dashboards/mimir-tenants.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-top-tenants.json:/var/lib/grafana/dashboards/mimir-top-tenants.json:ro
- ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-writes.json:/var/lib/grafana/dashboards/mimir-writes.json:ro
ports:
- 9000:3000
prometheus:
image: prom/prometheus:latest
command:
- --config.file=/etc/prometheus/prometheus.yml
- --log.level=error
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention=60d
- --storage.tsdb.wal-compression
- --web.console.libraries=/usr/share/prometheus/console_libraries
- --web.console.templates=/usr/share/prometheus/consoles
volumes:
- ./config/prometheus.yaml:/etc/prometheus/prometheus.yml
- ../../../../../operations/mimir-mixin-compiled/rules.yaml:/etc/prometheus/rules.yaml
depends_on:
- "mimir-1"
- "mimir-2"
- "mimir-3"
load-balancer:
image: nginx:latest
volumes:
- ./config/nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
- "mimir-1"
- "mimir-2"
- "mimir-3"
ports:
- 9009:9009
mimir-1:
image: grafana/mimir:latest
command: ["-config.file=/etc/mimir.yaml"]
hostname: mimir-1
depends_on:
- minio
volumes:
- ./config/mimir.yaml:/etc/mimir.yaml
- ./config/alertmanager-fallback-config.yaml:/etc/alertmanager-fallback-config.yaml
- mimir-1-data:/data
mimir-2:
image: grafana/mimir:latest
command: ["-config.file=/etc/mimir.yaml"]
hostname: mimir-2
depends_on:
- minio
volumes:
- ./config/mimir.yaml:/etc/mimir.yaml
- ./config/alertmanager-fallback-config.yaml:/etc/alertmanager-fallback-config.yaml
- mimir-2-data:/data
mimir-3:
image: grafana/mimir:latest
command: ["-config.file=/etc/mimir.yaml"]
hostname: mimir-3
depends_on:
- minio
volumes:
- ./config/mimir.yaml:/etc/mimir.yaml
- ./config/alertmanager-fallback-config.yaml:/etc/alertmanager-fallback-config.yaml
- mimir-3-data:/data
volumes:
mimir-1-data:
mimir-2-data:
mimir-3-data:
minio-data:
I have maintained all the default settings and followed the setup guide available here.
I tried adding the following setting to the Prometheus server but I doesnt solve anything:
- --storage.tsdb.retention=60d
- --storage.tsdb.wal-compression
What am I missing ?