I am collecting OpenTelemetry (OTEL) metrics from my services, forwarding them to an OTEL Collector, and then sending them to Prometheus through the OTEL endpoint. Here is a sample OTEL Collector configuration I am using:
# Receivers
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
auth:
authenticator: bearertokenauth
# Processors
processors:
batch: {}
# Exporters
exporters:
otlphttp/logs:
endpoint: "http://loki-write:3100/otlp"
tls:
insecure: true
otlphttp/metrics:
endpoint: "http://prometheus-server/api/v1/otlp"
tls:
insecure: true
otlphttp/traces:
endpoint: "http://tempo-distributor:4318"
tls:
insecure: true
# Pipelines
service:
extensions:
- health_check
- bearertokenauth
pipelines:
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/logs]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/metrics]
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp/traces]
Problem:
I can view most metrics in Prometheus (via Grafana), but histogram metrics are missing. For example, metrics like http_server_request_duration_sum
do not appear in Prometheus.
Troubleshooting Steps:
- To rule out issues with the OTEL pipeline, I tested sending the same metrics to a VictoriaMetrics backend. In VictoriaMetrics, I can see all metrics, including histograms (e.g.,
http_server_request_duration_sum
). The only difference is that.
in metric names is not replaced with_
. - I enabled native histograms in Prometheus by modifying the deployment configuration, but this did not resolve the issue.
- I am using the latest version of Prometheus deployed via a Helm chart in an EKS cluster.
Additional Context:
- OTEL metrics, except histograms, are visible in Prometheus.
- The environment is Kubernetes-based, and Prometheus is integrated using a Helm chart.
- I have reviewed multiple articles and GitHub issues but have not found a solution to this problem.
Question:
What could be the reason for histogram metrics not appearing in Prometheus? Are there any specific configurations in the OTEL Collector, Prometheus, or Grafana that I need to check?