I’m working on setting up Prometheus and OpenTelemetry Collector in my Kubernetes environment using Helm. I’m using the prometheus-community/prometheus Helm chart for deploying Prometheus, and I need to configure it to scrape metrics from my OpenTelemetry Collector.
Here are the steps I’ve taken so far:
helm upgrade --install prometheus prometheus-community/prometheus -f values.yaml
extraScrapeConfigs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['otel-collector.default.svc.cluster.local:4317']
After deploying Prometheus and configuring the values.yaml file, I don’t see the OpenTelemetry Collector in the list of Prometheus targets when I check the Prometheus UI (/targets). It’s as if Prometheus isn’t discovering or scraping the OpenTelemetry Collector at all.
Also want to disable other jobs example kubernetes-apiservers
Here are a few things to try:
Restart the prometheus pod.
Try to curl the metrics endpoint with
kubectl exec -it [promethues pod name] sh -- "curl http://otel-collector.default.svc.cluster.local:4317"
Solution:
I found a solution for the OpenTelemetry Collector not showing up:
You need to ensure that the extraScrapeConfigs are correctly formatted. Here’s how you should define it in your values.yaml:
extraScrapeConfigs: |
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['otel-collector-opentelemetry-collector.pipeline.svc.cluster.local:8889']
This resolved the issue for me, and now Prometheus is successfully scraping the OpenTelemetry Collector metrics.
However, I’m still trying to figure out how to disable other default scrape jobs like kubernetes-apiservers. If anyone has suggestions or knows the solution, I’d appreciate the help!