I added open-telemetry helm chart
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
Install opentelemery operator
helm install opentelemetry-operator open-telemetry/opentelemetry-operator
--set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s"
Install opentelemery collector
helm install opentelemetry-collector open-telemetry/opentelemetry-collector --set mode=daemonset --set image.repository="otel/opentelemetry-collector-k8s" --set command.name="otelcol-k8s" -f values.yaml
values.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlp:
endpoint: jaeger-collector.default.svc.cluster.local:14250
tls:
insecure: true
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp]
metrics:
receivers: [otlp]
exporters: [debug]
logs:
receivers: [otlp]
exporters: [debug]
Install Jaeger
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install jaeger jaegertracing/jaeger
Deploy an application
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: otelcol-demo-app
name: otelcol-demo-app
spec:
replicas: 1
selector:
matchLabels:
app: otelcol-demo-app
template:
metadata:
labels:
app: otelcol-demo-app
spec:
containers:
- image: quay.io/rbaumgar/otelcol-demo-app-jvm
imagePullPolicy: IfNotPresent
name: otelcol-demo-app
env:
- name: OTELCOL_SERVER
value: 'http://opentelemetry-collector.default.svc.cluster.local:4317'
---
apiVersion: v1
kind: Service
metadata:
labels:
app: otelcol-demo-app
name: otelcol-demo-app
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
name: web
selector:
app: otelcol-demo-app
type: ClusterIP
kubectl get pods
NAME READY STATUS RESTARTS AGE
jaeger-agent-gsrxq 1/1 Running 0 3h40m
jaeger-cassandra-0 1/1 Running 0 3h40m
jaeger-cassandra-1 1/1 Running 0 3h38m
jaeger-cassandra-2 1/1 Running 0 3h37m
jaeger-cassandra-schema-dqbvd 0/1 Completed 1 3h40m
jaeger-collector-68dbd7f84b-tznkk 1/1 Running 5 (3h38m ago) 3h40m
jaeger-query-684f47b76b-pgjc6 2/2 Running 5 (3h38m ago) 3h40m
opentelemetry-collector-agent-hkksh 1/1 Running 0 9m3s
opentelemetry-operator-54dd75d64f-t5l9v 2/2 Running 4 (3h54m ago) 6h12m
otelcol-demo-app-596d9f8484-fn5jr 1/1 Running 0 3m1s
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jaeger-agent ClusterIP 10.43.47.36 <none> 5775/UDP,6831/UDP,6832/UDP,5778/TCP,14271/TCP 3h41m
jaeger-cassandra ClusterIP None <none> 7000/TCP,7001/TCP,7199/TCP,9042/TCP,9160/TCP 3h41m
jaeger-collector ClusterIP 10.43.209.8 <none> 14250/TCP,14268/TCP,14269/TCP 3h41m
jaeger-query ClusterIP 10.43.36.237 <none> 80/TCP,16685/TCP,16687/TCP 3h41m
opentelemetry-collector ClusterIP 10.43.8.16 <none> 6831/UDP,14250/TCP,14268/TCP,4317/TCP,4318/TCP,9411/TCP 25m
opentelemetry-operator ClusterIP 10.43.237.13 <none> 8443/TCP,8080/TCP 6h13m
opentelemetry-operator-webhook ClusterIP 10.43.51.200 <none> 443/TCP 6h13m
otelcol-demo-app ClusterIP 10.43.147.209 <none> 8080/TCP 6h
Application port forward
kubectl port-forward svc/otelcol-demo-app 61179:8080
Send access
curl http://localhost:61179/
curl http://localhost:61179/hello
curl http://localhost:61179/sayHello/demo2
The opentelemetry-collect-agent pod didn’t get any adding log:
2024-07-19T08:08:56.207Z info [email protected]/service.go:115 Setting up own telemetry...
2024-07-19T08:08:56.208Z info [email protected]/telemetry.go:96 Serving metrics {"address": "10.42.0.226:8888", "level": "Normal"}
2024-07-19T08:08:56.208Z info [email protected]/exporter.go:280 Development component. May change in the future. {"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-07-19T08:08:56.208Z info memorylimiter/memorylimiter.go:160 Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "total_memory_mib": 21982, "limit_percentage": 80, "spike_limit_percentage": 25}
2024-07-19T08:08:56.208Z info memorylimiter/memorylimiter.go:77 Memory limiter configured {"kind": "processor", "name": "memory_limiter", "pipeline": "metrics", "limit_mib": 17585, "spike_limit_mib": 5495, "check_interval": 5}
2024-07-19T08:08:56.208Z info [email protected]/exporter.go:280 Development component. May change in the future. {"kind": "exporter", "data_type": "logs", "name": "debug"}
2024-07-19T08:08:56.209Z info [email protected]/service.go:193 Starting otelcol-k8s... {"Version": "0.104.0", "NumCPU": 8}
2024-07-19T08:08:56.209Z info extensions/extensions.go:34 Starting extensions...
2024-07-19T08:08:56.209Z info extensions/extensions.go:37 Extension is starting... {"kind": "extension", "name": "health_check"}
2024-07-19T08:08:56.209Z info [email protected]/healthcheckextension.go:32 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Endpoint":"10.42.0.226:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2024-07-19T08:08:56.209Z info extensions/extensions.go:52 Extension started. {"kind": "extension", "name": "health_check"}
2024-07-19T08:08:56.210Z info [email protected]/otlp.go:102 Starting GRPC server {"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4317"}
2024-07-19T08:08:56.210Z info [email protected]/otlp.go:152 Starting HTTP server {"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4318"}
2024-07-19T08:08:56.210Z info healthcheck/handler.go:132 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"}
2024-07-19T08:08:56.210Z info [email protected]/service.go:219 Everything is ready. Begin running and processing data.
If install opentelemetry-collector with deployment mode
helm install opentelemetry-collector open-telemetry/opentelemetry-collector --set mode=deployment --set image.repository="otel/opentelemetry-collector-k8s" --set command.name="otelcol-k8s" -f values.yaml
The opentelemetry-collector pod can receive data. But got this error:
2024-07-19T08:50:13.705Z error exporterhelper/queue_sender.go:90 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService", "dropped_items": 2}
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:90
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:52
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:43
How to setup correctly?
1