We have datadog setup in our organisation for monitoring.
We use datadog to check the metrics and view the exception. We don’t use it to see the logs.

We are planning to replace the datadog javaagent with opentelementry javaagent(due to better instrumentation support for vert.x) but keep our metrics that is flowing to the datadog backend intact.

We are following the documentation https://docs.datadoghq.com/opentelemetry/interoperability/otlp_ingest_in_the_agent/?tab=host

Our configuration for opentelementry java agent(Version 2.3.0) is

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://<datadog agent ip>:4318/v1/logs
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<datadog agent ip>:4318/v1/metrics
OTEL_EXPORTER_OTLP_ENDPOINT=http://<datadog agent ip>:4318/v1/{signal}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<datadog agent ip>:4318/v1/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=supply

Datadog agent version – 7.51.0 and configuration

## OTLP ingest related configuration
  otlp:
    receiver:
      protocols:
        # datadog.otlp.receiver.protocols.grpc - OTLP/gRPC configuration
        grpc:
          # datadog.otlp.receiver.protocols.grpc.enabled -- Enable the OTLP/gRPC endpoint
          enabled: false
          # datadog.otlp.receiver.protocols.grpc.endpoint -- OTLP/gRPC endpoint
          endpoint: "0.0.0.0:4317"
          # datadog.otlp.receiver.protocols.grpc.useHostPort -- Enable the Host Port for the OTLP/gRPC endpoint
          useHostPort: true

        # datadog.otlp.receiver.protocols.http - OTLP/HTTP configuration
        http:
          # datadog.otlp.receiver.protocols.http.enabled -- Enable the OTLP/HTTP endpoint
          enabled: true
          # datadog.otlp.receiver.protocols.http.endpoint -- OTLP/HTTP endpoint
          endpoint: "0.0.0.0:4318"
          # datadog.otlp.receiver.protocols.http.useHostPort -- Enable the Host Port for the OTLP/HTTP endpoint
          useHostPort: true
    logs:
      # datadog.otlp.logs.enabled -- Enable logs support in the OTLP ingest endpoint
      enabled: false

Goal -> We want to send the metrics to the datadog backend using the datadog agent by the opentelemetry instrumentation(java agent) as shown in the image above.

We are able to access the datadog agent http://:4318/v1/metrics from the service pod where we added the open telementry instrumentation.

We are unable to view the metrics for the service in the datadog UI for the service instrumented with opentelementry java agent.

We are seeing one exception in that service pod logs file

[otel.javaagent 2024-05-14 16:10:24:760 +0530] [OkHttp http://172.31.22.114:4318/...] WARN io.opentelemetry.exporter.internal.http.HttpExporter - 
Failed to export logs. Server responded with HTTP status code 404. Error message: Unable to parse response body, HTTP status message: Not Found