Summary

Expecting: consumer and producer have the same trace ID
Current result: their traceIds are different.

Description

I am enabling tracing in my spring cloud, spring boot 3 application.

The tracing tool used is micrometer and zipkin. The Sleuth is not compatible for spring boot 3.

I have enabled the producer trace and I can see the traceParent in the ProducerRecord. Here is my consumer send payload code.

public void sendNotificationPayload(NotificationPayload payload) {
        Span span = tracer.currentSpan();
        CompletableFuture<SendResult<String, Object>> future = template.send("notification", payload);
        // future.get() will block
        future.whenComplete((res, err) -> {
            if (Objects.isNull(err)) {
                log.info("send message=[{}] with offset=[{}]", payload.toString(), res.getRecordMetadata().offset());
                res.getProducerRecord().headers().forEach(
                        (h) -> log.info("header:{}, {}", h.key(), new String(h.value(), StandardCharsets.UTF_8))
                );
            } else {
                log.error("Unable to send message=[{}] due to: {}", payload.toString(), err.getMessage());
            }
        });
    }

Here is the producer config:

@Bean
    public KafkaTemplate kafkaTemplate(ProducerFactory producerFactory) {
        KafkaTemplate t = new KafkaTemplate(producerFactory);
        t.setObservationEnabled(true);
        t.setObservationConvention(new KafkaTemplateObservationConvention() {
            @Override
            public KeyValues getLowCardinalityKeyValues(KafkaRecordSenderContext context) {
                return KeyValues.of("topic", context.getDestination(),
                        "id", String.valueOf(context.getRecord().key()));
            }
        });
        return t;
    }

Here is consumer config:

    @Autowired
    private Tracing tracing;

    @Bean
    public KafkaTracing kafkaTracing(Tracing tracing) {
        return KafkaTracing.newBuilder(tracing).build();
    }

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        // Creating a Map of string-object pairs
        Map<String, Object> config = new HashMap<>();
        return new DefaultKafkaConsumerFactory<>(config);
    }

    // solution: https://piotrminkowski.com/2023/11/15/kafka-tracing-with-spring-boot-and-open-telemetry/
    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> listenerFactory(ConsumerFactory<String, String> consumerFactory) {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.getContainerProperties().setObservationEnabled(true);
        factory.setConsumerFactory(consumerFactory);
        return factory;
    }

I can clearly see the ProducerRecord has traceParent, here is producer log

2024-06-20T16:43:41.323-05:00  INFO [AUTH-SERVICE,6674a28d99aabe22eb47ae662c67a12d,8655877e8686f26f] 69037 --- [AUTH-SERVICE] [VICE-producer-1] [6674a28d99aabe22eb47ae662c67a12d-8655877e8686f26f] c.j.a.kafka.KafkaMessagePublisher        : header:traceparent, 00-6674a28d99aabe22eb47ae662c67a12d-8655877e8686f26f-01

But consumer does not.

Could you help me on this, thanks a lot

Reference

same issue with me: https://www.reddit.com/r/SpringBoot/comments/1b0qs23/struggling_with_propagating_traceid_to_kafka/

Kafka Tracing with Spring Boot and Open Telemetry: https://piotrminkowski.com/2023/11/15/kafka-tracing-with-spring-boot-and-open-telemetry/

Propagating & Managing TraceID across multiple events in Kafka

https://github.com/spring-cloud/spring-cloud-stream/issues/2647

https://www.linkedin.com/pulse/tracing-spring-boot-3-zipkin-kafka-transport-%D0%BD%D0%B8%D0%BA%D0%B8%D1%82%D0%B0-%D0%BD%D0%BE%D1%81%D0%BE%D0%B2-errwf/

I tried explicitly enabled producer and consumer observed as true.
I tried to debug: producer has traceParent, but consumer has different