I have some micro-services running a bunch of workloads, and I’m trying to trace the “route” of a given workload. I’m using .NET Aspire, and have the 3 background worker projects added and a Web API project.
The first background worker starts the activity when a workload arrives. When it’s done, it pushes it into a RabbitMQ queue. I take the Id of the current activity and push that along in the headers of the message. Worker 2 uses the same method when it pushes a message to worker 3.
basicProperties.Headers = new Dictionary<String, Object>();
if (Activity.Current?.Id != null)
{
basicProperties.Headers.Add(DiagnosticsHandlerLoggingStrings.TraceParentHeaderName, Activity.Current?.Id);
}
In worker 2 and 3 I start an activity in this way:
if (basicProperties.Headers?.TryGetValue(DiagnosticsHandlerLoggingStrings.TraceParentHeaderName, out var parentActivityIdRaw) == true &&
parentActivityIdRaw is Byte[] traceParentBytes) {
parentActivityId = Encoding.UTF8.GetString(traceParentBytes);
}
return activitySource.StartActivity("Incoming message from queue", kind: ActivityKind.Consumer, parentId: parentActivityId);
Using HttpClient, the 3rd and last worker calls the Web API.
The trace page in Aspire shows worker 1, worker 3 and the Web API. Not worker 2. And moreover no of the events I add in worker 3 is shown in the details view, only the events from HttpClient.
What is the right way to continue an activity across boundaries? It seems HttpClient and also the IHttpActivityFeature can handle it, and is actually continuing the activity, but when I use Activity.Current in my own code, nothing get put in the events of the activity, at least it’s not shown in the trace details.
Looking at the trace, it seems like I’m getting the activity setup properly, but somehow I can’t add events to it myself, even though HttpClient can.
4
After looking at the link @martindotnet provided, and tinkering a bit, I finally got it working.
This is a solution for RabbitMQ (pre v7), so pushing and popping the trace across the boundaries are done for that, but!
The thing that is important, is what you need to start an activity on the receiving end.
First the consumer/receiver:
var context = Propagators.DefaultTextMapPropagator.Extract(new PropagationContext(
new ActivityContext(),
Baggage.Current),
ea.BasicProperties,
(properties, key) => {
if (properties.Headers?.TryGetValue(key, out var propertyValue) == true && propertyValue is Byte[] propertyValueBytes) {
return new List<String> { Encoding.UTF8.GetString(propertyValueBytes) };
}
return new List<String> { };
});
using var activity = this.transparencyObservability.ActivitySource.StartActivity(
"Consume Message",
ActivityKind.Consumer,
context.ActivityContext);
Activity.Current?.AddEvent(new ActivityEvent(
"Got message from queue, unpacked, ready to handle"));
And the publisher:
var basicProperties = channel.CreateBasicProperties();
Propagators.DefaultTextMapPropagator.Inject(
new PropagationContext(Activity.Current?.Context ?? default, Baggage.Current),
basicProperties, (msg, key, value) => {
msg.Headers.TryAdd(key, value);
});
That’s it. The Propagators are part of OpenTelemetry, so no magic!