I have several services communicating with each other via SignalR connection. They are both standard .net core web api services, let’s say I have service A and service B.
Service A has SignalR hub configured and accepting connections, Service B is connected to it using HubConnection class from Microsoft.AspNetCore.SignalR.Client package. And is listening to some messages from service A with: hubConnection.On('...')
Service B also has a signalR Hub and is accepting connections from website clients.
Service A is sending about 2000 messages in a minute to service B, and service B also streams these messages to all the connected clients. there are about 80 client connections to service B at the same time.
The issue is that the messages from Service A are arriving very late, they are about 2 minutes late. I am logging the line when the message is sent from service A and is received in service B and it’s about 2 minute difference. I don’t have any ideas why this happens, 2k messages in a minute isn’t that much as I Believe.
here’s a diagram:
Service B does this type of logic on each received message:
hubConnection.On(method.Name, parameterTypes, @params =>
{
return Task.Run(() =>
{
var activity = new Activity("HubListener");
try
{
activity.Start();
activity.AddTag("TraceId", Guid.NewGuid().ToString());
activity.AddTag("ArrivalTime", DateTime.UtcNow);
var obj = _serviceProvider.GetRequiredService(method.DeclaringType)
method.Invoke(obj, @params);
}
catch (Exception e)
{
Log.Error(e, "Error In Hub Listener");
}
finally
{
activity.Stop();
}
});
});
the method is taken via reflection from my BroadcastService class (there are other methods too that’s why use reflection to subscribe dynamically):
public class BroadcastService {
private readonly IHubContext<BaseHub> _baseHubContext;
public BroadcastService(IHubContext<BaseHub> baseHubContext)
{
_baseHubContext = baseHubContext ?? BaseHub.CurrentContext;
}
public void BroadcastToClients(Message messageFromServiceA)
{
// it takes about 2 minutes for message to arrive at this place after it is sent from Service A.
// clients are separated in groups
foreach(var group in clientGroups){
_baseHubContext
.Clients
.Group(group.Name)
.SendAsync("onMessage",messageFromServiceA)
}
}
}
If I run service B on local machine and there are no connections from clients the messages are not delayed, this has something to do with number of client connections but 80 is not a lot. I don’t know if the messages on hubConnection are executed sequentially, but this should not be an issue because I use Task.Run and this should run handler asynchronously.
I would appreciate any kind of help, I will edit the question if more details are required.
3
I finally found answer to my question, the issue was that SignalR client (as well as server) processes messages sequentially, I understood this by logging start and end timestamps of the handlers. After some research I found out that this was by design in SignalR Core, here’s similar question and answer.
To fix my issue I just run task in background and don’t wait for it:
old code:
hubConnection.On(method.Name, parameterTypes, @params =>
{
return Task.Run(() =>
{
...
});
});
new code:
hubConnection.On(method.Name, parameterTypes, @params =>
{
_ = Task.Run(() =>
{
...
});
return Task.CompletedTask;
});
Now it works just fine.