I have a Function App targeting .NET 6.0 that logs this error every 10-20 minutes:
22:29:45Z [Verbose] Handling WorkerErrorEvent for runtime:dotnet-isolated,
workerId:dotnet-isolated. Failed with: System.TimeoutException: The
operation has timed out. at
Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel.PendingItem.OnTimeout()
in /_/src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs:line 176722:29:45Z [Verbose] Attempting to dispose webhost or jobhost channel for workerId: ‘########-####-####-#################’, runtime: ‘dotnet-isolated’
22:29:45Z [Verbose] Disposing language worker channel with id:########-####-####-#################
22:29:45Z [Verbose] Disposed language worker channel with id:########-####-####-#################
22:29:45Z [Verbose] No initialized worker channels for runtime ‘dotnet-isolated’. Delaying future invocations
22:29:45Z [Verbose] Restarting worker channel for runtime: ‘dotnet-isolated’
22:29:45Z [Verbose] Adding jobhost language worker channel for runtime: dotnet-isolated. workerId:########-####-####-#################
22:29:45Z [Information] Worker process started and initialized.
Instead of the Restarting worker channel for runtime: 'dotnet-isolated' step, this error is occasionally logged instead:
[Error] Exceeded language worker restart retry count for runtime:dotnet-isolated. Shutting down and proactively recycling the Functions Host to recover
[Verbose] Hosting stopping
[Verbose] Stopping file watchers.
[Verbose] Waiting for RpcFunctionInvocationDispatcher to shutdown
[Verbose] Draining invocations from language worker channel completed. Shutting down ‘RpcFunctionInvocationDispatcher’
[Information] Stopping JobHost
[Verbose] Stopping ScriptHost instance ‘########-####-####-#################’.
[Information] Stopping the listener ‘Microsoft.Azure.WebJobs.Extensions.Http.HttpTriggerAttributeBindingProvider+HttpTriggerBinding+NullListener’ for function ‘######’
[Information] Stopped the listener ‘Microsoft.Azure.WebJobs.Extensions.Http.HttpTriggerAttributeBindingProvider+HttpTriggerBinding+NullListener’ for function ‘######’
[Information] Job host stopped
[Verbose] Stopped ScriptHost instance ‘########-####-####-#################’.
[Verbose] Hosting stopped
[Verbose] Disposing ScriptHost.
[Verbose] Initiating background SyncTriggers operation
From the Azure Portal > Diagnose and solve problems > Availability and Performance > Functions that are not triggering > Last 24 hours:
| Exception Type | Total Occurrences | Exception Message |
|---|---|---|
| System.TimeoutException | 123 | Loading function failed. The operation has timed out. |
| Exceeded language worker restart retry count for runtime:dotnet- | 41 | Exceeded language worker restart retry count for runtime:dotnet-isolated. Shutting down and proactively recycling the Functions Host to recover |
| System.Net.Sockets.SocketException | 19 | Exception encountered while listening to EventStream. An existing connection was forcibly closed by the remote host. |
The first request after the Function App idles for 20 minutes or more takes around 5 – 10 seconds longer to process, even with the Function App set to Always On. Any requests directly after that is processed within a second.
Environment Variables:
FUNCTIONS_WORKER_RUNTIME: dotnet-isolated
FUNCTIONS_EXTENSION_VERSION: ~4
The errors stopped and the initial requests are no longer delayed after changing Settings > Environment variables > FUNCTIONS_WORKER_RUNTIME from dotnet-isolated to dotnet in all Function Apps in the App Service Environment.