I have a service which runs as a background job. It’s duty is to try and hit some external service for some external update. Now, I am to attempt retrying an HTTP request against the service for a specified number of times. Ultimately, after the third try, I am to backoff for a few minutes, and try one more time. The service should return on failure of the fourth try.

My service roughly looks like this:

 var cts = new CancellationTokenSource();
                var cancellationToken = cts.Token;
                //create a task responsible for syncing data
                Func<Task<RequestResponse<SomeResponse>>>> task = async () =>
                {
                    var url = $"https://google.com/v1/abcdefhj";
                    var response = await _requestService.Post<SomeResponse>>(
                                       url,
                                       customerProfile,
                                       nameof(SomeOperation),
                                       _auth,
                                       "Bearer"
                                   );
                    return new RequestResponse<SomeResponse>
                    {
                        Data = response.Data,
                        Successful = response.Successful,
                        Message = response.Message
                    };
                };
                
                var response = await AttemptUpdate(task, cancellationToken, _options.Value.MaxRetries);
              
                if (!response.Successful)
                {
                    _logger.LogDebug("Setting backoff time for {BackoffTime} minutes", _options.Value.BackoffTime);
                    //set the backoff time here
                    await Task.Delay(TimeSpan.FromMinutes(_options.Value.BackoffTime), cancellationToken);
                    
                    var url = $"https://google.com/v1/abcdefghij";
                    var responseFromRetry = await _requestService.Post<SomeResponse>>(
                                       url,
                                       customerProfile,
                                       nameof(SomeAction),
                                       _auth,
                                       "Bearer"
                                   );
                    _logger.LogDebug("Response from retry --> {Response}",
                        JsonConvert.SerializeObject(responseFromRetry));
                    response =  new RequestResponse<SomeResponse>>
                    {
                        Data = responseFromRetry.Data,
                        Successful = responseFromRetry.Successful,
                        Message = responseFromRetry.Message
                    };

                    if (!response.Successful)
                    {
                        _logger.LogDebug("Failed to sync profile after 3 attempts. Skipping.");
                        break;
                    }
                }

For some weird reason, my service never hits _logger.LogDebug("Response from retry --> {Response}", although the condition for failure is true.

This is how my retry function is:

private async Task<RequestResponse<SomeResponse>> AttemptUpdate(
            Func<Task<RequestResponse<SomeResponse>>>> task, CancellationToken cancellationToken,
            int maxAttemptCount = 3)
        {
            for (var attempted = 0; attempted < maxAttemptCount; attempted++)
            {
                _logger.LogDebug("Attempt {Attempted} to sync customer profile with Star Assurance", attempted + 1);
                
                // Check for cancellation
                if (cancellationToken.IsCancellationRequested)
                {
                    _logger.LogDebug("Cancellation requested. Exiting retry loop.");
                    return new RequestResponse<SomeResponse>>
                    {
                        Successful = false,
                        Message = "Operation cancelled"
                    };
                }
                var result = await task();
                _logger.LogDebug("Received response --> {Response}",
                     JsonConvert.SerializeObject(result));
                
                // Result is successful so we can return
                if (result.Successful)
                {
                    _logger.LogDebug("Succeeded to sync customer profile with response --> {Response}",
                        result.Data?.Message);
                    return result;
                }
                
                _logger.LogDebug("Failed to sync customer profile with response --> {Response}",
                    result.Data?.Message);
            }
            
            return new RequestResponse<SomeResponse>>
            {
                Successful = false,
                Message = "Failed to sync customer profile"
            };
        }

I have a feeling the introduction of asynchronicity is causing something I cannot detect. Or am I managing the threads wrongly? My desired goal is to return when the attempt post backoff fails. My backoff of 10 minutes does not run, but what happens is the list is iterated over again and enters AttemptUpdate again.