I need to process a batch of video items, format them, and call an asynchronous API to get additional data in a YAML format. However, I’m encountering an issue where the API responses are mostly empty, despite the tasks completing successfully. Below is the relevant code.

var fetchTasks = videos.Select(async item =>
{
    try
    {
        var formattedItem = new VideoItem
        {
            Title = item.Title,
            Subtitle = _extractService.GetCleanSubtitle(item.Subtitle ?? ""),
            Description = _extractService.GetCleanSubtitle(item.Description ?? ""),
        };

        if (string.IsNullOrEmpty(formattedItem.Subtitle) && string.IsNullOrEmpty(formattedItem.Description))
        {
            _logger.LogWarning($"Video {item.Id} has empty subtitle or description. Marking as not processed.");
            item.NotProcess = true;
            return new { item, processStatus = true, request = string.Empty, response = (string)null };
        }
        else
        {
            var builder = new RequestBuilder(formattedItem).WithYamlFormatPath(yamlPath);
            var yamlRequest = builder.Build();

            _logger.LogInformation($"Built YAML request: {yamlRequest}");

            var response = await _apiService.CallApiAsync(yamlRequest);

            _logger.LogInformation($"Response for video {item.Id}: {response}");

            return new { item, processStatus = true, request = yamlRequest, response };
        }
    }
    catch (Exception ex)
    {
        _logger.LogError($"Error fetching details for video {item.Id}: {ex.Message}");
        return null;
    }
});

var fetchedDetails = (await Task.WhenAll(fetchTasks)).Where(result => result != null);

CallApiAsync Method

public async Task<string> CallApiAsync(string response)
{
    try
    {
        var completionResult = await _openAiService.ChatCompletion.CreateCompletion(new ChatCompletionCreateRequest
        {
            Messages = new List<ChatMessage>
            {
                ChatMessage.FromSystem("You are a helpful assistant."),
                ChatMessage.FromUser(response)
            },
            Model = Models.Gpt_3_5_Turbo,
        });
        if (completionResult.Successful)
        {
            Console.WriteLine(completionResult.Choices.First().Message.Content);
            return completionResult.Choices.First().Message.Content;
        }
        return "";
    }
    catch (Exception ex)
    {
        // Log or handle exception as neede
        Console.WriteLine($"Error calling ChatGPT API: {ex.Message}");
        throw;
    }
}

After the task has been completed to process the response.

foreach (var item in fetchedDetails)
{
    var video = item.video;
    var request = item.request;
    var response = item.response;
     var dbResponse = new Response
     {
         Request = request,
         YtVideoId = item .Id,
         Response = response
     };

}

Here while saving the response to the database I am often getting the item.response as empty.

When I put breakpoint and debug it in local environment it works fine, but not after deployed.

It seems like the call to api has been started but not awaited properly within the batch. The tasks appear to complete without waiting for the response from CallApiAsync.

How can I refactor this code to wait to the external API call to get the response correctly and process it in parallel?