I’m using the CosmosDB Linux emulator in a Docker container on WSL2. I’m connecting from a .NET Core app (in Windows or WSL2).

When connecting via gateway mode, requests to replace a document (UpsertItemAsync and ReplaceItemAsync) fail with a 408 timeout (see messages below) while read and write operations succeed.

Using direct mode works fine. All testing so far is with clients that trust the self-signed certificates (to remove a variable).

Un-trusted Certificate

Related and useful for others, this is what led me here. I do not want to trust these self-signed certificates.

Microsoft’s documentation led me to the gateway issue: they recommend Gateway mode, overriding only the HTTP SSL validation.

This broke document replacement (see above). My solution has been to keep the ConnectionMode as Direct and override the generic TCP certificate validation request handler.

var options = new CosmosClientOptions()
{
  ServerCertificateCustomValidationCallback = (certificate, chain, errors) => true
};
var client = new CosmosClient(endpoint, key, options);

The question

Why does gateway connection mode fail for document replacements specifically in the Linux docker container? (Windows Emulator locally installed works fine).

Gateway error logs

These eventually result in 408 timeout. Note that my connection endpoint is to https://127.0.0.1:8081 so I know I can already reach this endpoint from my client.

ocDBTrace Warning: 0 : ClientRetryPolicy: Gateway HttpRequestException Endpoint not reachable. Failed Location: https://127.0.0.1:8081/; ResourceAddress: dbs/****/colls/****/docs/****
DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://127.0.0.1:8081/ unavailable for read
DocDBTrace Information: 0 : Current WriteEndpoints = (https://127.0.0.1:8081/) ReadEndpoints = (https://127.0.0.1:8081/)
DocDBTrace Information: 0 : Endpoint https://127.0.0.1:8081/ unavailable for Read added/updated to unavailableEndpoints with timestamp 08/06/2024 23:25:46
DocDBTrace Information: 0 : GlobalEndpointManager: Marking endpoint https://127.0.0.1:8081/ unavailable for Write
DocDBTrace Information: 0 : Current WriteEndpoints = (https://127.0.0.1:8081/) ReadEndpoints = (https://127.0.0.1:8081/)
DocDBTrace Information: 0 : Endpoint https://127.0.0.1:8081/ unavailable for Write added/updated to unavailableEndpoints with timestamp 08/06/2024 23:25:46
DocDBTrace Information: 0 : ClientRetryPolicy: Failover happening. retryCount 1
DocDBTrace Information: 0 : Current WriteEndpoints = (https://127.0.0.1:8081/) ReadEndpoints = (https://127.0.0.1:8081/)

With a final error from the .NET SDK:

Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: RequestTimeout (408); Substatus: 0; ActivityId: 157f65ad-1735-4d39-996f-605b76721427; Reason: (GatewayStoreClient Request Timeout. Start Time UTC:08/06/2024 23:25:46; Total Duration:64999.1244 Ms; Request Timeout 65000 Ms; Http Client Timeout:65000 Ms; Activity id: 157f65ad-1735-4d39-996f-605b76721427;);

Configuration

• Windows 11 with WSL2 running Ubuntu 22.04
• Client: .NET SDK 8.0.303 / Azure Cosmos SDK 3.41.0
• Cosmos DB Emulator (2.14.0) – best guess from tag in JSON response (:latest as of 2024-08-07)

My Cosmos service in docker-compose.yml is as follows:

services:
  cosmos:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
    environment:
      AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE: "127.0.0.1"
      AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE: "true"    
    ports:
      - 8081:8081
      - 10251:10251
      - 10252:10252
      - 10253:10253
      - 10254:10254