I have an Azure App Service hosting an API that retrieves data from Azure Cosmos DB and Azure AI Search. Currently, the API configuration, including the Cosmos DB container and AI Search index to use, is loaded at startup. New data is loaded onto a new cosmos DB container and indexed. I want to switch to the newer cosmos DB container and AI Search index when refreshed without requiring app restarts and avoiding any downtime.
I’m looking for guidance on:
How to best design for the above scenario without requiring app restarts or disruption in service.
Monitoring and verifying the performance and availability of the API during configuration updates.
Any code examples, documentation references, or insights into best practices would be greatly appreciated. Thank you!
DB layer
You will want all app calls to be routed through a new DB layer
which knows the dynamic configuration for the proper database
to rely on ATM.
Equivalently, write a switch_to_newly_updated_server()
routine which arranges for current unmodified app code
to send queries to a new DB server.
push
Publish config update messages during index refreshes,
to kafka, 0mq, RabbitMQ, or similar pub-sub framework.
This lets you switch over to refreshed data
with sub-second latency.
Optionally send a “refreshing B, so only use A now!”
message when the refresh begins.
Send a “B is now current, stop using A!” message upon completion
pull
Choose some number of seconds K that you’re willing to tolerate stale data.
Roughly every K seconds, with random jitter,
a background thread will send a timestamp query
to candidate DB backend servers.
It could query updated_at from a single-row config table,
or it could do an efficient indexed SELECT MAX(stamp) FROM
some table which is often refreshed.
Clearly this query also verifies health status.
Winner is server with most recent timestamp.
Arrange for all subsequent app queries to go to the winner.