How do I have Cloudflare respect Cache-Control directives for a FastAPI App deployed on Digitalocean App Platform?

Problem

I’m experiencing an issue with caching in my FastAPI application when it’s deployed in production on DigitalOcean App Platform, which uses Cloudflare. Locally, the FastAPI App caching works as expected using fastapi-cache2 and Redis, but in production, the cache seems to be controlled by Cloudflare, and my application’s cache-control headers are ignored.

Live humblAPI Deployment
humblAPI Github Repo

Description

Local Environment:

• Caching is handled by fastapi-cache2 with Redis as the backend.
• The x-fastapi-cache header correctly reflects cache HIT or MISS.
• Response times decrease on cache hits (x-process-time decreases).

Production Environment (with Cloudflare and DigitalOcean):

• Cloudflare caches the responses, evident from the cf-cache-status header.
• The x-fastapi-cache header always shows MISS, indicating my application’s cache isn’t utilized. What I think is happening is that the response has been cached, so the headers are still showing the same x-process-time and x-fastapi-cache, because the response is delivered quickly in around 20msvs the 1.2s process time. So it it showing the cached first response.
• Cloudflare ignores cache-control headers sent from the client.
• Even with Cloudflare’s development mode enabled (which is supposed to bypass cache), the issue persists. (not sure if this change is setup correctly on domain, but is the same domain, both the original DO URL and the custom Domain configured in DO)

Expected Behavior

• My FastAPI application’s caching logic should handle caching, not Cloudflare.
• Cache-Control directives like no-store or no-cache should instruct Cloudflare not to cache the response, so then the FastAPI App can also not cache and send fresh data.
• The x-fastapi-cache header should show HIT when appropriate.
• Subsequent requests should either be served from my application’s cache or processed anew if cache directives specify.

Actual Behavior

• Cloudflare caches the responses regardless of the cache-control directives.
• cf-cache-status header changes from MISS to HIT on subsequent requests.
• x-fastapi-cache header remains MISS even on subsequent requests.
• The x-process-time remains high/the same, indicating the application is fully processing each request. (I assume the response and its headers are cached since the process time is exactly the same and the response is served quicker than the process time suggests)
• Cache-control headers sent by the client are ignored after the first cached response. The only behaviour I have gotten to work is sending the first request with Cache-Control: no-store which doesnt cache the response. Example shown below…

Scenarios and Outputs

Local Environment

First Response (MISS):

curl -X 'GET' 
'http://localhost:8000/api/v1/user-table?symbols=XLE%2CXLU&membership=peon' 
-H 'accept: application/json' 
-I
    
HTTP/1.1 200 OK
date: Wed, 25 Sep 2024 14:46:04 GMT
server: uvicorn
content-length: 456
content-type: application/json
cache-control: max-age=86000
etag: W/1774689948358905112
x-fastapi-cache: MISS
x-process-time: 0.8525

Second Response (HIT):

curl -X 'GET' 
'http://localhost:8000/api/v1/user-table?symbols=XLE%2CXLU&membership=peon' 
-H 'accept: application/json' 
-I

HTTP/1.1 200 OK
date: Wed, 25 Sep 2024 14:41:14 GMT
server: uvicorn
content-length: 431
content-type: application/json
cache-control: max-age=85956
etag: W/-8458108256089087456
x-fastapi-cache: HIT
x-process-time: 0.0007

Third Response (w/ cache directive)

curl -X 'GET' 
'http://localhost:8000/api/v1/user-table?symbols=XLE%2CXLU&membership=peon' 
-H 'accept: application/json' 
-H 'Cache-Control: no-cache' 
-I

HTTP/1.1 200 OK
date: Wed, 25 Sep 2024 14:41:54 GMT
server: uvicorn
content-length: 431
content-type: application/json
cache-control: max-age=86000
etag: W/5034034483041136438
x-fastapi-cache: MISS
x-process-time: 0.4372

Observation:

• The x-fastapi-cache successfully changes from MISS to HIT with no Cache-Control Header.
• The x-fastapi-cache shows HIT when using Cache-Control:no-cache
• x-process-time significantly decreases on cache hit, showing a successful cache HIT.

Production Environment (with Cloudflare)

First Response:

curl -X 'GET' 
'https://api.humblfinance.io/api/v1/user-table?symbols=XLK%2CXLF&membership=peon' 
-H 'accept: application/json' 
-I

HTTP/2 200 
date: Wed, 25 Sep 2024 14:48:35 GMT
content-type: application/json
cache-control: max-age=86000
etag: W/-8963373628297997360
x-fastapi-cache: MISS
x-process-time: 1.1570
vary: Accept-Encoding
x-do-app-origin: d0a5619a-247a-4522-9871-90b758aa4e22
x-do-orig-status: 200
cf-cache-status: MISS
set-cookie: __cf_bm=R4VHBIbQrF810QB7N.THp6w3P21RZQT3YWQADdZHMQI-1727275715-1.0.1.1-YcSm6jyW4DnJew1XnlG7q2g_3Hso8hzd2wlEYlv3CW.Hh43Vz3fZmpC4Is9Ox4Xp43YKNjRF1g23lJh_SPAMgw; path=/; expires=Wed, 25-Sep-24 15:18:35 GMT; domain=.api.humblfinance.io; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 8c8bd0e00b1a43bb-EWR

Second Response:

curl -X 'GET' 
'https://api.humblfinance.io/api/v1/user-table?symbols=XLK%2CXLF&membership=peon' 
-H 'accept: application/json' 
-I

HTTP/2 200 
date: Wed, 25 Sep 2024 14:49:04 GMT
content-type: application/json
cache-control: max-age=86000
etag: W/-8963373628297997360
x-fastapi-cache: MISS
x-process-time: 1.1570
vary: Accept-Encoding
x-do-app-origin: d0a5619a-247a-4522-9871-90b758aa4e22
x-do-orig-status: 200
cf-cache-status: HIT
age: 29
set-cookie: __cf_bm=WfpRsdwXlKTDM2fftLfYF80XC9GoDFXVe2tCvppl2jA-1727275744-1.0.1.1-wMIMUrgEqHJIMRRLdSg.XiDov8qLd5uTO76SfMrtBiACKJqHM2dTj06E3zVBxMa2EexDCB28y7vOxAf1rsg7Xw; path=/; expires=Wed, 25-Sep-24 15:19:04 GMT; domain=.api.humblfinance.io; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 8c8bd19949f48c05-EWR

Third Response (w/ cache directive)

curl -X 'GET' 
'https://api.humblfinance.io/api/v1/user-table?symbols=XLK%2CXLF&membership=peon' 
-H 'accept: application/json' 
-H 'Cache-Control: no-cache' 
-I

HTTP/2 200 
date: Wed, 25 Sep 2024 14:50:55 GMT
content-type: application/json
cache-control: max-age=86000
etag: W/-8963373628297997360
x-fastapi-cache: MISS
x-process-time: 1.1570
vary: Accept-Encoding
x-do-app-origin: d0a5619a-247a-4522-9871-90b758aa4e22
x-do-orig-status: 200
cf-cache-status: HIT
age: 140
set-cookie: __cf_bm=c_Nsfz7eOqghHrA7P1Vfuu2pjzh1TP9LXTPSqvtOyMc-1727275855-1.0.1.1-SwfuWFoicklVE.MmW2tvZ4EFEUhh0aBnWkWpn3TuxJh7mLaWjWFMb1jnP1tT.3dK1jwcxiOcy2oOlE8VfxhRnA; path=/; expires=Wed, 25-Sep-24 15:20:55 GMT; domain=.api.humblfinance.io; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 8c8bd44fb90cc40e-EWR

--------- Using no-store ------------

curl -X 'GET' 
'https://api.humblfinance.io/api/v1/user-table?symbols=XLK%2CXLF&membership=peon' 
-H 'accept: application/json' 
-H 'Cache-Control: no-store' 
-I

HTTP/2 200 
date: Wed, 25 Sep 2024 14:51:46 GMT
content-type: application/json
cache-control: max-age=86000
etag: W/-8963373628297997360
x-fastapi-cache: MISS
x-process-time: 1.1570
vary: Accept-Encoding
x-do-app-origin: d0a5619a-247a-4522-9871-90b758aa4e22
x-do-orig-status: 200
cf-cache-status: HIT
age: 191
set-cookie: __cf_bm=eEPauzaFBlHbIqNqh0MrUqqdwt1M.mWuA0DWRDfd5ec-1727275906-1.0.1.1-RcI3ozlMa7E4BUrh3LmFfTKyGpk092TAbIcS5BcPvxRfhtUb2I2VNDqveb5qPVQckjN33Slu9iN66QliA_a.sw; path=/; expires=Wed, 25-Sep-24 15:21:46 GMT; domain=.api.humblfinance.io; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 8c8bd58d6d34c32e-EWR

Observation:

• cf-cache-status changes from MISS to HIT, and still shows HIT, even with Cache-Control:no-cache
• x-fastapi-cache remains MISS on both requests. (cached header form first response?)
• x-process-time remains the same, indicating no caching improvement from the application side. (cached header form first response?)
• Cloudflare serves cached responses despite cache-control directives.

Only Working Cache Bypass

The only thing that has partially worked, but doesn’t solve the issue, is sending a request to a new URL with Cache-Control: no-store. It does not cache the response, and as long as I keep the Cache-Control: no-store, and the cf-cache-status will always show MISS.

First Repsonse:

curl -X 'GET' 
'https://api.humblfinance.io/api/v1/user-table?symbols=SNAP%2CXLF&membership=peon' 
-H 'accept: application/json' 
-H 'Cache-Control: no-store' 
-I

HTTP/2 200 
date: Wed, 25 Sep 2024 14:55:03 GMT
content-type: application/json
x-process-time: 1.1306
vary: Accept-Encoding
x-do-app-origin: d0a5619a-247a-4522-9871-90b758aa4e22
cache-control: private
x-do-orig-status: 200
last-modified: Wed, 25 Sep 2024 14:55:03 GMT
cf-cache-status: MISS
set-cookie: __cf_bm=8sYFb_QKEIJvo7v8WHLYNQROYI6cIi5L.A_.2WdhW4Y-1727276103-1.0.1.1-D7x0TSp6mHkJRQlu5nRXWztEXxq8qDvRZWejZ6FO8wNoYEeB4cGw5.LOyFos.pTJ_Z81ew8bqERyHN5P4bCEYQ; path=/; expires=Wed, 25-Sep-24 15:25:03 GMT; domain=.api.humblfinance.io; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 8c8bda560da78c5d-EWR

Second Response:

curl -X 'GET' 
'https://api.humblfinance.io/api/v1/user-table?symbols=SNAP%2CXLF&membership=peon' 
-H 'accept: application/json' 
-H 'Cache-Control: no-store' 
-I
HTTP/2 200 
date: Wed, 25 Sep 2024 14:56:04 GMT
content-type: application/json
x-process-time: 1.0136
vary: Accept-Encoding
x-do-app-origin: d0a5619a-247a-4522-9871-90b758aa4e22
cache-control: private
x-do-orig-status: 200
last-modified: Wed, 25 Sep 2024 14:56:04 GMT
cf-cache-status: MISS
set-cookie: __cf_bm=DM3GpR.5iiA9uwOwsw0BlAKqkF_ThT4kMxrmkX35Hwg-1727276164-1.0.1.1-3G_xjzy5BCKl7aXqw86YgLIKB.x_e3fZvf3Q8yAPEUOYKWHDlBEbPmEp4BrknHbvrvVkobC2qzVqjtCvdh2cOQ; path=/; expires=Wed, 25-Sep-24 15:26:04 GMT; domain=.api.humblfinance.io; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 8c8bdbd5dd234378-EWR

Observations:

• x-process-time changes indicating new data from the FastAPI server.
• As soon as I change the directive, the data is cached and no subsequent directives work to bypass the cache.

Question:

Why is no-store working on the first request but then none afterwards?

Attempts to Resolve

Bypassing Cloudflare Cache:

• Sent requests with Cache-Control: no-cache, no-store, and max-age=0.
• Cloudflare continues to serve cached responses (cf-cache-status: HIT).

Cloudflare Development Mode:

• Enabled development mode to bypass the cache, not sure if the domain is routing properly, I would think this would work?
• Issue persists; responses are still served from Cloudflare’s cache.

Unique URLs with Cache-Control: no-store:

• Initial requests with Cache-Control: no-store result in cf-cache-status: MISS. As long as I keep this header, the requests will NOT be cached.
• After the header is changed, or the data cached. Subsequent requests, even with no-store, result in cf-cache-status: HIT.

Verified Application Cache Locally:

• Locally, the x-fastapi-cache header correctly changes to HIT.
• Application-level caching works as expected without Cloudflare.

Infrastructure Details

• Application Framework: FastAPI
• Caching Library: fastapi-cache2 with Redis backend

Local Environment:

• Running with uvicorn
• Testing with curl commands and via localhost:8000/docs

Production Environment:

• Hosting: DigitalOcean App Platform
• Reverse Proxy/CDN: Cloudflare?
• TLS Termination: Handled by Cloudflare?
• Origin Server: FastAPI application running on DigitalOcean

Redis Server:

• Accessible from both local and production environments
• Used for application caching

Questions:

• How can I configure Cloudflare to respect Cache-Control headers from the client or server, so that my application’s caching logic works as intended?
• Is there a way to bypass Cloudflare’s cache for specific endpoints or query parameters, allowing my FastAPI application’s cache to be effective?
• Why is the x-fastapi-cache header always MISS in production, even on subsequent requests? Is Cloudflare preventing my application from accessing its own cache?

What I’ve Tried So Far

• Setting Cache-Control headers to no-store, max-age=0, no-cache, revalidate

Conclusion

It seems that Cloudflare is caching the responses and not respecting the Cache-Control headers from either the client or the server. This interferes with my application’s caching logic using fastapi-cache2.

I need guidance on how to configure my application so that:

• When a Cache-Control:no-cache directive is used, the Cloudflare cache is bypassed and fresh data is served from the FastAPI App, which fastapi-cache2 will also respect to send fresh data.
• Responses are not served from Cloudflare’s cache when they shouldn’t be.

Any insights or solutions would be greatly appreciated!