I’ve been banging my head on this problem for a few days now, and have gotten a bit stuck, so I thought I’d ask here. Would really appreciate any help.

Context:

The issue that I’m currently facing with my site is that we are getting a lot of nginx 499 errors being raised when clients disconnect before hitting a refresh_token endpoint. (The purpose of this endpoint is to generate a new access token and refresh token from the provided refresh token in the request, and return it to the client). We also block the refresh_token being used to make this request. The problem is, that when the client disconnects, their local storage does not get updated with the new token, and so, the next time they reconnect they are forcibly logged out, as their local storage token is treated as blocked when it hits the refresh_token endpoint the following time.

High level solution:

What I wanted to do to approach this problem was to essentially catch the 499 error raised by nginx when it occurs, and hit another endpoint on my backend to cache this token. On subsequent hits for this user to the refresh_token endpoint, it can also check if the token exists in the cache, and generate a new one in those cases.

The Problem:

So, from what I’ve found, the phase in which I can catch the nginx response status at the end of the request is log_by_lua (specifically in my case, log_by_lua_file). The problem is that log_by_lua_file does not have access to cosocket apis, as per these docs, and therefore I am unable to make request to new endpoints with lua scripts here. Now the workaround that the docs recommend is to use ngx.timer.at (Context for the workaround is in the same place as the cosocket apis information docs I linked to previously). I have implemented this workaround using resty-http, however, am currenlty getting 404 errors on the endpoint. Is there something I am missing? To be sure that I am using the right endpoint, I also made a curl request from the nginx pod directly to the endpoint, and it worked. (I also logged the url to ensure that it is resolving to the correct url, since I know that ngx.timer doesn’t have access to ngx.var, so I want to make sure that data.authenticator field had been passed correctly).

My script that I am trying to use with log_by_lua_file is below: 
-- Function to send the original request data to the internal endpoint on 499 error
local function send_499_notification(premature, data)
    local http = require "resty.http"
    local cjson = require "cjson"

    if premature then
        ngx.log(ngx.WARN, "Premature timer trigger")
        return
    end
    ngx.log(ngx.WARN, "Request url: ", "http://" .. data.authenticator .. "/handle_499_error/")
    ngx.log(ngx.WARN, "Request method: ", data.method)
    ngx.log(ngx.WARN, "Request body: ", data.body)
    ngx.log(ngx.WARN, "Request headers: ", cjson.encode(data.headers))
    ngx.log(ngx.WARN, "Request args: ", cjson.encode(data.args))

    local httpc = http.new()
    local res, err = httpc:request_uri("http://" .. data.authenticator .. "/handle_499_error/", {
        method = data.method,
        body = data.body,
        query = data.args,
        headers = data.headers,
    })

    if not res then
        ngx.log(ngx.WARN, "Failed to notify internal endpoint: ", err)
    else
        ngx.log(ngx.WARN, "Successfully notified internal endpoint for 499 error, res status ", res.status)
    end
end

-- Function to handle 499 error
local function handle_499()
    local req_body = ngx.ctx.req_body or ""
    local req_headers = ngx.ctx.req_headers or {}
    local req_args = ngx.ctx.req_args or {}
    local req_method = ngx.ctx.req_method or ""

    local data = {
        authenticator = ngx.var.authenticator,
        method = req_method,
        body = req_body,
        args = req_args,
        headers = req_headers,
    }

    local ok, err = ngx.timer.at(0, send_499_notification, data)
    if not ok then
        ngx.log(ngx.ERR, "Failed to create timer: ", err)
    end
end

-- Retrieve NGINX status code
local nginx_status = ngx.status

if nginx_status == 499 then
    handle_499()
end

ngx.log(ngx.WARN, "NGINX status code: ", nginx_status)

While I cannot share the whole log sequence, I do see the final log, with:

2024/07/22 11:15:59 [warn] 7#0: *13017 [lua] handle_499.lua:27: Successfully notified internal endpoint for 499 error, res status 404, context: ngx.timer, client: 10.76.170.103, server: 0.0.0.0:80

Also, just for reference, the ngx.ctx fields were created in the access_by_log phase, I am aware that access request methods and fields are not available in log_by_lua, which was why I saved them in ngx.ctx earlier.