Context

I have a python API with dozen of routes. I created a custom logger to log every I/O of my app. The file which is the destination of my TimedRotatingFileHandler is missing a lot of logs that appear in the stderr (using StreamHandler). I figured it must be because of concurrency since my app can receive dozens of calls a second.

Here is my config dict:

version: 1
disable_existing_loggers: true
formatters:
  io_truncated:
    (): src.logging.formatters.IOFormatter
    truncate_payload: 100
  io_full:
    (): src.logging.formatters.IOFormatter

handlers:
  io_file:
    class: logging.handlers.TimedRotatingFileHandler
    level: INFO
    formatter: io_full
    filename: logs/io-payload.log
    when: midnight
    backupCount: 7
  io_stderr:
    class: logging.StreamHandler
    level: INFO
    stream: ext://sys.stderr
    formatter: io_truncated

loggers:
  IO:
    level: INFO
    propagate: false
    handlers:
    - io_file
    - io_stderr

my I/O logger uses extra arguments to populate the log. In the end, it looks something like

2024-02-29 20:11:36 | INFO I/O | papai_predict.py:52 <<< MinIO (test/bonjour)
2024-02-29 20:11:36 | INFO I/O | papai_predict.py:68 >>> MinIO (test/bonjour)
2024-02-29 20:11:36 | INFO I/O | papai_predict.py:12 <<< API (/hey) | {"data": {"input_dataset_bucket":"BUCKET_NAM...
2024-02-29 20:11:36 | INFO I/O | papai_predict.py:450 >>> API (/hey) | {"response": "SUCCESS"}
2024-02-29 20:11:36 | ERROR I/O | papai_predict.py:990 >X> API (/hey) | Error while trying to send this message to url | {"response": "SUCCESS"}

Issue

I tried creating a queue handler, but I have no control over the spawning of the process since Gunicorn or celery (there are two different services running independently but I have the same issue) are doing that for me. This answer was the most relevant thing but, again, I cannot gain control over the creation of new workers.

Some lead

It then came to my mind to mimick what microsoft office is doing with using the filesystem as a global lock. I could create a file when a thread is logging in the file, then delete it when the logger is done writing the log to the file. I’m just afraid of the overhead it would create.

I could also create one file per thread but it would become messy real fast.

Question

Do you have any other idea on how to implement that?

Thank you for your time.