We have written an application that spawns at least 9 parallel processes. All processes generate a lot of logging information.
Currently we are using Python’s QueueHandler to consolidate all logs into one file. Unfortunately, this sometimes results in very messy files which are hard to read (making it difficult to track exactly what’s going on in one thread).
Do you think it is a viable option to separate all messages into dedicated files? Or will this make things even messier, due to the high number of files?
What are your general experiences when writing log files for multiprocessed/multithreaded applications?
1
Tag each log line with the process id as an identifier, then use grep when looking at the logs. Or something else besides the process id.
I prefer letting each process log to it’s own file and later merge them together if needed/wanted. This way I don’t have to concurrency/contention on one single log file. Being separate processes the log events are usually pretty disjoint anyhow.
Performance wise this is the most efficient way to handle it. The joining of log files is typically wasteful since no one reads most of it anyhow, I would stick to doing so on demand.
If you are mostly using your log files as debug instrumentation instead of auditing purposes and it’s helping you to get your code to work or debug multi-threading issues the joining of log files may result in loss of event ordering unless you can use an acceptable high enough resolution timer to include in your log events. However your safest bet would be to log to a log service of some sort that synchronizes for you.
1
It really depends on your application, and how you analyse the logs. Consider the high end server extreme case where you have many threads all interacting. Consoldating this at log time into a single queue, and writing into a file in time order allows you to determine fairly precisely the order events occured in is one solution, but a better solution would be to have multiple logs, have each log event record the tick time, and then use splunk to perform the analysis.