I’m writing a log parser that continuously looks for specific patterns and then fires off events to another system when it finds these matches. The firing of the event is time critical to catch errors or ensuing problems. This part is non-negotiable in terms of design.
The log files belong to Applications and the Operating System; thus they can be rapid in logging, large in size and rotate based on time, or size.
What is a better approach to reading these log files, detecting when they change, and reading from their last read location as I wrote code to do this using a RandomAccessFile which kept the last read location for every file.
For dealing with large files and rapid logging I did this:
- DataProducer – would get last read point in file and then find any
changes then add to a Queue - DataConsumer – any changes to the Queue
by way of size, would then cause this thread to wake and start
parsing all the items in the Queue for that log file. Deleting what
it has parsed.
I am looking for some suggestions (in light of what I have done) that may improve the current way I manage reading a file, and picking up when there are changes, and parsing the file efficiently (considering size and rotation). Code or an API will do.
Sounds like a job for Splunk
You probably don’t want to re-invent the wheel here. Splunk has a free option and can provide you a lot of extra useful functionality that you will probably need further down the road.
1