Getting a scenario where a few Java SpringBoot-based applications are running upon AWS EC2 Instances and producing logs in two ways: log files and console output. My question is how to manage these logs to get them centralized and allow the querying and monitoring.
As I am not experienced with this job I’d like to know which are the best cost-effective and easy maintenance ways to support the logs infrastructure.
The options that came to mind would be to store the logs onto the local filesystem and send them periodically to an S3-like storage and then use a Splunk-like server to get the logs source from that, however, it would cause a delay in getting the logs available. In this way, I’d have plenty of doubts as:
- How to control the log file’s lifecycle? Since the file can grow up indefinitely incurring in storage costs.
- How to aggregate the files in the Storage?
- How to get a real-time ready log to be queried?
The dropped questions aim to get an idea of the doubts I’ve in mind, of course, an article walking through these concepts would be helpful as an answer.