I’m pretty new to k8s. I can see there’s lots of options available for configuring the sending of email alerts for k8s cron jobs. Just need some advice on which ones are out there and what I might want to invest time learning to sort out my needs. I’m a bit overwhelmed with choice and don’t have enough time to learn everything.

I have about 5 types of cron jobs. They run python scripts. I’d like to be able to dispatch email alerts to various parties when errors occur in these jobs.

• the jobs log to stdout – id prefer to keep it that way – leave the jobs agnostic to whatever is parsing their logs.
• I can change the log format to contain whatever format (json etc) and to contain whatever info I need.

Requirements:

• the system should be able to assess when a job instance has completed and analyse the logs for that job run on its own. (That way each alert can say things like “the downloader job failed, here are the downloads relevant to you that did not succeed” etc)
• The system must be able to look in the log messages themselves – they contain info on who the alerts should be sent to.

I’ve seen a few options – I’m not sure which ones are appropriate.

• prometheus: but this seems like it is more geared towards sending alerts it jobs fail outright, rather than being able to conditionally alert specific people if the log contents contain particular info.
• Fluentd (I don’t know enough about this to know if it’s capable of doing what I need)
• Elasticsearch + other stuff (worried it might be overkill)
• Open to others!

Criteria:

• I can install anything on my cluster to do this. Ideally
• something relatively popular – eventually I’ll need to train support teams to use the solution – would like it to be something well documented online.
• Simple installation config is preferred within reason.