I’ve set up an Apache NiFi process to send CentOS logs to ElasticSearch.

I used ListenSyslog to receive system logs from the CentOS server.

Now unfortunately, the silly thing then strips out the timestamp and implant’s its own. The timestamp that WAS there was ISO6801 (Centos Syslog Timestamp(2024-05-23T20:29:27Z)) but then it inserts the silly timestamp that looks like this:
“May 23 09:25:59”

So, what I tried, was a lot of things, but most recently a flow like this:
ListenSyslog > ExtractText > UpdateAttribute > ReplaceText > ConvertRecord > PutElasticsearchJson

Once the log is ingested:
ExtractText
To get the timestamp (I don’t want to generate a new one, just in case there’s a delay in ingesting logs, rather keep the timestamp from the log itself.
UpdateAttribute
This would append the year to the timestamp
ReplaceText
This would format the date correctly
ConvertRecord
Translated the log to a Json object
PutElasticsearchJson
Actually send the log to Elasticsearch

But the entire thing fails because of the timestamps.

Now, I CAN get it to send the logs, but then the times are treated like keywords, and you can’t use them to specify a time range, you can only use them as a filter (How many events occurred at X time) or display the 5 times (up to the second) where there are the most logs (and then any other time is filtered into the massive pile know as “other”.

Some Specifics:
ExtratText
I have a custom field called “orig_timestamp” with value ‘(w{3,} d{2} d{2}:d{2}:d{2})’

UpdateAttribute
I have a custom field called “orig_timestamp_with_year” with ${orig_timestamp:toDate("yyyy MMM dd HH:mm:ss"):format("yyyy-MM-dd'T'HH:mm:ss.SSSZ")}

ReplaceText
Replacement Strategy: Regex Replace
Search Value: w{3,} d{2} d{2}:d{2}:d{2}
Replacement Value: ${'$1 '}${now():format('yyyy')}

ConvertRecord
Record Reader: SyslogReader
Record Writer: JsonRecordSetWriter

Schema Write Strategy: Do Not Write Schema
Date Format: yyyy-MM-dd
Time Format: HH:mm:ss

PutElasticsearchJson
Index Operation: index
Client Service: PutElasticsearchJson

What the flow looks like