There is an endless data stream of XML messages (and “heartbeats”), that I receive via a telnet connection and through a site-to-site VPN IPsec tunnel.

I’m still pondering. What is the best/most elegant solution to process the XML messages without losing any data, without redundance and with a (nearly) constant processing time? A never-ending process/script? Writing the stream in file(s) and processing it/them periodically step by step? Or something completely different?

The messages usually come every few seconds. Sometimes every second. Sometimes maybe every 10 seconds. It differs but not a lot. One XML message within the stream contains 45 rows. The messages should be stored afterwards.

Note: The concrete structure of the XML messages and the infrastructure of the participating systems are negligible in my opinion.