I’m designing a system that is built on several small applications written in python, some of these will be services and others will be programs that only run during special situations. What I need to achieve is to let the server communicate to a service when a certain event occurs with some small amount of data. So that the service knows about the new situation and can take the appropriate action, sometimes immediately and sometimes in the future.
I’ve been looking at memcached, but it seems like an overkill solution for my system (which is just a few separate applications running on a raspberry pi), or writing to files and using something like pyinotify but I don’t think that is very elegant and finally I’ve been thinking about storing state in a database which I also don’t like very much.
Are there any preferred ways to handle this sort of communication between processes that I haven’t thought of(I’m sure there is) or which of the above mentioned ones would be the smartest to go with?
2
To my knowledge, the only standard for specific technologies is “whatever works for your needs.”
That said, you might want to look into Redis. I’ve used it before to communicate between disparate systems, and it works well.
Basically, what it is is an in-memory datastore. You set the service to watch for a specific key, and when your system does something with that watched key, the service reacts.
In a more general sense, what you’re asking for is often known as a “pub-sub” (publisher-subscriber) setup. So if Redis doesn’t work for you, you might be able to use the more general term to find a solution that does work. And really, if you wanted to, you could leverage Twitter to be the intermediary, if you really wanted to (the system would post a tweet, and the service would watch the twitter feed for updates). If I recall correctly, GitHub uses Hubot and Campfire in a similar manner (Hubot sits in a Campfire chat room, waiting for someone to send commands, to which Hubot responds).
1
Standard input and standard output seems to be the simple overlooked answer (or maybe I’m not understanding the problem). For anyone reading this now, stdin and stdout is the simplest way for two applications to communicate.
The main application can spawn a different process and send messages through stdin and stdout. It’s simple, direct, and fast. However, you have to come up with a protocol for the two processes to communicate. If the messages are simple, coming up with this ‘protocol’ is trivial.
I’m no python expert but I know you can create subprocesses and redirect the standard input and standard output of the subprocess to your main application.
https://docs.python.org/2/library/subprocess.html
There is no need to use an external service, you can use sockets.
At least if you have one-to-one communications, if there are many listeners for certain events, a pub-sub solution, like @Shauna recommended, might be better.
for an overview on sockets and their use in python see the docs