I have a job that runs at a fairly high frequency (~1x every second).
This job needs to be executed constantly as long as the web app is running. However, I really only want 1 instance of the job running at a time.
Currently, I just fire up a thread that performs the job in the background, sleeping for ~800ms each loop. That causes a problem when I want redundancy, because the thread starts up as part of the app.
I know I can do something simple, like add a table in a database and have a “primary” and “secondary” server lock the record, track who’s in charge, etc. But it seems a bit ugly, so I was curious if anyone had experience with a similar situation, and what solution you ended up using?
2
I see this ultimately as a problem of interprocess communication (IPC). You have one process generating work and then another one consuming the tasks and doing the work.
This falls into a classic producer and consumer layout (tutorial from RabittMQ).
The nice thing about message queues is that they guarantee the contract of “this message is guaranteed to be delivered to one, and only one consumer (if it is consumed at all).” This simplifies the IPC as you don’t need to worry about the semaphores and shared memory segments or other channels of communication.
Beyond this, you can start to get fancy. You can have two consumers running – though only one will grab the message indicating the task from the queue. You could have the producer pumping out single “consider this” tasks as fast as it can (up to a certain queue fullness) and the consumers processing them once every 800ms – though the exact implementation depends on the problem.
The message queue has an advantage over databases in that its what its designed to do. You can do similar things via mechanisms in the database, but then you will start worrying about transactions and locks. Some databases have gone on to take additional abilities of message queueing as part of the core set of features as described in this post, but a message queue has a general advantage over databases being able to handle more concurrent messages, blocking until you get a message, avoiding deadlocks and races.
How one designs a solution using a message queue for this specific problem is still a bit up in the air as there are many aspects of the domain that are left undescribed, but in general the message queue should be one of the first tools that comes to mind when one starts thinking about how to send messages between one or more processes.
1
I can recommend the Hangfire as a solution. Key highlights for your case (extracted from the web site),
Persistent
Background jobs are saved into a persistent storage – SQL Server, Redis, PostgreSQL, MongoDB and others. You can safely restart your application and use Hangfire with ASP.NET without worrying about application pool recycles.
Reliable
Once a background job created and placed into a storage without exceptions, Hangfire takes the responsibility to process it at least once. You are free to throw unhandled exceptions or kill your application – background jobs will be re-tried automatically.
Distributed
Background method calls and their arguments are serialized and may overcome the process boundaries. You can use Hangfire on different machines to get more processing power with no configuration – synchronization is performed automatically.
2
Good Question,
I would move the task out to a seperate service. This gets around your link with the website but you’ll still have to make the service redundant. You can go two ways with this.
1: services dont know about each other. If you can change the job so that no error occurs when it runs concurrently with itself this is the best solution as you can simply fire up 2 or more services and not worry.
2 the services know about each other. This is more complicated but worth it if the job is long running as it allows you to spread the load over workers. You can use a queue (msmq,zmq,rabbit) or a database table (mark jobs as in progress) to affect the communication
1
The task should be running as a separate ‘service’ which starts when the server starts. You can use a singleton pattern to ensure there is never more than once instance running. For a .net specific solution look at Mutex. It creates a system wide singleton pattern.
While you say the process should only be running when the web app is what is really important is that the process completes some periodic work when the web app is running. It can accomplish this a number of different ways. One would be to have the service listen to see if the web app is running between sleep cycles.
The one thing you don’t have is the ability to detect if the service fails to run and restart it. There are ways to accomplish that as well plus throw in alerting but if you are going to all that you should look into making your ‘service’ an actual windows service.
5