Given a web application running across 10+ servers, what techniques have you put in place for doing things like altering the state of your website so that you can implement certain features.
For instance, you might want to:
- Restrict Logins/Disable Certain Features
- Turn Site to “Read Only”
- Turn Site to Single “Maintenance Mode” page.
Doing any of the above is pretty trivial. You can throw a particular “flag” in an .ini file, or add a row/value to a site_options table in your database and just read that value and do the appropriate thing.
But these solutions have their problems.
Disadvantages/Problems
For instance, if you use a file for your application, and you want to switch off a certain feature temporarily, then you need to update this file on all servers. So then you might want to look at running something like ZooKeeper, but you are probably overcomplicating things.
So then, you might decide that you want to store these “feature” flags in a database. But then you are obviously adding unncessary queries to each page request. So you think to yourself, that you will throw memcached in to the mix and just cache the query. Then you just retrieve all of your “Features” from memcached and add a 2ms~ latency to your application on every page. So to get around this, you decide to use a two tier-cache system, whereby you use an inmemory cache on each machine, (like Apc/Redis etc). This would work, but then it gets complicated, because you would have to set the key/hash life to perhaps 60 seconds, so that when you purge/invalidate the memcached object storing your “Features” result, your on machine cache is prompt enough to get the the new states.
What suggestions might you have? Keeping in mind that optimization/efficiency is the priority here.
1
If you deal with more than a handful of servers, you need some sort of configuration management/automation, even if it is a glorified multiplexed ssh client.
Once you have that in place, most systems have a “apply this on this set of servers now” operation, which you can use for instance to tweak your web server configuration and restart it (to enable a single-page maintenance site and disable your regular site) or change your website configuration.
There can be many answers here for the way you state the question, all correct that could potentially produce the same result.
- Configuration management as stated in another answer.
- Switch board pattern. You can have a database table for configuration with “server name”, “configuration” and “value” fields. Before each operation a check could ensure that the operation is permitted. Results can be cached locally for performance.
- Continuous Deployment: switching off things fast from source control/CI system.
Generally it would be easier to suggest the optimal solution if the rationale for doing so was known.
Instead of fetching config data from the DB upon every request, store it in memory (in c# use a static variable or the cache object or something) – then when you need to update a config value, update it in the DB. Then have a protected URL on each server that you can hit that forces the app to reload the config data. Then you could even write a front-end for updating server config data that also hits those URLs.
Or if you’re in ASP.NET land you could just use the cache object and check out SqlCacheDependency.