In our current setup, most of our database calls are updates where we increment a key by 1.
If all one is doing is incrementing certain keys on a document (a lot) in mongodb, and there are several application servers doing so at once, does it ever make sense to somehow aggregate these increments at the application layer and then send the pre-aggregated documents to the main mongodb database every set time interval?
I understand it could be risky because if you somehow lose one of these pre-aggregated documents you can potentially lose more data, but is this technique ever used? I also understand sharding is the accepted way to scale writes, but I was just curious if this technique is ever used because it seems like it could potentially help scale if done right, I could be completely wrong here though.
0
Sure, it is a smart way if it is viable for your product. It saves lots of calls to the database and that makes scaling easier. Things which don’t take up resources don’t have to be scaled.
Most important consideration is trust. How much data can you loose and could you detect the fact that you are losing data.
The pattern you describe looks a bit like putting a queue in front of your database to reduce impact of scaling on the database server. You could take a look at that kind of scenarios.