I’m creating a notification system for particular events.
A user can set the criteria that match particular events (ie. New item, changed item, closed item etc) Items have different characteristics (ie. Minor, Major, Critical). So a user may set up a search for All New Critical Items.
What is the most efficient way to identify the ones that match the users’ saved searches and send notifications to them every time an event happens?
One way would be to run ALL the saved searches for all users, every time an event happens and see which items are in those search results and send notifications.
The problem with this method I see is that it is inefficient (saved searches could become quite a few) and the complexity of this algorithm is directly proportional to the number of searches.
Does anyone know how websites, such as property websites do it? They send out emails for new properties that match users’ searches.
I’m thinking of a way to map new events to users, given their search somehow without running all the searches.
4
We’re currently building a web service to solve problems like this (with very complex rules, including negative rules – things not happening). Unfortunately it’s in private beta right now and we’re capped on users. However, hopefully I can share the details on how to build a much simpler version (minus negation).
As you outlined as your data grows it will become very slow to run through all the saved searches in a traditional relational database. If I’ve read your question correctly your events should contain all the data about whether a notification should be fired or not. I would create a namespace like naming convention for events like ‘new-item:minor’, ‘new-item:critical’, ‘closed-item:major’. Use these namedspaced events as the keys in a key value store like redis with the value being what should happen if that event arrives.
When an item is saved, updated, destroyed etc. use an aggregator like statsd to push the namespaced event key to a log.
Then you’ll have a data collector like fluentd watching that log file and running a background worker whenever an event key is pushed in. The worker looks up the redis key and either finds nothing and does nothing or sends out the appropriate event.
That’s about it, if you change fluentd’s config file to check every second (I think the default is a minute) you can have near real-time event notifications.
First, classify your listeners. Second, classify your events. Then, match up events to listeners.
I’m going to assume that you don’t have a capability such as following a particular topic, since you didn’t mention it.
Classify your listeners by grouping them up into criteria groups. All listeners looking for “New” x “Major” events are in one group. All listeners looking for “New” x “Critical” events are in another group, with overlapping listeners as needed. You will end up with 9 groups, the cross product of action and severity.
When an event occurs, you use the cross product of action and severity to select one of the groups. All of the listeners in the group should receive a notification.
This will have the minimum number of lookups and comparisons, given that user changing their preferences is much rarer than new events happening.