suppose im writing a dating website, similar to okcupid. there are profiles, and i need to compute the (N^2) “match” table – given every 2 profiles whats the match between them?
I was thinking this could be done by creating a spout to listen on a “new/updated profiles” queue (say kafka, doesnt matter), but then how do I break down the matching to achieve any degree of parallelism?
If I have a single bolt that compares the profile vs the entire DB that wont scale.
If I create another spout, for “all profiles” it will run in a continous loop and never stop (?)
Obviously the assumption is that the “churn rate” (rate of new/updated profiles) is less of an issue than the sheer size of the database.
Any suggestions on how to design the topology would be very welcome.
4
The obvious answer to this problem is “don’t do N^2”, at least not for the full value of N – you can very rapidly reject large parts of your data set for all sorts of reasons:
- Wrong sex. While I’m a liberal kind of guy, I’m still not interested in dating other guys. There may obviously be some exceptions to this – e.g. bisexuals.
- Wrong country. It doesn’t matter how hot they are, I’m not going to want to date someone in Australia if I’m based in the UK. Again, there may be exceptions but most people aren’t looking for a long-distance relationship. (More generally, get people to say how far they’re prepared to travel for a relationship. New York to California probably isn’t going to work for most people).
There’s probably lots more criteria I haven’t thought of yet – by doing this you’ve hopefuly solved the “sheer size of the database” problem and got it down to something tractable.
As an aside, I suspect you’ll find Data Science of Love interesting viewing. Disclaimer: I know the presenter personally, but have no professional relationship with either him or eHarmony.
1
Although I have not worked in this area, it seems like this is the type of problem one would use something like Apache Giraph to handle. You essentially want a huge population of “things” which you suspect are “related” to each other possibly across a large number of dimensions, and you want to pair them up optimally based on the strongest matches.
You don’t want to hard code what those things are, the possible dimensions they may be related, and the rules that define each relationship. Not only would that be a lot of work, it wouldn’t scale well.
This link is a proprietary graph product, but does a good job showing how a graph data model would apply to online dating. Same concept could be done with Apache Giraph, or any other graph computational tool:
https://linkurio.us/recommendation-and-graphs-an-online-dating-use-case/
0