I’m making an Rails app that in theory should scrub new posts from the users facebook and twitter accounts and put them in a timeline for an analyst to analyze and determine if they are good or bad posts, if the user posted “bad content” the analyst marks it as so and warns the user.
I’ve made the authentication with omniauth and have made both facebook and twitter apps, users can already sign in and in return I get a oauth key and secret, with these I can make the respective API calls.
Now for the real question, how am I supposed to check for new updates?
I’ve thought using resque or sidekiq to queue each user’s facebook and twitter accounts periodically, but wouldn’t it be too much if I had many users, making a new API client (I’m thinking Koala and Twitter gems) with each user’s key and secret every certain time?
What’s the best way to do this?
And once I got the users’ posts should I store them in the DB or just put them in another resque or sidekiq queue for them to be analyzed and discarded once they are?
There’s actually no other way to get this info than fetching the data.
You would have to dive into each API to look for ways to minimize your work. Some could accept ETag, or “Last-Modified” so your request ends quickly if nothing is there 😉
Facebook for example allows you to ask for few resources at once (https://developers.facebook.com/docs/graph-api/using-graph-api/v2.5#multirequests)
Depending on your app, to minimize requests number you could
-
analyze statistically how different users post, and modify priorities
accordingly (depending on their location most of them will not tweet
when they sleep – some will, when using solutions like bufferapp) -
fetch data when user enters your page. You can also call some of those api’s on client side and then push information back to your system to analyze them.