I am writing a web application that pulls small parts of articles from Wikipedia and presents them to users.
I do not wish to delete entries of the articles because I do not want to be sending a ridiculous amount of requests to the WikiMedia API, putting unnecessary load on them.
My question is, since Wikipedia is by nature, collaborative and dynamic, what would be the best way to serve reasonably updated content to users (the content does not have to reflect the current article 100%, but it shouldn’t be too far behind) without constantly querying their servers?
I’m leaning towards storing the snippets of the articles in my database along with a timestamp and whenever an user requests an article with a timestamp over x days, I would re-query the API and update the entry in my database.
The other problem with that is I am grabbing the articles via client-side Javascript, then calling an API on my server to store it. Updating an article would look like:
User queries article -> my server returns article -> client-side checking decides the timestamp is too old -> client-side re-queries WikiMedia and displays the content to the user -> client-side fires request to my server to store the data.
This seems like it could be done better, but I do not want to be querying WikiMedia directly from my server…
How could I make this better?
Architecture: SlimPHP framework, RethinkDB + PHP driver, jQuery/JS, HTML, CSS
It is not a good idea to store on your server data originating from (possibly malicious) client.
client-side fires request to my server to store the data.
Since client-side code is in hands of client, isn’t it possible to feed your server with false data?
So your caching layer should decide which articles to reload on server side based on time since last synchronization, and possibly do synch on client demand.
To minimize load on peer servers you can request only HTTP header (see HEAD on Wikipedia and Responses). This way server responses instantly with no need to generate whole page dynamically.
First thing that comes to mind is using proxy cache. Proxy should respect cache directive from the API; no programming required.
The other way to do it is implement some kind of cache yourself with your own preferred algorithm. Even if you go this way if the server load is high, you should put a proxy too as either as reverse proxy or just proxy or both.
Best way is to put a proxy and have a profile so you know where to optimise.