I’m currently working on a service that is supposed to aggregate data from a number of APIs, unify the data, and offer it through another API to the users. I had a couple of ideas for solving this, but I’m not confident my best solution is the best possible solution.
Solutions I considered:
- Let the browser poll our API through javascript, which in turn polls the third party APIs. This introduces quite a bit of lag between the request and the user seeing the data, but doesn’t waste resources on users that rarely log in.
- Let the browser poll the third party APIs and afterwards send the data to our API. This is potentially insecure, and requires me to provide extra API endpoints and a lot more data cleanup.
- The option I currently consider the most optimal is letting a service on the server (an actual executable) periodically poll the third party APIs. I would, for example, poll some endpoints every 30 minutes, and others maybe once a day. This is the safest option, and introduces little lag to user requests, and prevents users from reaching the request limit of the API. But data will only be up-to-date right after polling.
Are there any alternatives? If not, is my current optimal solution good enough?
1
Another option is not to poll at all, but to implement a caching solution where, when a request comes in to your API, you use a cached value if you have one that’s recent enough, otherwise you update your cache with a call to the external API.
If you have control over any of the apis, and for your own api if you are expecting the browser to display “live” updates, you may want to consider implementing “long polling”, also known as “comet”. In long polling, whena request is made, the server checks to see if anything has changed since the client last polled (it can be helpful if the client includes a time stamp from the response to its last poll for working this out) and if so returns the changed data immediately. Otherwise, it waits until data does change before returning. Therefore, the client only needs to make a single request and wait for the data to come back, rather than n having to continuously poll. This reduces load on the server and makes clients respond faster.
1