Short answer: you need a test suite for a third-party vendor API – so you will have to develop one.

Don’t expect anyone else to do it for you, and don’t expect a “magic bullet” for generating automatically the right tests.

Some things you could try additionally:

• ask the vendor if they provide a list of “breaking changes” for each new release
• ask them how they care about API compatibility/inform them that this is an important feature for you
• check if the API provides specific testing hooks, logging output or something like that for parts which could not be tested easily either
• wrap important API calls with your own logging code, writing input and related output of the API to a log file, this will make it easier to debug things if something unexpected happens
• add assertions to API calls to check pre- and postconditions, so if a new release of the API shows up unexpected behaviour within your application, you get informed early by an error message

If these things work or not depends on who is your vendor and what kind of API you have in mind. An API which produces some inspectable output like files is much easier to test than an API which controls some physical device where you have to observe the behaviour of the thing to decide whether the API call was successful or not.

Implement learning tests for your area of interest (features that you plan to use). Learning tests are integration tests that are written by the developer against the public contract of the API. The tests should not be written against the internal implementation details even if the source code for the API is available. This kind of learning tests serves two purpose –

1. It dramatically improves your understanding of the third-party API.
2. The tests help to verify whether the claimed new version is actually backward compatible or not.

There are 2 approaches for this issue…

your app is live in production with real user traffic:

if you have an app in production that has live traffic and depends on an external api you have no choice but to closely monitor and have good thresholds to know as fast as possible when the external api makes changes without notifying.

you should always take into account that :

• api’s change over time
• the api vendor can have bugs
• api vendors test-kits can have bugs or not fully cover all the production api’s functionality

your app is an installation and has planned version/releases:

in this case you have a grace period to fail… live user aren’t effected immediately of the external api breaking changes.

in my opinion this is a more easy task. write a test (full end-to-end test) that makes real transactions/http/requests to you application that invoke the external api and check that there are no failures. no test-kits no mocks real transaction.

after this task is done you can choose to run this every 24H , 1 min etc…

good practices:

• automate everything
• have a person you can quickly contact from the vendor of the external api
• dont blindly trust the vendor test everything
• fail fast – if your service depends heavily on the external api dont let your service crash. fail fast and return proper error messages

tools:

• monitoring live application : hyperic new-relic
• api testing: apimetrics runscope soapui

Based on the poster’s phrasing, it’s more than just testing, IMO. After you write your unit test for the API and make sure everything is working as expected, you need to monitor third-party APIs so you catch issues before the users do. That’s the real risk with third-party APIs – it’s not your code and you have no control over how much testing was done on the API or when/if it changes.

(Disclaimer: product names used here) If you use soapUI to write your API tests, those tests can be re-used in AlertSite as an operational monitor to make sure the API keeps working as expected. If it fails the test, you can get alerted before your users call you and complain that your app doesn’t work.