I’m working on my first AWS Lambda based application and I ran into a question I can’t find the answer for anywhere.
According to the best practices on functions, you should set up database connections outside the function handler so they may be reused. What happens if the connection becomes unhealthy in some unrecoverable way? There is a long list of things that can happen to cause this from the connection failing, database location changing, or even just bugs in the driver itself. It doesn’t matter what the actual cause is. There is a class of problems that are self-healing if this function is torn down and recreated, reestablishing the database connection.
What does lambda offer to deal with this? Does AWS do any kind of anomaly detection on results and pull unhealthy functions out of rotation? If so are there docs on this and what it looks for? Is there an API for reporting errors from inside the function that can say “not only did I fail, but I may be in an unhealthy state and shouldn’t get anymore traffic”?
I’ve been looking through docs and have found several places where they reference setting up this state shred state and holding the connection open, but I haven’t found anything about how to handle when it fails.