I am writing a program to scrap some data from the web. The pages are sequential ( 1,2,3 … ), but I have no idea when will it stops. I combine a prefix and a integer to make a link for the python urllib to parse on it. For example : ‘http://some.domain.com/page’ + ‘1’ + ‘.htm’.
So the request would fail if the link is invalid, however, there could be some other error such as network error, connection timed out which will resolve itself. I could retry a couple of times on these error.
Aside from ‘Page not found’ error, there could be other error such as ‘Internal Server Error’ which was thrown when the server is down. There error won’t fix themselves. I should probably move on or stop the program.
Back to my program. Because I don’t know when is the page index are going to end, so I set the end integer to 9999. The links may run out after some hundreds. I should identify this and send a ‘break’ to it.
What will you do? Gather all the possible error and put them on the exception line and treat them differently? Now I just stop after 10 failures.
1
This is what HTTP status codes are for.
The server will return a 3-digit code together with a human-readable text, and you can tailor your exception handling to the 3-digit code. All HTTP-handling libraries should return that code with the exception if they throw one.
A 404 means Page Not Found, so you can abort your program at that moment, you probably reached the highest index number already.
The 50x errors indicate server failures, but a 502, 503 or 504 error can be temporary, so you could retry the same request (with a delay perhaps to let the server recover). The same goes for 408, but that one is seen only rarely and only from proxy servers.