I’ve set up a TorProxy on my Ubuntu server for web crawling due to specific network requirements. When my crawler begins to operate, it functions correctly for about 3 to 4 minutes. However, after this period, I receive a 490 response code, causing the TorProxy to malfunction and disrupting the crawling process. Before encountering the 490 response code, the crawling proceeds smoothly. Could someone assist me in understanding why I am receiving this response code and how to resolve this issue?
My TorProxy is been set up in a docker image. Crawling process would go smooth for about 4 minutes. I did not do anything, because I do not have any in mind. Actually, I’m not familiar with TorProxy configuration so well.
Aref is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.