I am doing a tutorial on scrapy scraping and want to scrap the data from a website. I created the scrapy project and opened the scrapy shell to test it. However, the requests gets redirected and gets status 301 and this causes an AttributeError: ‘Decompressor’ object has no attribute ‘process’.
My input to scrapy shell:
r = scrapy.Request(‘http://worldometers.info/world-population/population-by-country’)
fetch(r)
Output
2024-05-01 01:26:38 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.worldometers.info/world-population/population-by-country/> from <GET https://www.worldometers.info/world-population/population-by-country>
AttributeError: ‘Decompressor’ object has no attribute ‘process’
The request gets redirected and this leads to an AttributeError. Have anyone encountered this issue? Thanks in advance!
vju_8 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.