Tried to use scrapy-splash, but at the moment of checking the image in my localhost:8050 interface (the port is being used with docker of the image scrapinghub/splash), every website i try is not catching the dynamic objects. As an example: i tried one webside that is meant to practice and it is not working this one, getting me this output: .
Have reiniciated the image in docker, removing it and pulling it again and again, i have had the same result, and i tried to run it with : --disable-private-mode and it doesn’t work.
Hence, i just ignore it and tried to use it in the scrapy shell, believing it was just an error of the integrated interface of the image, but at fetching the url and trying to get the text over it looks like this:
scrapy shell
>>>fetch("http://localhost:8050/render.html?url=https://www.scrapingcourse.com/javascript-rendering")
2024-09-18 09:27:13 [scrapy.core.engine] INFO: Spider opened
2024-09-18 09:27:14 [pymongo.connection] DEBUG: {"clientId": {"$oid": "66eae342332417bb5ecd7225"}, "message": "Connection pool created", "serverHost": "localhost", "serverPort": 27017}
2024-09-18 09:27:14 [pymongo.connection] DEBUG: {"clientId": {"$oid": "66eae342332417bb5ecd7225"}, "message": "Connection pool ready", "serverHost": "localhost", "serverPort": 27017}
2024-09-18 09:27:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://localhost:8050/render.html?url=https://www.scrapingcourse.com/javascript-rendering> (referer: None)
>>>response.css("div.product-grid::text").getall()
['n ', 'n ', 'n ', 'n ', 'n
', 'n ', 'n ', 'n ', 'n ', 'n ', 'n ', 'n ', 'n ']
>>>>>> response.css('::text').getall()
['n ', 'n ', 'n ', 'n ', 'JS Rendering Challenge to Learn Web Scraping - ScrapingCourse.com', 'nn ', 'n ', 'n ', 'n ', 'n ', 'n ', "n window.dataLayer = window.dataLayer || [];n function gtag(){dataLayer.push(arguments);}n gtag('js', new Date());n gtag('config', 'G-NZGD14H87G');n ", 'n ', 'n', 'n', 'n ', 'n ', 'n ', 'n ', 'n ', 'n ', 'n ', 'Scraping', 'n ', 'Course', 'n ', 'n ', 'n ', 'n ', 'nn ', 'n ', 'n ', 'nn ', 'n JS Renderingn', 'nn ', 'n ', 'n ', 'n ', 'Challenge', 'n ', 'n ', 'n Enable JavaScript to see productsn ', 'n', 'nn ', 'n ', 'n ', 'n ', 'n
I encountered all the other questions about this same error…
1.This one
made the scrolling script; changed the script of lua, it is not rendering too…
1