I’ve been attempting to scrape data from a website using Scrapy 2.11.2 along with its Scrapy/Playwright plugin (0.0.34). This is the website that I’m trying to scrape: here.

The problem is the website doesn’t seem to fully render its dynamic portions of its content. I do get an HTML response back and I do see certain JS scripts and other artifacts being downloaded in the logs. I examine the HTML response and I see element product-availability-root is empty, e.g.:

<product-availability-root></product-availability-root>

whereas when I browse to it on the browser, that element has a lot of children/content in it.

So I’ve tried using Scrapy with Playwright, in both headless and full non-headless mode, both without success in populating that element.

Here’s what I’m using in my Scrapy request:

            yield scrapy.Request(url,
                                 meta
                                     "playwright": True,
                                     "playwright_include_page": True,
                                     "playwright_launch_options": {
                                         "headless": False # I've used both True and False
                                     },
                                     "playwright_page_methods": [
                                         PageMethod("wait_for_timeout", 30000),
                                         PageMethod("evaluate", "window.scrollBy(0, document.body.scrollHeight)"),
                                         PageMethod("screenshot", path="screenshot.png", full_page=True)
                                     ],
                                     "errback": self.errback
                                 },
                                 callback=self.parse,
                                 dont_filter=True)

The screenshot comes back as a blank page. I’ve tried the same code on a different website and I know the scrolling to the end works – and the other site fully renders. So there’s something tricky about this website. I’ve put in a large timeout to give the browser plenty of time to render. I’ve also tried Playwright with Chromium and Firefox, with no difference in results.

I’ve also tried using DOM breakpoints against the product-availability-root element in Chrome trying to debug this. However, the combination of obfuscated code and a huge call stack makes it hard to figure out what’s going on.

Any hints or suggestions?