i’m trying to use puppeteer to scrape a livejournal community i mod, but it’s having issues with the read more cut links.

this is the section of the script:

    // Function to click all "read more" links with enhanced error handling
    const clickAllReadMoreLinks = async () => {
        let hasMoreLinks = true;
        while (hasMoreLinks) {
            const readMoreLinks = await page.$$('a.ljcut-link-expand');
            if (readMoreLinks.length === 0) {
                hasMoreLinks = false;
                continue;
            }
            for (let link of readMoreLinks) {
                try {
                    // Ensure link is visible and interactable
                    await page.evaluate(el => el.scrollIntoView({ behavior: 'smooth', block: 'center' }), link);

                    // Wait for the link to be clickable
                    await page.waitForFunction(
                        el => el && el.getBoundingClientRect().top >= 0 && el.getBoundingClientRect().bottom <= window.innerHeight && !el.disabled,
                        { timeout: 15000 },
                        link
                    );

                    // Click the link using page.evaluate to handle any issues with standard Puppeteer click
                    await page.evaluate(el => el.click(), link);

                    // Wait for content to expand
                    await page.waitForFunction(
                        () => !document.querySelector('a.ljcut-link-expand'),
                        { timeout: 30000 }
                    );

                } catch (error) {
                    console.error('Error clicking "read more" link:', error);
                    // Attempt to recover by reloading the page or retrying if needed
                    await page.reload({ waitUntil: 'networkidle2' });
                }
            }
        }
    };

i keep getting a ‘Error clicking “read more” link: timeout exceeded’ error.

any ideas? i’ve tried increasing the timeout time but no luck.
thanks in advance!

i’ve tried extending the time out time and i’ve tried adding more parameters in to check there’s new content before scraping etc. but still the same error.