I’m building an Apify scraper to target transaction data on a dynamic webpage. The table containing these transactions loads asynchronously via AJAX, taking under 30 seconds.
This is the page when loading:
This is the fully loaded page:
This is my code:
// Step 4: Scrape data from the table
log.info('Finding selector accountDiv...');
await page.waitForSelector('#accountDiv', { timeout: 60000 });
The #accountDiv
selector consistently fails to be found with a waitForSelector
timeout of 60 seconds. I suspect the issue lies in identifying the correct element to wait for before scraping the transaction table.
What strategies can I employ to pinpoint the optimal element to trigger the wait logic, ensuring the scraper captures the fully loaded transaction data?
I tried mulitple selector ids, without success.
avall is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1