I have a chrome extension that i’m building that is doing some basic scraping of text on a Facebook page (which means 99% of DOM is just randomly generated classes that could change at any moment). I have a function that is basically a “scrape the whole page” function, and it auto scrolls to the bottom to load more data (scrolling to the bottom triggers the page to load more content) and i’m struggling with having it scroll to the bottom and then wait to see if more data is loaded (when the new data is loaded, the scroll bar is no longer at the bottom, so presumably I can perform a check to see where I’m at on the page). If more data is loaded then it should scroll to the bottom again, and do this over and over until no more data is loaded, and then resolve.
Right now it is scrolling to the bottom and not waiting long enough to check if more data has loaded, and it is resolving immediately, but I have tried a few things to this and nothing seems to do what I need.
I will be 100% honest, I am not a developer and I am learning JS at the moment, and this was cobbled together from a few different things I found, and it’s the closest I can get to what I need, although I can clearly tell that some things are out of order and some things are clearly just wrong. I could use some help fixing this.
function startScrollAndScrape() {
window.scrollTo(0, document.body.scrollHeight);
return new Promise(async (resolve, reject) => {
await holdASec();
var i = setInterval(async () => {
var diff = document.body.scrollHeight - (window.scrollY + window.innerHeight);
console.log(diff);
if(!document.querySelector('[aria-label="User content"] [role="progressbar"]') && Math.floor(diff) > 2 ) {
window.scrollTo(0, document.body.scrollHeight);
await holdASec();
} else if(!document.querySelector('[aria-label="User content"] [role="progressbar"]') && Math.floor(diff) < 2) {
clearInterval(i)
window.scrollTo(0, 0);
resolve()
}
},100)
})
}