I need to scrap/parse comments from Debank site’s posts.
For example: https://debank.com/stream/2057406
The problem is that if I scroll the site the html changes and rendering comments in real time.
Every comment has it’s id, and when I scroll the site the new comments adds and the previous one disappears.
Dynamic code
For example first comment has attribute data-index="0"
, second has data-index="1"
and it works like this:
Page loaded
A little scroll to look at the comment section
We see comments and in f12 mode their ID’s
data-index="0" data-index="1" data-index="2" data-index="3"
We scroll more
data-index="1" data-index="2" data-index="3" data-index="4"
We scroll more
data-index="2" data-index="3" data-index="4" data-index="5"
And so on..
I tried using requests, selenium and even downloading source code and trying parsing it on my computer, but all I get is a first few comments, but I need them all.
user24899514 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.