I am scraping a dynamic web which applied JS and react function on it (a blockchain explorer). I attempted to build a program which supposed to be able to scrape with JS running. However, it return me with result null for the element I am looking for. When I double check the element in source code via Chrome, the element exist with value. Why does that happened? What could I do to solve the issue?
(By the way, I tried to find the element by class name, css selector, and Xpath as well, but still, not working)
for your information, class name=tag-item address-tag info-tag tag-item-fit-content
css selector=.tag-item;
Xpath=/html/body/div[1]/div[1]/div[2]/main/div[1]/div/div[1]/div[1]/div[1]/div[2]/div[1]/div
I am looking for the address label. As you may have see there, the element exist text content USDT Token. enter image description here. API is not working for labels obtaining for some kind of address in this website, that why I am using this method.
When I scrape it, it returns me null.
const puppeteer = require('puppeteer');
const fs = require('fs').promises;
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://tronscan.org/#/address/TAzsQ9Gx8eqFNFSKbeXrbi45CuVPHzA8wr',{
waitUntil: 'networkidle0',
});
const cookies = await page.cookies()
console.log(cookies);
const f = await page.$$('.tag-item')
const text = await (await f.getProperty('textContent')).jsonValue()
console.log("label is: " + text)
const html = await page.content();
await fs.writeFile('reactstorefront.html', html);
await browser.close();
})();
Return:
[]
const text = await (await f.getProperty(‘textContent’)).jsonValue()
^
TypeError: f.getProperty is not a function
Node.js v20.15.0
After checking, the reason of this error may probably because the element is null.
I have no idea why the value is null. I wonder if the reason be the cookie or react but the code above suppose to solve both of the problem already. I could not check where is the problem now since it turns to error now and would not output an updated reactstorefront.html
KyT is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1