I was going to scrap some data from the webpage, but I cannot get what I want. So I checked if I can get the correct output at first stage.
import requests
from bs4 import BeautifulSoup
url = 'https://kream.co.kr/products/36939'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
print(soup)
Output shows tags only
import requests
from bs4 import BeautifulSoup
url = 'https://kream.co.kr/products/36939'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
product_info_div = soup.find('div', class_='product_info')
print(product_info_div)
Output is None
I would like to scrap the product number from
<div class="product_info" data-v-8b8c047e=""> MAW031L-00198-001 </div>
Why can’t I scrape that number and how I should do it?
SlowButSteady is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
2
Something happens with that page that can’t be parsed with Beautifulsoup, you can see the status of requests:
url = 'https://kream.co.kr/products/36939'
response = requests.get(url)
print(response.status_code)
print(response.text)
In this case, instead of Beautifulsoup, you can use Selenium:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://kream.co.kr/products/36939")
for element in driver.find_elements("xpath", "//div[@class='product_info']"):
print(element.text)
Use “find_element()” (specific match) or “find_elements()” (return a list of matches if there are multiple matches) to find the elements in the HTML and then, you can use “.text” to get the content of the specific tag.
You just need to adjust the “xpath” to your necessities. See the link for that.