When scraping code it results in following error:
https://www.amazon.com:443 "GET /s?k=Chicken&i=wholefoods&disableAutoscoping=true HTTP/1.1" 200 None
urls = {
'Whole Foods Market': f'https://www.amazon.com/s?k={food}&i=wholefoods&disableAutoscoping=true',
}
……
url = urls.get(shop)
items = []
if not url:
app.logger.error(f"No URL found for shop: {shop}")
return []
try:
response = requests.get(url, headers=HEADERS)
response.raise_for_status()
soup = BeautifulSoup(response.content, "html.parser")
if shop == 'Target':
# Implement Target scraping logic
pass
elif shop == 'Whole Foods Market':
links = soup.find_all("a", attrs={'class': 'a-link-normal s-no-outline'})
if links:
product = 'https://www.amazon.com' + links[0].get('href')
new_webpage = requests.get(product, headers=HEADERS)
new_soup = BeautifulSoup(new_webpage.content, "html.parser")
title = new_soup.find("span", attrs={'id': 'productTitle'}).text.strip()
price = new_soup.find("span", attrs={'class': 'a-offscreen'}).text.strip()
image = new_soup.find("img", attrs={'id': 'landingImage'}).get('src')
items.append({'name': title, 'price': price, 'image': image})
New contributor
user812271 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
4
Try to set Accept-Language HTTP header to select your region, e.g.:
import requests
from bs4 import BeautifulSoup
headers = {
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0",
"Accept-Language": "en-US,en;q=0.5",
}
url = "https://www.amazon.com/s?k=Chicken&i=wholefoods"
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")
for product in soup.select("[data-asin]:has(img)"):
print(product.img["alt"])
print(product["data-asin"])
print()
Prints:
Sweet Sue Chunk White Chicken in Water, 5 oz Can (Pack of 1) - 11g Protein per Serving - Gluten Free, Keto Friendly - Grea...
B00OMUEAVO
Beware of Chicken: A Xianxia Cultivation Novel: Book 1
B09Y2H2MKP
Bestylez Welcome To Our Coop Funny Chicken Sign Chicken Coop Sign Chicken Decor 12" * 8" (215)
B0BP2TL9QS
Swanson White Premium Chunk Canned Chicken Breast in Water, Fully Cooked Chicken, 12.5 OZ Can (Pack of 6)
B0CVS67HCD
McCormick Grill Mates Nashville Hot Chicken Seasoning, 3 oz
B0BSMJHG93
Campbell's Condensed Gluten Free Cream of Chicken Soup, 10.5 oz Can
B0BYQG86MQ
...