I have faced a problem. Whenever I used User Agent in my Python Code, it shows abnormal activity. Please see the code bellow –
#import necessary Library
import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
# Load the webpage content
HEADERS = ({'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36'})
url = 'https://www.alibaba.com/trade/search?spm=a2700.7735675.the-new-header_fy23_pc_search_bar.recommendItem_pos_1&tab=all&SearchText=panjabi'
page = requests.get(url, headers=HEADERS)
print(page.status_code, page.request.headers)
# Convert to a Beautiful Soup Object
soup = BeautifulSoup(page.content, 'html.parser')
print(soup.title.text)
Output:
200 {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
Mobile Phones-Mobile Phones Manufacturers, Suppliers and Exporters on Alibaba.com5G smartphone
But When I Used different url which are given bellow, it show as
Output:
503 {‘User-Agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36’, ‘Accept-Encoding’: ‘gzip, deflate’, ‘Accept’: ‘/‘, ‘Connection’: ‘keep-alive’}
Sorry! Something went wrong!
Some different urls these are not response:
https://www.startech.com.bd/laptop-notebook/laptop
https://www.amazon.com/s?k=ear+buds&crid=3DWKKDPQ687S4&sprefix=%2Caps%2C393&ref=nb_sb_ss_recent_2_0_recent
https://www.amazon.com/s?k=office+calculators&i=office-electronics&crid=2EBQDTXKUNIYC&sprefix=%2Coffice-electronics%2C344&ref=nb_sb_ss_recent_3_0_recent
https://buyee.jp/ebay/search?category=293
Finally, My question is several url why does not response? What is the Solution?
Md Masuduzzaman is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.