I’m trying to learn about Web Scraping, and I’m currently using my university’s website to extract specific tables based on a code. I’m at the stage where a function acts as a “user,” sending the necessary inputs so that when the “Search” button is clicked, it returns the corresponding table.
Here is the code I’m using:
<code>import requests
# URL of the page
url = "https://www.ssa.ingenieria.unam.mx/horarios.html"
# Function to send the POST request
def get_html_response(clave):
payload = {
'optHorarioAsignatura': 'clave',
'clave': clave
}
response = requests.post(url, data=payload)
return response.text
# Example usage of the function
print(get_html_response("1858"))
</code>
<code>import requests
# URL of the page
url = "https://www.ssa.ingenieria.unam.mx/horarios.html"
# Function to send the POST request
def get_html_response(clave):
payload = {
'optHorarioAsignatura': 'clave',
'clave': clave
}
response = requests.post(url, data=payload)
return response.text
# Example usage of the function
print(get_html_response("1858"))
</code>
import requests
# URL of the page
url = "https://www.ssa.ingenieria.unam.mx/horarios.html"
# Function to send the POST request
def get_html_response(clave):
payload = {
'optHorarioAsignatura': 'clave',
'clave': clave
}
response = requests.post(url, data=payload)
return response.text
# Example usage of the function
print(get_html_response("1858"))
However, when I run this code, I always get the following response:
<code><html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
</code>
<code><html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
</code>
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
I’m not sure if there’s a solution to this or what advice you could offer to achieve this task. Is it possible that I’m sending the data incorrectly? Do I need to add any additional headers to the request? I would greatly appreciate your help.