Here is the problem I’m trying to solve. A user needs to enter search criteria to identify individuals. I’m using Selenium to auto the workflow that will do the following:
- Ask the user to enter the criteria (name, city, state)
- The app will then pull up the website, enter the user inputs into the respective data fields and then click send.
- The app will then scrape the resulting web page containing the table with the desired data.
I’ve been successful with tasks 1 and 2. However, I am having difficulties scraping the resulting webpage with the table.
I noticed when a user performs a search, it doesn’t load the results dynamically in the same webpage. Instead, it pulls up a completely new webpage that contains the text (i.e., table) that I need to scraped. When I use the url with the results, I get redirected back to the search query page.
Can anyone give me some guidance on this?
Here’s the code that I have so far:
enter image description here
Jerald Bouie is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.