I’m developing a data scraper using Selenium and Python. This data scraper is supposed to be run on Digital Ocean Kubernetes pod as a Docker image.
Local scenario:
- I build the Docker image.
- I run the Docker image successfully.
- Data scraper selects an option from dropdown.
- this.form.submit() gets executed.
- Page refreshes and selected option gets applied, just as expected.
Server scenario:
- I build the Docker image.
- I push it to the server and it starts.
- Data scraper selects an option from dropdown.
- this.form.submit() gets executed.
- Page refreshes and the option gets cleared.
Background:
Kubernetes droplet has 8gb RAM, 4CPUs, so the resources doesn’t seem to be the problem.
It doesn’t seem to have anything to do with timings as well, because I have tried all the delays possible – implicit, explicit and Javascript injection to wait before form submission.
The unbeatable problem:
Why does the same Docker image runs without any problems on my local machine, but can’t successfully execute this.form.submit() within a pod of Kubernetes?
Side note:
I have noticed that whenever a cookies consent dialog appears (which happens rarely for unknown reasons to me) and the scraper accepts the cookies, this.form.submit() executes successfully and option gets selected after the page refresh.
I have been trying to solve this for almost a week now and without any success, I’m hoping to get an answer here. ANY answer is highly appreciated and I will be as responsive as possible to let you know about the outcomes.