this.form.submit() refreshes the page, clears and ignores selection from dropdown (Selenium, Python, Docker)

Question

I’m developing a data scraper using Selenium and Python. This data scraper is supposed to be run on Digital Ocean Kubernetes pod as a Docker image.

Local scenario:

I build the Docker image.
I run the Docker image successfully.
Data scraper selects an option from dropdown.
this.form.submit() gets executed.
Page refreshes and selected option gets applied, just as expected.

Server scenario:

I build the Docker image.
I push it to the server and it starts.
Data scraper selects an option from dropdown.
this.form.submit() gets executed.
Page refreshes and the option gets cleared.

Background: Kubernetes droplet has 8gb RAM, 4CPUs, so the resources doesn’t seem to be the problem. It doesn’t seem to have anything to do with timings as well, because I have tried all the delays possible - implicit, explicit and Javascript injection to wait before form submission.

The unbeatable problem: Why does webdriver browser of the same Docker image execute this.form.submit() without any problems on my local machine, but can’t successfully execute this.form.submit() within a pod of Kubernetes?

Side note: I have noticed that whenever a cookies consent dialog appears (which happens rarely for unknown reasons to me) and the scraper accepts the cookies, this.form.submit() executes successfully and option gets selected after the page refresh.

I have been trying to solve this for almost a week now and without any success, I’m hoping to get an answer here. ANY answer is highly appreciated and I will be as responsive as possible to let you know about the outcomes.

Bobby Iliev · Answer

Hey!

Indeed, this sounds very tricky to troubleshoot. One thing that comes to my mind here is to add some extra logging to see where exactly in the code the problem might be arrising.

For example, add logging statements before and after form.submit() to ensure that the proper dropdown option is selected and visible before the form is submitted, eg:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info(f"Selected option before submission: {selected_option}")
logger.info("Submitting the form...")
form.submit()
logger.info("Form submitted.")

That way you will be able to see where the two environments are diverging.

Also, as you’ve observed that the cookies consent dialog appears occasionally, you can try explicitly accepting the consent dialog to make sure that it is not blocking the submission.

If you are still seeing the issue, feel free to share your Dockerfile and a code snippet that I can use to reproduce this on my end and try to further troubleshoot it!

Best,

Bobby

Question

this.form.submit() refreshes the page, clears and ignores selection from dropdown (Selenium, Python, Docker)

Try DigitalOcean for free

Popular Topics

Featured on Community

Get our biweekly newsletter

Hollie's Hub for Good

Become a contributor

Featured Tutorials

DigitalOcean Products

Welcome to the developer cloud