SeleniumBase Driver Initialization Fails with Gunicorn + Nginx, but Works Manually with Flask

Description:

I am experiencing an issue where the SeleniumBase Driver initialization works perfectly when the code is executed manually using python3, or when Flask is run manually. However, when the Flask app is integrated with Gunicorn and Nginx for automatic startup, the driver initialization fails with the following error:

Error loading driver: Message: session not created: cannot connect to chrome at 127.0.0.1:9222

from chrome not reachable

The issue arises when trying to initialize the Driver or SB (SeleniumBase) with undetectable Chrome in headless mode.

Steps to Reproduce:

1. Create a Python script (test.py):

   
   from seleniumbase import Driver
   
   import logging
   
   logger = logging.getLogger(__name__)
   
   def main():
   
       try:
   
           driver = Driver(uc=True, headless=True, no_sandbox=True, block_images=True)
   
           logger.info('Driver defined')
   
           driver.get("https://www.zillow.com/profile/APT212Marketing")
   
           logger.info('Driver opened')
   
           title = driver.title
   
           driver.quit()
   
           return {"title": title}
   
       except Exception as e:
   
           return {"error": str(e)}
   
   if __name__ == "__main__":
   
       print(main())
   
   

2. Run the script manually:

   • Execute the script with python3 test.py.

   • The script runs as expected and fetches the webpage title.

3. Add the same functionality in a Flask app:

   
   from flask import Flask, request
   
   from seleniumbase import SB
   
   import logging
   
   logger = logging.getLogger(__name__)
   
   app = Flask(__name__)
   
   @app.route("/fetch", methods=["POST"])
   
   def fetch_title():
   
       data = request.json
   
       review_page_url = data.get('query')
   
       try:
   
           logger.info("Loading driver...")
   
           try:
   
               with SB(uc=True, xvfb=True, headless2=True) as sb:
   
                   logger.info("Display started successfully.")
   
                   sb.driver.get(review_page_url)
   
                   logger.info("Driver initialized successfully.")
   
                   title = sb.driver.title
   
           except Exception as e:
   
               logger.error(f"Error loading driver: {e}")
   
               return {"error": str(e)}, 500
   
           return {"title": title}
   
       except Exception as e:
   
           app.logger.error(f"Error loading driver: {e}")
   
           return {"error": str(e)}, 500
   
   if __name__ == "__main__":
   
       app.run()
   
   

4. Run Flask manually:

   • Start the Flask app with python3 app.py.

   • Send a POST request using curl or Postman to the /fetch endpoint.

   • The driver loads successfully, and the page title is returned.

5. Configure Flask with Gunicorn and Nginx:

   • Set up Gunicorn and Nginx to start the Flask app automatically (as per typical production setups).

   • Start the server and send a POST request to the /fetch endpoint.

6. Error Observed:

   • The request fails with the error:

     
     Error loading driver: Message: session not created: cannot connect to chrome at 127.0.0.1:9222
     
     from chrome not reachable
     
     

Observations:

• The driver works fine when the script is run manually or when Flask is started manually.

• The issue occurs only when Flask is served with Gunicorn and Nginx.

Environment Details:

• SeleniumBase Version: 4.33.12

• Flask Version: 3.1.0

• Gunicorn Version: 23.0.0

• Ubuntu Version: 22.04.4 LTS (Jammy Jellyfish)

• Nginx Version: 1.18.0

• Python Version: 3.10.12

What I Have Tried:

1. Verified that the environment variables are correctly set in the gunicorn.service file, including:

   • PATH

   • Chrome driver binary path

2. Added a virtual display (xvfb) for headless mode.

3. Used the uc=True parameter with SeleniumBase SB and Driver.

4. Increased the timeout duration.

5. Switched the Gunicorn worker class to gthread.

6. Verified that all required dependencies are installed.

7. Checked system resources (sufficient memory and CPU available).

8. Tested the code outside Gunicorn (both manually and with Flask), and it worked as expected.

9. I have used selenium for some websites which works as expected

Additional Context:

The issue seems related to Gunicorn's process management or Nginx's proxy setup interfering with the uc=True argument or the headless Chrome browser session. Further guidance on resolving this issue is appreciated.

Let me know if you need more details to write the issue.