1

I'm working on a web scraping script using SeleniumBase on an Ubuntu virtual machine (VM), and I'm running into issues with headless mode and xvfb. This script works perfectly on my local Windows machine, but when I run it on Ubuntu, it gets stuck during execution, specifically at the point where it should interact with elements on the Servipag site.

Here’s the code that works locally on Windows but fails on the Ubuntu VM:

def scrape_servipag_service_reading(service_type, company, identifier, companyType=None, propertyId=None):
max_retries = 5
result = None

for attempt in range(max_retries):
    with SB(uc=True, xvfb=True) as sb:
        try:
            user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
            sb.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": user_agent})
            sb.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
            sb.set_window_size(1920, 1000)

            url = f"https://portal.servipag.com/paymentexpress/category/{service_type}"
            sb.uc_open_with_reconnect(url, 4)
            sb.sleep(2)
            # Additional code to handle interactions with Servipag

        except Exception as e:
            logger.error(f"Attempt {attempt + 1}/{max_retries} failed with error: {e}")
        finally:
            if result is not None:
                break
return result

What I’ve tried in the VM:

Running in headless mode (the driver doesn’t run). Using xvfb=True as part of the SeleniumBase initialization. Verifying that it’s not an element locator issue, as the same code works flawlessly on Windows. On Ubuntu, it seems the driver fails to launch correctly or gets stuck when opening the URL. I'm using the uc=True (undetected Chrome) setting and specifying a user_agent to avoid detection, but no luck so far.

Environment:

OS: Ubuntu (in a virtual machine)

Chrome version : 129.0.6668.101 (Tried both headless and with xvfb)

SeleniumBase version: 4.32.9

Python version: 3.10.0

Question:

Has anyone experienced similar issues with SeleniumBase and xvfb on Ubuntu? Are there additional configurations required for xvfb or for undetected Chrome mode to work on an Ubuntu VM?

Any suggestions or advice would be greatly appreciated!

1
  • Have you installed ALL dependencies? Commented Nov 13, 2024 at 18:07

1 Answer 1

1

I was having the same problem as you the solution for me was to find the binary location for the google-chrome and include it in the sb probably, this is what will work with you

chrome_binary_path = "/opt/google/chrome/google-chrome" with SB(uc=True, xvfb=True, binary_location = chrome_binary_path)

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.