Selenium Webdriver tests always fail on Azure DevOps but work perfectly on dev machine

Question

Using a .net core 3.1 XUnit test project and the C# bindings for the Selenium WebDriver API, my tests succeed when I run them locally on my development machine but always fail when run in an Azure DevOps release pipeline.

Here is the Yaml definition for the test task

steps:
- task: VSTest@2
  displayName: 'VsTest - testAssemblies'
  inputs:
    testAssemblyVer2: '**\--REDACTED--.dll'
    uiTests: true
    runInParallel: false
    rerunFailedTests: true

I have followed the microsoft guidelines and activated uiTest and ensured runInParallel is false in the pipeline definition. I have also tried running the tests with every possible permutation of these two

The ChromeDriver version is 80.0.3987.106, which is the version pre-installed on the agent I'm using. I target the ChromeDriver binaries pre-installed on the agent using the ChromeWebDriver environment variable (its value is C:\\SeleniumWebDrivers\\ChromeDriver). I have tried using the windows-2019 and the vs2017-win2016 agents. This did not change anything.

The web driver is running in headless mode.

Here is a relevant sample of the output from the Azure DevOps pipeline. I believe the test does not actually fail because an element could not be located (since the tests consistently work in local), rather because the driver itself fails.

[...]
2020-03-06T14:38:02.2076961Z Starting ChromeDriver 80.0.3987.106 (f68069574609230cf9b635cd784cfb1bf81bb53a-refs/branch-heads/3987@{#882}) on port 49976
2020-03-06T14:38:02.2078676Z Only local connections are allowed.
2020-03-06T14:38:02.2079642Z Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
2020-03-06T14:38:28.0289610Z [xUnit.net 00:00:31.64]       OpenQA.Selenium.NoSuchElementException : no such element: Unable to locate element: {"method":"css selector","selector":"#autocomplete"}
2020-03-06T14:38:28.0296747Z   (Session info: headless chrome=80.0.3987.122)
2020-03-06T14:38:28.0297355Z [xUnit.net 00:00:31.64]       Stack Trace:
2020-03-06T14:38:28.0328125Z ##[error][xUnit.net 00:00:31.64]     --REDACTED--Test.ArticleSearchTest.SearchJuraSaturdaySaturdayWithoutDatesAndNoOsmSelectionResultIsPageUfi [FAIL]
2020-03-06T14:38:28.0338184Z [xUnit.net 00:00:31.64]            at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
2020-03-06T14:38:28.0339005Z [xUnit.net 00:00:31.64]            at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
2020-03-06T14:38:28.0339833Z [xUnit.net 00:00:31.64]            at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(String mechanism, String value)
2020-03-06T14:38:28.0340530Z [xUnit.net 00:00:31.64]            at OpenQA.Selenium.Remote.RemoteWebDriver.FindElementById(String id)
2020-03-06T14:38:28.0341218Z [xUnit.net 00:00:31.64]            at OpenQA.Selenium.By.<>c__DisplayClass16_0.<Id>b__0(ISearchContext context)
2020-03-06T14:38:28.0341907Z [xUnit.net 00:00:31.64]            at OpenQA.Selenium.By.FindElement(ISearchContext context)
2020-03-06T14:38:28.0342538Z [xUnit.net 00:00:31.64]            at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(By by)
2020-03-06T14:38:28.0343418Z [xUnit.net 00:00:31.64]         C:\agent\_work\25\s\Tests\--REDACTED--Test\SeleniumSearchUtils.cs(16,0): at --REDACTED--Test.SeleniumSearchUtils.SetDestinationToJuraWithoutSelectingDestinationInDropdown(IWebDriver driver)
2020-03-06T14:38:28.0344566Z [xUnit.net 00:00:31.64]         C:\agent\_work\25\s\Tests\--REDACTED--Test\Tests\ArticleSearchTest.cs(118,0): at --REDACTED--Test.ArticleSearchTest.SearchJuraSaturdaySaturdayWithoutDatesAndNoOsmSelectionResultIsPageUfi()
2020-03-06T14:38:28.0345417Z [xUnit.net 00:00:31.65]       Output:
2020-03-06T14:38:28.0346024Z [xUnit.net 00:00:31.65]         Selenium target url: http://test.--REDACTED--.com/de/frankreich/camping-bretagne
2020-03-06T14:38:28.0375453Z [xUnit.net 00:00:31.65]   Finished:    --REDACTED--Test
[...]

As advised by Greg, I increased the allowed wait time before an element not found exception is thrown. I set it to 120 seconds, which lead to a timeout error Relevant stack trace

   ---- System.Net.WebException : The operation has timed out.
2020-03-06T15:39:37.4581244Z [xUnit.net 00:02:24.81]       Stack Trace:
2020-03-06T15:39:37.4582083Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.Remote.HttpCommandExecutor.MakeHttpRequest(HttpRequestInfo requestInfo)
2020-03-06T15:39:37.4582955Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.Remote.HttpCommandExecutor.Execute(Command commandToExecute)
2020-03-06T15:39:37.4583816Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.Remote.DriverServiceCommandExecutor.Execute(Command commandToExecute)
2020-03-06T15:39:37.4584863Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
2020-03-06T15:39:37.4585662Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(String mechanism, String value)
2020-03-06T15:39:37.4586387Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.Remote.RemoteWebDriver.FindElementById(String id)
2020-03-06T15:39:37.4587146Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.By.<>c__DisplayClass16_0.<Id>b__0(ISearchContext context)
2020-03-06T15:39:37.4587838Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.By.FindElement(ISearchContext context)
2020-03-06T15:39:37.4588507Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.Remote.RemoteWebDriver.FindElement(By by)
2020-03-06T15:39:37.4589490Z [xUnit.net 00:02:24.81]         C:\agent\_work\25\s\Tests\--REDACTED--Test\SeleniumSearchUtils.cs(16,0): at --REDACTED--Test.SeleniumSearchUtils.SetDestinationToJuraWithoutSelectingDestinationInDropdown(IWebDriver driver)
2020-03-06T15:39:37.4590637Z [xUnit.net 00:02:24.81]         C:\agent\_work\25\s\Tests\--REDACTED--Test\Tests\ArticleSearchTest.cs(118,0): at --REDACTED--Test.ArticleSearchTest.SearchJuraSaturdaySaturdayWithoutDatesAndNoOsmSelectionResultIsPageUfi()
2020-03-06T15:39:37.4613895Z [xUnit.net 00:02:24.81]         ----- Inner Stack Trace -----
2020-03-06T15:39:37.4614741Z [xUnit.net 00:02:24.81]            at System.Net.HttpWebRequest.GetResponse()
2020-03-06T15:39:37.4615470Z [xUnit.net 00:02:24.81]            at OpenQA.Selenium.Remote.HttpCommandExecutor.MakeHttpRequest(HttpRequestInfo requestInfo)
2020-03-06T15:39:37.4616428Z [xUnit.net 00:02:24.81]       Output:
2020-03-06T15:39:37.4617060Z [xUnit.net 00:02:24.81]         Selenium target url: http://test.--REDACTED--.com/de/frankreich/camping-bretagne
2020-03-06T15:39:37.4617775Z [xUnit.net 00:02:24.82]   Finished:    --REDACTED--Test

Note: I tried adding the no-sandbox option to the ChromeDriver, to no avail. UPDATE: After going through the logs in details again, I stumbled upon this line:

OpenQA.Selenium.WebDriverException : The HTTP request to the remote WebDriver server for URL http://localhost:49946/session/a0dac255bd41cc2bb8c6882761408f12/element timed out after 60 seconds.

I am absolutely positive the URL passed to the GoToUrl method is http://test.--REDACTED--.com/de/frankreich/camping-bretagne, valid URL (which is on a remote server, DNS resolution and all)

Could the localhost:49946 reference some kind of local selenium component that hasn't been launched properly ? I could not find any more information about this "back-end" in the error logs or on the internet ...

If you are still getting test failures, please add more info to your question: HTML, c# to initialize the web driver, c# code that tried to interact with the page, and which browser, version and web driver and version is being used in Azure DevOps. — Greg Burghardt
– Greg Burghardt, Commented Mar 7, 2020 at 0:41
From website github.com/actions/virtual-environments/blob/master/images/win/…, 80.0.3987.106 is IE Driver, not ChromeDriver, do you miss something? Could you provide more information of your project? — Cece Dong - MSFT
– Cece Dong - MSFT, Commented Mar 9, 2020 at 9:27
@CeceDong-MSFT as per this blog post chromereleases.googleblog.com/2020/02/…, 80.0.3987.106 is a valid Google Chrome version — Mathieu VIALES
– Mathieu VIALES, Commented Mar 10, 2020 at 11:43
@MathieuVIALES Could you please provide a sample project and detailed steps to reproduce your issue? — Cece Dong - MSFT
– Cece Dong - MSFT, Commented Mar 11, 2020 at 8:52
Thank you for the updates. At this point you have gotten around your original exception. You now have a different problem. Please post a different question. Basically Selenium is making a web service call to the web driver process, and it didn't hear back. This could be any number of issues. A mismatch between the browser version and web driver version. In DevOps, it could even be a firewall issue preventing Selenium from making web service calls to the web driver process. — Greg Burghardt
– Greg Burghardt, Commented Apr 16, 2020 at 15:26

Greg Burghardt · Accepted Answer · 2020-03-06 15:22:27Z

2

You are getting a very explicit exception:

OpenQA.Selenium.NoSuchElementException : no such element: Unable to locate element: {"method":"css selector","selector":"#autocomplete"}

It is failing, because it cannot find the element. Since these tests work locally this leads me to believe the failure is due to a race condition between the web browser and Selenium. The common fix for this is to use a WebDriverWait object:

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(30));
var element = wait.Until(ExpectedConditions.PresenceOfElementLocated(By.CssSelector("#autocomplete")));

// now element exists

These race conditions are common when you run tests on a fast local machine, but then those same tests fail on a build server, which is likely a lot busier than your machine.

answered Mar 6, 2020 at 15:22

Greg Burghardt

19.2k10 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mathieu VIALES Over a year ago

I have just increased the implicit timeout to 120 seconds, just in case, leading to a Timeout Error in the build pipeline. Mu timeout already was of 10 seconds, which I consider to be very high ... if the pipeline is expected to be THAT slow I don't think it would be a viable option for running a battery of tests. Are you sure that the error comes from the agent's slowness ?

Mathieu VIALES Over a year ago

I will try to use an explicit WebDriverWait element, just in case the implicit timeout feature is the origin of the problem.

Mathieu VIALES Over a year ago

I have updated my question, it seems I missed part of the logs :-/

undetected Selenium · Accepted Answer · 2020-03-06 21:30:44Z

@GregBurghardt answer was near perfect (+1). A bit of more information in terms of:

The relevant HTML
Line of code
Binary version information

would have helped us to analyze the issue in a better way.

However, as you are seeing:

OpenQA.Selenium.NoSuchElementException : no such element: Unable to locate element: {"method":"css selector","selector":"#autocomplete"}

To locate/interact/click the element, you need to induce WebDriverWait in conjunction with ExpectedConditions as ElementToBeClickable() and you can use either of the following Locator Strategies:

Id:

var element = new WebDriverWait(driver, TimeSpan.FromSeconds(20)).Until(ExpectedConditions.ElementToBeClickable(By.Id("autocomplete")));

CssSelector:

var element = new WebDriverWait(driver, TimeSpan.FromSeconds(20)).Until(ExpectedConditions.ElementToBeClickable(By.CssSelector("#autocomplete")));

XPath:

var element = new WebDriverWait(driver, TimeSpan.FromSeconds(20)).Until(ExpectedConditions.ElementToBeClickable(By.XPath("//*[@id='autocomplete']")));

Reference

You can find a couple of relevant discussions in:

Collectives™ on Stack Overflow

Selenium Webdriver tests always fail on Azure DevOps but work perfectly on dev machine

2 Answers 2

3 Comments

Reference

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Reference

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related