7

I am using VS 2022 and have a c# library that targets framework 4.5 up to .net 6. I have a MS Test project for the library. I use dynamic data heavily and recently noticed that not all of my test data was appearing in my test results. I have broken out the offending problem into as small as I can to prove out what I had been seeing.

If I put a breakpoint in HasExpectedItems2() and debug the test, I see it hit 3x and index is 0, 1, 2 respectively, as I would expect. However, if I put a breakpoint in HasExpectedItems1() and debug the test, it only hits 2x, and items contains 3 items, and is null, respectively. I am not seeing items ever contain 0 items, which should be one of the data rows. If I comment out the 3rd List in DumbItems (which contains the 3 objects), HasExpectedItems1() still hits 2x, and items contains 0 items, and is null, respectively, as expected. So it appears by having a populated list, it somehow ignores the empty list. I've been trying to resolve this for days and am completely baffled on what is going on here.

using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using Microsoft.VisualStudio.TestTools.UnitTesting;
#if NET45_OR_GREATER
using System.Web.Mvc;
#elif NETCOREAPP3_0_OR_GREATER
using Microsoft.AspNetCore.Mvc.Rendering;
#endif

[TestClass]
[ExcludeFromCodeCoverage]
public class StaticTestingUnitTests
{
    protected static readonly IEnumerable<SelectListItem>[] DumbItems = new IEnumerable<SelectListItem>[]
    {
        null,
        new List<SelectListItem>(),
        new List<SelectListItem>
        {
            new SelectListItem { Text = "One", Value = "1" },
            new SelectListItem { Text = "Two", Value = "2" },
            new SelectListItem { Text = "Three", Value = "3" },
        },
    };

    protected static IEnumerable<object[]> GetDumbTest1()
    {
        foreach (object item in DumbItems)
        {
            yield return new object[] { item };
        }
    }

    protected static IEnumerable<object[]> GetDumbTest2()
    {
        for (int i = 0; i < DumbItems.Length; i++)
        {
            yield return new object[] { i };
        }
    }

    [DataTestMethod]
    [DynamicData(nameof(GetDumbTest1), DynamicDataSourceType.Method)]
    public void HasExpectedItems1(IEnumerable<SelectListItem> items)
    {
        Assert.AreEqual(5, 5);
    }

    [DataTestMethod]
    [DynamicData(nameof(GetDumbTest2), DynamicDataSourceType.Method)]
    public void HasExpectedItems2(int index)
    {
        Assert.AreEqual(5, 5);
    }
}
3
  • I believe this should be reported as a bug to github.com/microsoft/testfx/issues Your tests are running fine with MSTest.TestAdapter/TestFramework version 2.2.3. If you upgrade to 2.2.5 you will see incorrect results you described (2.2.4 was deprecated, so I can only imaging a lot of things were broken there). Downgrade to 2.2.3 and test it yourself. Commented Apr 20, 2022 at 19:54
  • @Peska, Thank you very much. This has been driving me nuts for days and I didn't think to move back to a previous version. I reported as a bug so I'll see what happens with it. Commented Apr 21, 2022 at 18:14
  • I've just hit the exact same issue using version 2.2.10. Do you have some updates? Can you share the link to the bug report you made? Commented Sep 25, 2023 at 14:53

1 Answer 1

13

This is happening due to an undocumented breaking change in MsTest 2.2.4 which changes the default behavior for discovering ITestDataSource test cases (i.e. including DynamicData) to happen during Test Discovery instead of during Test Execution.

To restore the previous behavior, upgrade to MsTest 2.2.6 or newer, and add the following assembly attribute:

[assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]
Sign up to request clarification or add additional context in comments.

1 Comment

This pointed me to the right direction, thanks. TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringDetection) (which is obviously the default) has some very strange side effects. For example, data sources which use DateTime.Now are evaluated and cached at detection time and hence contain unexpected values at execution time This should be documented somewhere and changed to opt-in model, as requested by many!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.