When group of tests contains sub-group, VS 2019 Test Explorer shows Not Run, while all tests pass separately

[wlf84k]

Setup:

• NUnit 3 Test Adapter 3.17.0 (tried both VSIX and nuget versions)
• VS 2019 Professional 16.6.3
• NUnit 3.11.0
• Test project is .NET Framework 4.7.2 x86, dependencies might be same or .NET Standard and AnyCPU

Issue:

Test project contains 3000+ tests, in Test Explorer they are categorized and grouped so that the tree of categories/tests has multiple levels. All tests are discovered correctly. All tests can be run separately and they pass. It is also possible to run a group of tests but only if it doesn't contain a sub-group.

If group of groups of tests is run then nothing is executed and each test is marked as Not Run with a blueish exclamation mark icon. No additional information or error is provided

Some groups always work and some always not work (within the same test assembly/project)

This worked fine with VS 2017 15.9.23 and NUnit 3 Test Adapter 3.9.0

[OsirisTerje]

@wlf84k Are you able to produce a repro for this? Screenshots? Logs?
Do you use Explicit or Ignore attributes?
Are the tests datadriven, using TestCase or TestCaseSource ?
Can you enable the Dumps upload relevant parts of those?

[wlf84k]

@OsirisTerje

I've been playing with a debugable version of test adapter and I've found that this is related to TestCaseSource attribute and it is probably also a test discovery related issue. I managed to create a simple repro:
NUnitTestProject1.zip

• Test Explorer incorrectly discovers 4 tests but there should be 3 (1 regular test, and 1 "TestCaseSource" test with 2 data items)
• When you run all tests when top node, middle node and "Tests" is selected (see picture below), then no test is run the same for the last test on the list (TestCaseSOurceTest).
• When "A) First Scenario", "B) Second Scenario" or "SimpleTest" are run separately, they pass.

[Edit] Version 4.0.0 from GitHub master branch also produces the same issues

I have also a general remark for test execution: when there is an issue with execution/discovery process or test assembly could not be loaded for some reason, Test Explorer never produces any info about it. It just stops execution and says "Not Run". Is it planned to support more info in Test Explorer?

[OsirisTerje]

@wlf84k Thanks !

1. The "ghost" TestCaseSourceTest comes from the Real Time Discovery in Visual Studio Test Explorer. If you turn that off, it will disappear. The MS PG is aware of this, and will be fixed by them, hopefully soon.

2. The exception that stops execution however, we might be able to fix.

I agree it would be great with more info in the Test Explorer. That is outside our scope [as an adapter] though. I'll see if we can get some more output in our own logs though.

[wlf84k]

@OsirisTerje Thanks for the info!

In the meanwhile I found a less clean workaround for this which doesn't require turning off real time discovery.

For the code:

    `
    public class CustomTestData
    {
        public double[] numbers;
    }

    public static IEnumerable<TestCaseData> TestDataFactory()
    {
        yield return new TestCaseData(new CustomTestData
        {
            numbers = new[] { 0.0, 1.0, 2.1 }
        }).SetName("A) First scenario");
    }

    [Test, TestCaseSource("TestDataFactory")]
    public void TestCaseSOurceTest(CustomTestData data)
    {
        Assert.That(data.numbers, Is.Not.Empty);
    }
    `

The workaround would be to change this:

    `
    public class CustomTestData
    {
        public double[] numbers;
        private string label;
        public CustomTestData(string label) => this.label = label;
        public override string ToString() => this.label;
    }

    public static IEnumerable<TestCaseData> TestDataFactory()
    {
        yield return new TestCaseData(new CustomTestData("A First scenario")
        {
            numbers = new[] { 0.0, 1.0, 2.1 }
        });
    }
    `

So that SetName() is not called and ToString() from data class is used instead.
Note that some characters like parenthesis should also be avoided for test labels.
This way VS Test Explorer would display "TestCaseSOurceTest" as a group of dynamically generated test cases (no ghosts).

Maybe someone wil find this useful but I think your proposal to just turn off real-time discovery is much easier and cleaner.

[OsirisTerje]

Thanks! Interesting approach :-)

It can be useful as a check later. These things should behave equally, so when we get to have the RTD fixed, this should be one of the checks.

[Eilon]

We just hit this as well in VS2019 and dotnet test while working on dotnet/MobileBlazorBindings#152.

[sliddington]

I was about to raise a separate issue but then looked more closely at this - my issue is the same but it runs all tests in the assemby not none and turning off live discovery fixes it. I was about to submit this:

• NUnit 3.13.12 and NUnit3TestAdapter 4.2.1
• Visual Studio 2019 Professional - version 16.11.12
• .net framework 4.7.2

In the below test explorer if I right click any of the levels in the hierarchy and run tests then it runs all tests in the assembly not the ones under that group.

If I individually select the actual tests as shown then it runs the correct tests.

If I then include any one of the 3 test function names which have test cases provided via TestCaseSource attribute then it runs all the tests in the assembly again.

[OsirisTerje]

@sliddington
Can you please upload a repro project?

You can a) Add it as a PR to https://github.com/nunit/nunit3-vs-adapter.issues, add it as a separate folder named IssueXXX where the XXX is the issue number, or b) Add it as a zipped attachment here , or c) Add it to your own repo and just post the link.

Make it as small as you can, it should compile "as is".
Please also confirm this ONLY happens when live discovery is on ?

[sliddington]

Hi @OsirisTerje.
I'm a bit busy at the moment but may be able to over the next few weeks.
Yes, it only happens when live discovery is on due to the addition of the function names for tests driven via a TestDataSource appearing in the test list. So it is another symptom of the same thing as reported here.