Incorrect format for TestCaseFilter Error: Missing '(' #691

RabidArts · 2020-01-09T13:00:34Z

Environment

NUnit 3.12
NUnit Adapter 3.16
VS Professional 2019 Version 16.4.2
.NET Framework 4.7.2

Description
When executing a category of tests (e.g. Run All), if a test is included in the list of tests that contains an escaped double quote followed by a closing parenthesis, the following error may occur and the test run will fail:

An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('.

Note
There are a number of recently closed issues (thanks :) ) that are similar regarding braces or quotes in the test case name or arguments, but this is not quite the same (esp. since it breaks despite resolution of the other issues)

Reproduction
Run this code, and examine the Test Output window to see the exception as per above.

    public class Tests
    {
        [TestCase("Works fine", ")")]
        [TestCase("Works fine", "()")]
        [TestCase("Works fine", "(")]
        [TestCase("Works fine", "\"(")]
        [TestCase("Works fine", ")\"")]
        [TestCase("Works fine", "(\"")]
        [TestCase("Works fine", "()\"")]
        [TestCase("Breaks test executor when using Run All", "\"()")]
        [TestCase("Breaks test executor when using Run All", "\")")]        
        public void TestName(string expectedBehaviour, string data)
        {
            //Irrelevant
        }
    }

The text was updated successfully, but these errors were encountered:

OsirisTerje · 2020-01-09T15:25:24Z

Thanks! We're aware of these cases too, but thanks for the test code!
There are even more bizarre behavior here. I added some more with explanations at https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue691/Issue691

Part of this is a Visual Studio issue, part of it is the name handling in the adapter - and these things are not properly aligned yet.

RabidArts · 2020-01-10T07:16:52Z

Thanks for the response & update.

I can confirm that I'm also getting test explorer issues as described in issue 691, as well as general issues with test explorer being glitchy and not updating status from the nunit test run.

This is why I cringe every time I update something that works :'( (VS in this case)

RudeySH · 2020-05-12T20:35:50Z

I just ran into this issue as well. I'm using dotnet test as an alternative for now.

bklooste · 2020-05-15T02:49:34Z

Still open ?

igabesz · 2020-06-18T19:11:48Z

A workaround for fellow developers / testers who just want to make the tests run again: Wrap your test case data in a TestCaseData object, and return with an IEnumerable<TestCaseData> instead of an IEnumerable<object[]>. Fill the TestName of each of these cases; avoid special characters, and "tadaa" the tests will run again properly.

// Still looking forward to a proper fix though.

DrPepperBianco · 2021-01-22T12:11:15Z

I had just the same error. I solved it by removing all tuples form my tests. At least from the TestData and the function interface. Inside the test function I could again use tuples.

BTW: Both kind of tuples caused problems. Meaning ValueTuples (like (string a, string b) = ("Foo", "Bar");) as well as classical Tuples (like Tuple<string, string> t = Tuple.Create("Foo", "Bar");).

After I replaced all places, where I used those Tuples in my test data (TestCaseSource), everything works now.

@igabesz To wrap all data in a new object probably works, but I actually used tuples in some places, to not have to write a new class for every test.

davidcatriel · 2021-02-05T20:56:43Z

Same issue happening with nunit test adapter v 3.17.0.0.

davidcatriel · 2021-02-05T21:03:52Z

Found out this issue is also documented in 622 (#622), which in turn has been addressed in the nunit test adapter 3.16. I've just gone all the way up to 4.0.0.0 beta, though, and no change :(

hfickes · 2021-09-27T14:50:43Z

I am running the 4.0 release and am seeing this problem. Could the fix not have made it from 3.16 to 4.0?

OsirisTerje · 2021-09-27T17:20:01Z

I assume you refer to the fix in 3.16 that had to be reverted in 3.16.1. It can still be invoked using runsettings options, see https://docs.nunit.org/articles/vs-test-adapter/Adapter-Release-Notes.html#nunit3-test-adapter-for-visual-studio---version-3161---january-16-2020

OsirisTerje · 2021-10-06T18:23:09Z

It seems this works now, using the latest VS2019, version 16.11.3. I have added the repro to the repro-repo together with the same for #711 and #549 which are similar to this one. All works. See repro solution here: https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue711

This works both with and without Real Time Test Discovery

mikhail-barg · 2021-10-06T20:21:07Z

Just got this in 16.11.3

OsirisTerje · 2021-10-07T19:53:58Z

@mikhail-barg You mean it still failed for you? Can you please add a repro for that.

mikhail-barg · 2021-10-07T20:55:08Z

Yes, I think so. Will double-check and try to create a repro

OsirisTerje · 2021-10-07T21:48:38Z

Awesome if you also could check the TestFrom691 in the repo and sln https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue711 , and check that both with and without Real time test discovery. If those works, and your new repro doesn't we possibly have another issue here. If those doesn't work, then there is something else here that is "rotten".

mikhail-barg · 2021-10-10T13:08:41Z

Okay, so I've got a (not minimal) repro here: https://github.com/mikhail-barg/jsonata.net.native/tree/repro_for_nunit_691
But then I've decided to update VS to 16.11.4 and now the issue is gone. So it seems that it really was fixed, but the final fix came in 16.11.4.

mikhail-barg · 2021-10-10T13:16:59Z

Well, no, it still does reproduce:

and I was able to locate the culpit. It's a test named
variables.case011 (( $foo := "defined"; ( $foo := nothing; $foo ) ))

mikhail-barg · 2021-10-10T14:06:48Z

So I was able to minimize the repro to this:

    public class Tests
    {
        [TestCaseSource(nameof(GetTestCases))]
        public void Test1()
        {
            Assert.Pass();
        }

        public static List<TestCaseData> GetTestCases()
        {
            return new List<TestCaseData>() {
                new TestCaseData() {
                    TestName = "name ( () )"
                }
            };
        }
    }

Here's the project: Test691.zip

mikhail-barg · 2021-10-10T14:11:15Z

I also noticed that VS created test project with not latest package versions, so I updated everything to latest version and got a bit different output:

========== Starting test run ==========
NUnit Adapter 4.0.0.0: Test execution started
При вызове исполнителя "executor://nunit3testexecutor/" возникло исключение: Неверный формат фильтра TestCaseFilter Ошибка: отсутствует "(". Задайте правильный формат и повторите попытку. Обратите внимание, что использование неверного формата может привести к тому, что ни один из тестов не будет выполнен.
Трассировка стека:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Discovery.DiscoveryContext.GetTestCaseFilter(IEnumerable`1 supportedProperties, Func`2 propertyProvider)
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_TfsTestCaseFilterExpression() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 100
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_IsEmpty() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 102
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.InitializeForExecution(IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 248
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 105
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 39 ms ==========

Test691up.zip

mikhail-barg · 2021-10-10T14:15:40Z

Awesome if you also could check the TestFrom691 in the repo and sln https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue711 , and check that both with and without Real time test discovery.

So I just checked out the repo, opened the Issue711\NUnitTestAdapterCharacters.sln and pressed "Run" on the topmost test group in the tree. This gave me:

Log level is set to Informational (Default).
Loaded 0 test records from the solution cache in 0,466 sec.
Test data store opened in 0,026 sec.
Starting test discovery for requested test run
========== Starting test discovery ==========
NUnit Adapter 4.0.0.0: Test discovery starting
NUnit Adapter 4.0.0.0: Test discovery complete
========== Test discovery finished: 62 Tests found in 1,1 sec ==========
========== Starting test run ==========
NUnit Adapter 4.0.0.0: Test execution started
При вызове исполнителя "executor://nunit3testexecutor/" возникло исключение: Неверный формат фильтра TestCaseFilter Ошибка: недопустимое условие "FullyQualifiedName=UnitTests.TestFilterBug.Invalid = TestNameEquals". Задайте правильный формат и повторите попытку. Обратите внимание, что использование неверного формата может привести к тому, что ни один из тестов не будет выполнен.
Трассировка стека:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Discovery.DiscoveryContext.GetTestCaseFilter(IEnumerable`1 supportedProperties, Func`2 propertyProvider)
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_TfsTestCaseFilterExpression() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 100
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_IsEmpty() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 102
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.InitializeForExecution(IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 248
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 105
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 44 ms ==========

mikhail-barg · 2021-10-10T14:18:28Z

Did the same as above with the Issue691\Issue691\Issue691.sln and everything seem to be working:

Log level is set to Informational (Default).
Loaded 0 test records from the solution cache in 0,231 sec.
Test data store opened in 0,010 sec.
Starting test discovery for requested test run
========== Starting test discovery ==========
NUnit Adapter 3.16.0.0: Test discovery starting
NUnit Adapter 3.16.0.0: Test discovery complete
========== Test discovery finished: 9 Tests found in 891,4 ms ==========
========== Starting test run ==========
NUnit Adapter 3.16.0.0: Test execution started
Running selected tests in d:\Projects\nunit3-vs-adapter.issues\Issue691\Issue691\Issue691\bin\Debug\netcoreapp3.1\Issue691.dll
   NUnit3TestExecutor converted 9 of 9 NUnit test cases
NUnit Adapter 3.16.0.0: Test execution complete
========== Test run finished: 9 Tests (9 Passed, 0 Failed, 0 Skipped) run in 234 ms ==========

And all tests are green.

OsirisTerje · 2021-10-10T14:32:18Z

The last one runs 3.16.0, that version doesn't run methods separately, so there should be no issue. What about 4.0 ?

mikhail-barg · 2021-10-10T14:36:32Z

Updated all packages in Issue691.sln to latest versions, and still everything work:

========== Starting test discovery ==========
NUnit Adapter 4.0.0.0: Test discovery starting
NUnit Adapter 4.0.0.0: Test discovery complete
========== Test discovery finished: 9 Tests found in 531 ms ==========
========== Starting test run ==========
NUnit Adapter 4.0.0.0: Test execution started
Running selected tests in d:\Projects\nunit3-vs-adapter.issues\Issue691\Issue691\Issue691\bin\Debug\netcoreapp3.1\Issue691.dll
   NUnit3TestExecutor discovered 9 of 9 NUnit test cases using Current Discovery mode, Non-Explicit run
NUnit Adapter 4.0.0.0: Test execution complete
========== Test run finished: 9 Tests (9 Passed, 0 Failed, 0 Skipped) run in 240 ms ==========

OsirisTerje · 2021-10-10T14:41:54Z

@mikhail-barg Thanks for checking this out :-)

redwyre · 2021-11-14T21:29:32Z

I'm getting this error with the latest packages (for vs2019) running in vs 2022:

========== Starting test run ==========
Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
NUnit Adapter 4.1.0.0: Test execution started
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.
Stack trace:
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Discovery.DiscoveryContext.GetTestCaseFilter(IEnumerable`1 supportedProperties, Func`2 propertyProvider)
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_TfsTestCaseFilterExpression() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 100
   at NUnit.VisualStudio.TestAdapter.VsTestFilter.get_IsEmpty() in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\VsTestFilter.cs:line 102
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.InitializeForExecution(IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 248
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 105
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

========== Test run finished: 0 Tests (0 Passed, 0 Failed, 0 Skipped) run in 131 ms ==========

I've created a gist with the relevant files I can think of: https://gist.github.com/redwyre/0fb0c9f177af278f76aa8cf49c1eb1dc

It seems like NUnit is just calling ToString() on all the parameters to make the test name, and in some cases it doesn't work properly with more complex objects.

I think I can work around this by wrapping each record in TestCaseData and name them, but it overrides the full name of the test not just the postfix. I'm not very experienced with NUnit myself so not sure if there is a better way.

Nunit bug is that InlineData in strings need to have balanced quotes. Update test case. This shouldn't lose coverage since the quote comes after the intellisense cursor.

nthornton2010 · 2022-04-22T00:58:21Z

I was on 4.0.0.0 and experienced this issue when using ValueSource with an array of tuples. I updated to 4.2.0.0 and am still experiencing this.

OsirisTerje · 2022-04-22T06:13:04Z

@nthornton2010 Have you checked the repro above? Does it appear there? How does your code differ from that repro?

Lohoris · 2022-04-22T10:24:41Z

I have a huge project and I have no clue where this is coming from. Is there a way to add the bad test name to the logs?

OsirisTerje · 2022-04-22T10:34:31Z

@Lohoris You can enable the VSTest trace logs . Since this error crashes the adapter, the only way is to use the vstest logs.

Lohoris · 2022-04-22T13:42:05Z

@Lohoris You can enable the VSTest trace logs . Since this error crashes the adapter, the only way is to use the vstest logs.

I mean, the normal logs that end up in the Output/Tests pane do not show anything interesting, or do you mean there's a way to increase the verbosity of that? Or finding some logfiles somewhere?

OsirisTerje · 2022-04-22T18:10:44Z

@Lohoris Have a look at https://github.com/microsoft/vstest-docs/blob/main/docs/diagnose.md

nthornton2010 · 2022-04-22T22:19:08Z

@OsirisTerje I saw the findings in this issue - #935 - and figured out that I'm suffering from the same problem. After disabling that setting in Visual Studio, it started working.

Lohoris · 2022-04-25T13:58:36Z

@nthornton2010

I saw the findings in this issue - #935 - and figured out that I'm suffering from the same problem. After disabling that setting in Visual Studio, it started working.

It's already off here, so unfortunately I'll need to investigate further. I'll have a look to the diagnose tool @OsirisTerje linked, thanks!

OsirisTerje added the is:bug label Jan 9, 2020

OsirisTerje added External Tracked here, but must be fixed in external tool VS Issue labels Jan 9, 2020

jnm2 mentioned this issue May 9, 2020

NUnit Injection "Attack" #743

Closed

RudeySH mentioned this issue May 12, 2020

Fix media link classes not being added xoofx/markdig#435

Merged

aolszowka mentioned this issue May 18, 2020

Incorrect format for TestCaseFilter Error: Invalid Condition - Using = in SetName #711

Closed

sbomer mentioned this issue Jun 9, 2020

Run All Tests not working in VS dotnet/linker#1259

Closed

OsirisTerje added the is:FQNIssue label Sep 27, 2021

OsirisTerje closed this as completed Oct 6, 2021

OsirisTerje added the closed:fixedinnewversion This has been fixed in a newer version label Oct 6, 2021

mikhail-barg added a commit to mikhail-barg/jsonata.net.native that referenced this issue Oct 10, 2021

repro for nunit/nunit3-vs-adapter#691

c68365c

mikhail-barg added a commit to mikhail-barg/jsonata.net.native that referenced this issue Oct 10, 2021

repro for nunit/nunit3-vs-adapter#691

8d541d6

loop-evgeny mentioned this issue Dec 27, 2021

An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error when using Real Time Discovery #935

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect format for TestCaseFilter Error: Missing '(' #691

Incorrect format for TestCaseFilter Error: Missing '(' #691

RabidArts commented Jan 9, 2020

OsirisTerje commented Jan 9, 2020

RabidArts commented Jan 10, 2020 •

edited

Loading

RudeySH commented May 12, 2020

bklooste commented May 15, 2020

igabesz commented Jun 18, 2020

DrPepperBianco commented Jan 22, 2021

davidcatriel commented Feb 5, 2021

davidcatriel commented Feb 5, 2021 •

edited

Loading

hfickes commented Sep 27, 2021

OsirisTerje commented Sep 27, 2021

OsirisTerje commented Oct 6, 2021 •

edited

Loading

mikhail-barg commented Oct 6, 2021

OsirisTerje commented Oct 7, 2021

mikhail-barg commented Oct 7, 2021

OsirisTerje commented Oct 7, 2021

mikhail-barg commented Oct 10, 2021 •

edited

Loading

mikhail-barg commented Oct 10, 2021 •

edited

Loading

mikhail-barg commented Oct 10, 2021

mikhail-barg commented Oct 10, 2021

mikhail-barg commented Oct 10, 2021

mikhail-barg commented Oct 10, 2021

OsirisTerje commented Oct 10, 2021

mikhail-barg commented Oct 10, 2021

OsirisTerje commented Oct 10, 2021

redwyre commented Nov 14, 2021

nthornton2010 commented Apr 22, 2022 •

edited

Loading

OsirisTerje commented Apr 22, 2022

Lohoris commented Apr 22, 2022

OsirisTerje commented Apr 22, 2022

Lohoris commented Apr 22, 2022

OsirisTerje commented Apr 22, 2022

nthornton2010 commented Apr 22, 2022 •

edited

Loading

Lohoris commented Apr 25, 2022

Incorrect format for TestCaseFilter Error: Missing '(' #691

Incorrect format for TestCaseFilter Error: Missing '(' #691

Comments

RabidArts commented Jan 9, 2020

OsirisTerje commented Jan 9, 2020

RabidArts commented Jan 10, 2020 • edited Loading

RudeySH commented May 12, 2020

bklooste commented May 15, 2020

igabesz commented Jun 18, 2020

DrPepperBianco commented Jan 22, 2021

davidcatriel commented Feb 5, 2021

davidcatriel commented Feb 5, 2021 • edited Loading

hfickes commented Sep 27, 2021

OsirisTerje commented Sep 27, 2021

OsirisTerje commented Oct 6, 2021 • edited Loading

mikhail-barg commented Oct 6, 2021

OsirisTerje commented Oct 7, 2021

mikhail-barg commented Oct 7, 2021

OsirisTerje commented Oct 7, 2021

mikhail-barg commented Oct 10, 2021 • edited Loading

mikhail-barg commented Oct 10, 2021 • edited Loading

mikhail-barg commented Oct 10, 2021

mikhail-barg commented Oct 10, 2021

mikhail-barg commented Oct 10, 2021

mikhail-barg commented Oct 10, 2021

OsirisTerje commented Oct 10, 2021

mikhail-barg commented Oct 10, 2021

OsirisTerje commented Oct 10, 2021

redwyre commented Nov 14, 2021

nthornton2010 commented Apr 22, 2022 • edited Loading

OsirisTerje commented Apr 22, 2022

Lohoris commented Apr 22, 2022

OsirisTerje commented Apr 22, 2022

Lohoris commented Apr 22, 2022

OsirisTerje commented Apr 22, 2022

nthornton2010 commented Apr 22, 2022 • edited Loading

Lohoris commented Apr 25, 2022

RabidArts commented Jan 10, 2020 •

edited

Loading

davidcatriel commented Feb 5, 2021 •

edited

Loading

OsirisTerje commented Oct 6, 2021 •

edited

Loading

mikhail-barg commented Oct 10, 2021 •

edited

Loading

mikhail-barg commented Oct 10, 2021 •

edited

Loading

nthornton2010 commented Apr 22, 2022 •

edited

Loading

nthornton2010 commented Apr 22, 2022 •

edited

Loading