-
Notifications
You must be signed in to change notification settings - Fork 319
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
System.InvalidOperationException and System.Collections.Generic.KeyNotFoundException errors when running tests #4208
Comments
This is currently being investigated, we will post back when we have more info. |
I have this same issue and have found this to happen whenever you delete or rename a test method. |
I have the same issue - run the same test twice and then I can't run it a third time until I restart vs2022 community |
I'm having the same problem - Visual Studio Community 17.4.3 - but for a mixture of C++ and C# tests. Most of the time, a machine reboot seems to make the issue go away for a while - this is less than ideal :/ (Debug spew after about info) Cheers Microsoft Visual Studio Community 2022 Installed Version: Community Visual C++ 2022 00482-90000-00000-AA725 ADL Tools Service Provider 1.0 ASA Service Provider 1.0 ASP.NET and Web Tools 17.4.326.54890 ASP.NET Web Frameworks and Tools 2012 17.4.326.54890 Azure App Service Tools v3.0.0 17.4.326.54890 Azure Data Lake Tools for Visual Studio 2.6.5000.0 Azure Functions and Web Jobs Tools 17.4.326.54890 Azure Stream Analytics Tools for Visual Studio 2.6.5000.0 C# Tools 4.4.0-6.22580.4+d7a61210a88b584ca0827585ec6e871c6b1c5a14 CodeLensOopProviderCommands Extension 1.0 Common Azure Tools 1.10 Cookiecutter 17.0.22263.6 Linux Core Dump Debugging 1.0.9.33020 Microsoft Azure Hive Query Language Service 2.6.5000.0 Microsoft Azure Stream Analytics Language Service 2.6.5000.0 Microsoft JVM Debugger 1.0 NuGet Package Manager 6.4.0 P4VS - Helix Plugin for Visual Studio 2021.2.235.1465 Python - Django support 17.0.22263.6 Python - Profiling support 17.0.22263.6 Python with Pylance 17.0.22263.6 Razor (ASP.NET Core) 17.0.0.2246202+61cc048d36a3fc9246d2f04625988b19a18ab8f0 SQL Server Data Tools 17.0.62207.28050 Test Adapter for Boost.Test 1.0 Test Adapter for Google Test 1.0 ToolWindowHostedEditor 1.0 TypeScript Tools 17.0.10921.2001 Visual Basic Tools 4.4.0-6.22580.4+d7a61210a88b584ca0827585ec6e871c6b1c5a14 Visual C++ for Linux Development 1.0.9.33020 Visual F# Tools 17.4.0-beta.22512.4+525d5109e389341bb90b144c24e2ad1ceec91e7b Visual Studio IntelliCode 2.2 System.NullReferenceException: Object reference not set to an instance of an object. |
I have same issue on It seems that explorer fail to refresh tests. |
I'm having the same issue when renaming a test method. New test methods are detected and run correctly. When loading a project after the test window has initialized, then newly discovered tests will just fail to run. Everything worked fine in 2.2.10. I know there have been a lot of changes between this and v3. Hope it can be resolved :) For now, simply restarting Visual Studio gets everything working again. |
3.0.1 has been released, have you tried those yet? |
Hey there, we are still investigating the issue but at the moment it doesn't seem that the issue is caused directly by MSTest (it's possible this is linked to an updated dependency to Test Platform - we are looking into it). |
This looks to be an issue with the Test Explorer in Visual Studio. The problem is triggered by some new functionality in MSTest 3, however that is not where the problem lies. We are working on a fix currently and will let you know when we know more. |
Ues, I have the last version 3.0.1 |
Hey @Evangelink, I definitely think it's an issue caused directly by MSTest because it still throws exception even though I removed all other dependencies and other TestMethods in my test project. |
Hi @g-u-r-k-a-n, Yes issue is caused by MSTest but what I meant and that @peterwald explained is that we have used a new functionality for Test Explorer, through the use of Test Platform Object Model that seems to be causing the issue. So yes it's because you are using MSTest v3 but it's not linked to something we implement but rather either by Test Explorer change of behaviour or by some change on the object model implementation. Nonetheless, we are investigating the issue and will provide a fix as soon as we can. |
Hi, having the same issue with VS 2022 Enterprise version 17.4.3 Here is the VS 2022 Preview version info. Installed Version: Enterprise ADL Tools Service Provider 1.0 ASA Service Provider 1.0 ASP.NET and Web Tools 17.5.194.32986 Azure App Service Tools v3.0.0 17.5.194.32986 Azure Data Lake Tools for Visual Studio 2.6.5000.0 Azure Functions and Web Jobs Tools 17.5.194.32986 Azure Stream Analytics Tools for Visual Studio 2.6.5000.0 Bundler & Minifier 2.9.3 C# Tools 4.5.0-2.22580.13+ea0b6351c129c7decd9bf76f4388239509e28e0d Clean Solution 1.4.35 CleanBinAndObjCommand Extension 1.2.58 Clear MEF Component Cache 1.4 Code alignment 15.1.118.0 Common Azure Tools 1.10 Cookiecutter 17.0.22318.1 CreateUnitTestBoilerplateCommand Extension 1.0 Entity Framework Core Power Tools 2.5 Extensibility Message Bus 1.4.1 (main@2ee106a) File Differ 3.0.43 File Icons 2.7 IndentRainbowPackage Extension 1.0 Markdown Editor v2 2.0.130 Menu Extension 1.0 Microsoft Azure Hive Query Language Service 2.6.5000.0 Microsoft Azure Stream Analytics Language Service 2.6.5000.0 Microsoft Azure Tools for Visual Studio 2.9 Microsoft JVM Debugger 1.0 Mono Debugging for Visual Studio 17.5.2 (d15e47f) Node.js Tools 1.5.41130.1 Commit Hash:8e2bd0c47a236f67dd4a99f60aacfa306308387c NuGet Package Manager 6.5.0 OpenOnGitHub 1.3 Python - Profiling support 17.0.22318.1 Python with Pylance 17.0.22318.1 Rainbow Braces 1.0.40 Razor (ASP.NET Core) 17.0.0.2255201+5530de5c8323cd1ecacb497bbab5637907d0fc02 ResX Resource Manager ResXManager Select Next Occurrence 1.4.182 SonarLint for Visual Studio 6.9.0.54300 SQL Server Data Tools 17.2.31208.0 ToolWindowHostedEditor 1.0 TypeScript Tools 17.0.11130.2001 Visual Basic Tools 4.5.0-2.22580.13+ea0b6351c129c7decd9bf76f4388239509e28e0d Visual F# Tools 17.5.0-beta.22607.4+f5794982b82d308590542bd1e101162dc4f45a13 Visual Studio IntelliCode 2.2 Visual Studio Spell Check Everywhere VSSpellCheckEverywhere Visual Studio Spell Checker VSSpellChecker VisualStudio.DeviceLog 1.0 VisualStudio.Mac 1.0 VSColorOutput64 2022.2 Word Highlight With Margin Extension 1.0 Xamarin 17.5.0.120 (main@4c08778) Xamarin Designer 17.5.2.64 (remotes/origin/d17-5@627f860876) Xamarin Templates 17.5.3 (3c53880) Xamarin.Android SDK 13.1.99.61 (main/2e10309) Xamarin.iOS and Xamarin.Mac SDK 16.1.1.27 (933c6c2c9) |
Ok, this is really weird! I added Since I read (above) that some folks have resolved the issue by rebooting, I closed VS and rebooted my computer. And... now the problem has magically disappeared. Don't know if this helps the investigation at all. |
I am consistently hitting this issue too in VS 2022 17.5.0 Preview 2.0 with: <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.4.1" />
<PackageReference Include="MSTest.TestAdapter" Version="3.0.0" />
<PackageReference Include="MSTest.TestFramework" Version="3.0.0" /> with .NET SDK 7.0.101 and targetting
Makes running tests in VS impossible after just a few runs. |
Same problem here. Interestingly the R# Testrunner seems to work. Restarting VS solves the problem for me. But its annoying. |
The error is coming from VS Test Explorer window trying to exploit some new feature enabled in v3 of MSTest. Because this feature is for VS Test Explorer only, it makes sense there is no impact outside of VS. |
Same here. Sometimes I can make only one test run before the error occurs. |
Same problem. It's only happpend in integration tests. Unit tests are working fine. Message: Stack Trace: System.NullReferenceException: Object reference not set to an instance of an object. |
@peterwald @ocallesp Can we have some news on the investigation/fix of this issue? |
I started seeing this error when I updated my nuget packages. MSTest.TestAdapter and MSTest.Framework got up graded to version 3.0.2. I switched them back to version 3.0.0 and the issue went away. |
Here's my workaround. Close your solution and close Visual Studio Enterprise, then kill vbcscompiler.exe process. |
We hear the feedback and are actively working on a fix. It's a bit of a tricky bug/fix and it is taking some time to validate, but we will have a fix shortly. |
Starting from your suggestion, I made this attempt: use default packages when creating a new test project. But it was not enough to go back to version 3.0.0 of Testframework and Testadapter, I had to do it for all the packages used for testing, including fakes. Then clean solution / projects and rebuild. So far the issue has not recurred. here's the package versions, the same of a new test project created with VS 17.4.3: Microsoft.NET.Test.Sdk, Version="17.3.2" |
I've temporarily switched to using the VSTest console instead (https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2022) accessed within VS Enterprise 2022 via:
It's certainly not ideal but better than restarting VS and the debug tools multiple times. Not sure if this will end up with the same issue but reading the above I suspect not as it doesn't use the VS Test dialog at all. Edit: and I can also still debug the failing unit tests as well via Visual Studio, so it's clunky but I can (a) work out what is failing using the Console and (b) debug to find out why using Visual Studio |
@bvbellomo @Dmytro-Hryshyn-89 @KieranDevvs Can you confirm what version of VS you are running? The 17.4.5 version of VS has not shipped yet. So, unless you have upgraded to a preview version of VS, then it is expected that you would have the same issue. |
If the fix hasn't shipped yet, then the issue isn't solved thus the ticket should remain open. |
There are multiple versions that this fix was shipped in. The 17.5 previews have been shipped for a few weeks now and we've had responses above confirming the fix. As I mentioned earlier, it takes a much more involved process to ship servicing updates to earlier releases and thus there will be a delay until the 17.4 line receives this update. |
I imagine the time loss around the world caused by this bug alone. Do you have any idea when the fix will be part of something non-preview? |
Hi @peterwald sorry for delay. I am running Microsoft Visual Studio Professional 2022 (64-bit) - Current |
The 17.4.5 release is now publicly available including this fix. |
Thanks, 17.4.5 fixed this issue for me. |
Any intermediate solution for this? I have this from commandline, without VS...
|
This is not fixed in Visual Studio 2022 Enterprise V17.5.4 I am using NUNIT, not MSTest System.InvalidOperationException: TestsForOrigin: Unknown TestCaseRecord.Origin value Unknown |
@peterwald I will let you handle this. |
@TomKuhn, the original issue was due to an interaction between MSTest and the Visual Studio Test Explorer. You must be seeing a different condition that generates a similar error. Can you help us understand how to reproduce the error with NUnit? |
@peterwald It is very similar, in that seemingly randomly, it will fail to execute any tests with the test explorer displaying many errors (exceptions) matching the above post. Only a restart of visual studio will fix it. Sadly I have had to restart VS about 30 times today :( It does not seem to have anything to do with renaming/removing tests, but repeatedly debugging a test and changing source code while debugging seems to make it happen. Versions: Visual Studio: 17.5.4 Old-style project format: ... ...If there's anything else I can provide, I'm happy to do so Tom |
We do use [TestCaseSource(nameof(LanguagesTestData))] a lot which allows us to dynamically create tests based on executing code, which is an Nunit feature that's quite scary! |
I've been unable to reproduce the problem using data-driven (TestCaseSource) NUnit tests. Is it possible for you to share a skeleton of what your tests look like? Are you using generics in your tests? Unusual or custom test names? What types are the TestCaseSources providing? Any other tips on what operations trigger the error? |
Shall the issue be split? The original once was about MSTest, the new one is about NUnit, even if similar - it would benefit everyone to track it separately. Agreed? |
@abatishchev @TomKuhn I agree it would be better to open a separate issue. Test Explorer related issues (TestStore entry in the exception) are usually better tracked at https://developercommunity.visualstudio.com/home but if it's more convenient for you please do open another issue here and we will tag it with IDE. |
This issue still here on VS 2022 17.6.2 |
I'm seeing these exceptions too in 17.6.3, using XUnit. And I haven't renamed anything; literally just opened VS and tried to run tests. This is a very annoying and disappointing flaw in what is supposed to be a developer productivity product; I've been resisting the switch to Rider that many of my teammates have made, but the fact that JetBrains are at least able to ensure that core functionality doesn't break for no good reason... is becoming more and more difficult to ignore. |
I have opened the following issue on dev community for you to track and follow - please feel free to add comment to it: https://developercommunity.visualstudio.com/t/SystemInvalidOperationException-and-Sys/10411793. Closing this ticket as issue is not on Test Platform side. |
Same issue using (latest version): https://developercommunity.visualstudio.com/t/SystemInvalidOperationException-and-Sys/10411793 is closed.
|
@olivier-spinelli the issue is marked as "under investigation" on dev community website (not as closed). Ping @AbhitejJohn @peterwald |
Glad to hear that... But why, being logged in, can't I see it? (Subscription "Visual Studio Enterprise (MPN)") Viewing as ol***li@signature.one Error details: ["Request failed with status code 403"] |
Oh thanks for the info! I logged in with my personal account and get the same error as you but I can see it correctly with work account. I will check with colleagues what's going-on and will get back to you! |
@olivier-spinelli the issue is now public! I didn't pay attention that I created it as internal. |
Many thanks! |
Description
I am running unit tests on a 2 c# test projects using the lates version of Visual Studio Community 2022
The 2 test project are named EngineTests and ServerCodeTests.
While the test projects compile correctly, I regularly get a series of error messages in the Output / Tests window when running unit tests:
System.InvalidOperationException: TestsForOrigin: Unknown TestCaseRecord.Origin value Unknown
System.Collections.Generic.KeyNotFoundException: The given key 'C:\Users\liamf\Source\Repos\TradingReplay\ServerCodeTests\bin\Debug\net6.0\ServerCodeTests.dll' was not present in the dictionary.
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Could not load type 'Microsoft.VisualStudio.TestTools.UnitTesting.StringEx' from assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
No test is available in C:\Users\liamf\Source\Repos\TradingReplay\ServerCodeTests\bin\Debug\net6.0\ServerCodeTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Steps to reproduce
Running any test from Test Explorer.
Reloading Visual Studio allows you to do 2-3 test runs before the issue arises again.
Diagnostic logs
ErrorLog.text attached
Environment
Windows 11 10.0.22621 Build 22621
Visual Studiio Community v17.4.3
Nuget packages:
MSTest.TesAdapter v3
MSTest.TestFramework v3
Microsoft.NET.Test.SDK v17.4
ErrorLog.txt
The text was updated successfully, but these errors were encountered: