DynamicDataAttribute and DataRowAttribute annotated test methods are being skipped in v3.8

[sramekpete]

Describe the bug

After updating MSTest to version 3.8.0 data-driven tests are being skipped. Update from 3.7.3 to 3.8.0 shouldn't bring any breaking changes.

Steps To Reproduce

1. Create a MSTest project. Update MSTest to v3.7.3.
2.  Create a data-driven test using DynamicDataAttribute or DataRowAttribute.
3. Run tests. None should be skipped.
4. Update MSTest to 3.8.0
5. Run tests. Tests using DynamicDataAttribute or DataRowAttribute are being skipped.

Working example can be found in repo polyline-algorithm-csharp@repro/mtstest-breaking-change-v3.8 and tests folder. There is side by side the same test project, one using 3.7.* and the other 3.8.*.

Expected behavior

Test methods using DynamicDataAttribute or DataRowAttribute are not being skipped after an update to 3.8.0.

Actual behavior

Test methods using DynamicDataAttribute or DataRowAttribute are being skipped after an update to 3.8.0.

Additional context

[Youssef1313]

I'm able to repro. Surprisingly, this works just fine when using Microsoft.Testing.Platform, and only reproduces with VSTest.

[Youssef1313]

@petesramek Can you please confirm if it works correctly with dotnet test for you, and that the issue happens for Test Explorer only?

[kgparmar739]

@Youssef1313 I have similar problem.
It works well with donnet test however it is not working with the VsTest

[sramekpete]

@Youssef1313 It looks like dotnet test command works fine for both versions, 3.7.3 and 3.8.0.

[stnoa]

Same issue. It works again when setting TestDataSourceUnfoldingStrategy.Fold

[Youssef1313]

Thank you for confirming @kgparmar739 and @petesramek.

We will ship a fix in 3.8.1. The issue happens under 3 conditions:

1. Running in Test Explorer
2. Running with VSTest (it's working fine with Microsoft.Testing.Platform)
3. Tests are expanded during discovery (aka unfolded)

[sramekpete]

@Youssef1313 I am not so sure about 3rd condition. I have the test skipped all the time. No matter if tests are or are not expanded during the discovery. Maybe I just don't know how to reproduce it exactly. Visual Studio version info might be relevant. Included below.

Microsoft Visual Studio Community 2022
Version 17.13.0
VisualStudio.17.Release/17.13.0+35806.99
Microsoft .NET Framework
Version 4.8.09032

Installed Version: Community

ASP.NET and Web Tools 17.13.124.35287
ASP.NET and Web Tools

Azure App Service Tools v3.0.0 17.13.124.35287
Azure App Service Tools v3.0.0

Azure Functions and Web Jobs Tools 17.13.124.35287
Azure Functions and Web Jobs Tools

C# Tools 4.13.0-3.25105.4+400efa2d94b217101a0ca61e84a3812fdc2fec52
C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

GitHub Copilot 17.13.430.18484
GitHub Copilot is an AI pair programmer that helps you write code faster and with less work.

ILSpy.AddIn 1.0
Integration of the ILSpy Decompiler into Visual Studio.

Microsoft JVM Debugger 1.0
Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

Node.js Tools 1.5.61125.7 Commit Hash:d387982a7546304d617ed77b6d390e04b6558274
Adds support for developing and debugging Node.js apps in Visual Studio

NuGet Package Manager 6.13.1
NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

Razor (ASP.NET Core) 17.13.3.2507801+90f580c3fa0908ad7d593a4ca7622e78a61aa927
Provides languages services for ASP.NET Core Razor.

SQL Server Data Tools 17.13.3.1
Microsoft SQL Server Data Tools

TypeScript Tools 17.0.31211.2001
TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools 4.13.0-3.25105.4+400efa2d94b217101a0ca61e84a3812fdc2fec52
Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual F# Tools 17.13.0-beta.25056.5+63a09289745da5c256e7baf5f4194a750b1ec878
Microsoft Visual F# Tools

Visual Studio IntelliCode 2.2
AI-assisted development for Visual Studio.

VisualStudio.DeviceLog 1.0
Information about my package

VisualStudio.Mac 1.0
Mac Extension for Visual Studio

VSPackage Extension 1.0
VSPackage Visual Studio Extension Detailed Info

Xamarin 17.13.0.169 (d17.13@5c8deea)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer 17.13.3.4 (remotes/origin/main@59e9def289)
Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin.Android SDK 13.2.2.0 (d17-5/45b0e14)
Xamarin.Android Reference Assemblies and MSBuild support.
Mono: d9a6e87
Java.Interop: xamarin/java.interop/d17-5@149d70fe
SQLite: xamarin/sqlite@68c69d8
Xamarin.Android Tools: xamarin/xamarin-android-tools/d17-5@ca1552d