Reorganize command line interface options/parameters #3182 - #3216
Open
sheddy123 wants to merge 8 commits into
Open
Reorganize command line interface options/parameters #3182#3216sheddy123 wants to merge 8 commits into
sheddy123 wants to merge 8 commits into
Conversation
Added documentation for the new `--profile` console argument, detailing how to select benchmark jobs by Id, use glob patterns, and distinguish it from `--profiler`. Included code and command-line examples, and updated the options summary.
Added tests to BenchmarkSwitcherTest.cs covering --profile usage: matching job IDs, glob patterns, attribute-defined jobs, error messages for unmatched profiles, and combined profile/filter scenarios. Introduced WithTwoNamedJobAttributes for attribute-based job selection tests.
Introduced JobIdFilterTests to validate filtering benchmarks by job ID, including exact, case-insensitive, glob patterns, and attribute-defined jobs. Tests also verify recording of observed and matched job IDs. Added helper benchmark classes for test scenarios.
Added tests to ensure correct handling of --profile and --filter arguments in BenchmarkDotNet config parsing. Verified that JobIdFilter is only added when a profile is specified, profiles are case-insensitive, and profile filters are not unioned with other filters. Improves test coverage for command-line argument parsing.
Introduced JobIdFilter in BenchmarkDotNet.Filters to enable filtering benchmarks using job IDs with glob pattern support. The filter tracks observed and matched job IDs, exposes them for reporting, and applies filtering logic via regular expressions in the Predicate method.
Introduced a Profiles command-line option to CommandLineOptions, enabling users to run only jobs with matching Id(s) using glob patterns. Updated help text to clarify its distinction from the Profiler option and added a usage example for --profile.
Ensure JobIdFilter is added when profiles are specified, so job selection by profile is always combined with other filters. Added comments to clarify filter handling logic.
Added TryPrintWrongProfileInfo to clarify when --profile returns no benchmarks, providing targeted error messages and listing available profiles. Integrated this logic into the filtering process and limited displayed profiles to 40. Added necessary using directive for BenchmarkDotNet.Filters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #3182's
@filzrev based on your suggestions, "I have specified a benchmark profile key from the command line".
A profile is now a job Id, matched with globs, case-insensitively:
dotnet run -c Release -- --filter 'JsonSerialization' --profile debug
dotnet run -c Release -- --filter '*' --profile net8 net9
Ids come from Job.Default.WithId("net8") in code or [SimpleJob(RuntimeMoniker.Net80, id: "net8")].
Job.Id already existed but was only ever used for display, folder names, and ordering, never selection.