Testing a module requires knowing too many things

[nohwnd]

I counted what you need to know to test a module, by walking the path a person actually takes.

To get one passing test against a public function:

1. get Pester 5/6 loaded and not the Pester 3 that Windows ships
2. *.Tests.ps1 naming so Invoke-Pester finds it
3. import in BeforeAll and not at the top of the file
4. $PSScriptRoot for the path, the working directory is not the test folder
5. -Force, and removing a stale or differently versioned copy
6. Describe / It / Should

To test anything internal, or to mock at all, add:

7. InModuleScope, or -ModuleName on every Mock
8. BeforeAll variables do not flow into InModuleScope, you need -Parameters
9. Should -Invoke with -ParameterFilter, and the filter evaluates in module scope too
10. mocking a command that is not installed on the machine needs a stub function first

To generate tests from the module surface, add:

11. -ForEach is evaluated at discovery, so the module has to be imported in BeforeDiscovery as well, you import it twice
12. $_ and hashtable key expansion rules

To run it in CI, add:

13. configuration object, exit code, Run.Throw
14. TestResult format and output verbosity
15. coverage paths, breakpoints or profiler
16. point all of it at the built module and not at src

6 to say hello, 10 to write a real test, 16 to ship it.

The part that bothers me

Items 3, 7, 8 and 11 are not four problems, they are one problem seen four times. We have two phases, discovery and run, and two scopes, test and module. The user has to hold that 2x2 in their head and put every line of the test file into the right cell. Nothing in the API tells them the grid is there. They find out when they get $null.

How common is the simple case

I checked, because most of the proposals below only work when a repo has one module. 400 repos tagged powershell-module, kept the ones that have Pester test files and at least one .psd1 whose content declares RootModule or ModuleToProcess. n=171.

modules in repo	repos
1	156 (91%)
2	10 (97% cumulative)
3	2
5, 10, 12	1 each

Most of the 2 module ones are not two products, they are one module plus a vendored dependency, a build helper, a template placeholder, or a second manifest for the same module. Genuinely multi module products were 4 of 171.

The content check matters. A first pass that used only path shape counted Pode's src/Locales/*/Pode.psd1 and PSKoans' drafts/ fixtures as modules, and reported 50%.

Proposals

Filed separately so they can be taken or dropped one by one.

• Infer the module under test instead of making people write -ModuleName #2997 infer the module under test, so -ModuleName is not something you have to know about. Validated rule, no new syntax, falls back to today's behaviour whenever it is unsure.
• Bring a function out of the module instead of moving the test into it #2998 bring a function out of the module instead of moving the test into it. Works with no proxy, the session state survives the assignment. Needs Infer the module under test instead of making people write -ModuleName #2997 to be useful, together they are what InModuleScope is doing for people today.
• One mechanism for testing modules and scripts, and no more dot sourcing #2999 one mechanism for modules and scripts, which also gets rid of dot sourcing. Wrap the file in a dynamic module, and for scripts that do things, hoist the top level statements into an entry point first. Verified against 6.1.0.
• Say what went wrong when a module test is written the wrong way #3000 say what went wrong when a test is written the wrong way. Error messages only, no API and no design decision, and probably the cheapest thing on the list.

Separately, #2992 came out of the same investigation, mocks lose parameter defaults, ArgumentCompleter and custom attributes.

@fflaten

🤖