From 308ab3555778898a0d885c0b8ab281c59555a2a4 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:58:46 +0000 Subject: [PATCH] chore: refresh upstream skill catalog --- .../optimizing-ef-core-queries/SKILL.md | 172 +++++++-- .../scripts/Compute-CrapScores.ps1 | 225 ++++-------- .../scripts/CoverageData.ps1 | 272 ++++++++++++++ .../scripts/Extract-MethodCoverage.ps1 | 236 +++--------- .../scripts/Find-UntestedSources.cs | 311 ++++++++++++---- .../scripts/find_untested_sources.py | 335 ++++++++++++++---- .../skills/agentic-workflows/SKILL.md | 3 + catalog/Tools/Roslynk/skills/roslynk/SKILL.md | 4 +- .../Roslynk/skills/roslynk/manifest.json | 2 +- .../skills/roslynk/references/tools.md | 33 +- .../astro/packages/astro/package.json | 2 +- .../.github/skills/agentic-workflows/SKILL.md | 3 + .../optimizing-ef-core-queries/SKILL.md | 172 +++++++-- .../scripts/Compute-CrapScores.ps1 | 225 ++++-------- .../scripts/CoverageData.ps1 | 272 ++++++++++++++ .../scripts/Extract-MethodCoverage.ps1 | 236 +++--------- .../scripts/Find-UntestedSources.cs | 311 ++++++++++++---- .../scripts/find_untested_sources.py | 335 ++++++++++++++---- external-sources/upstreams/roslynk/README.md | 8 +- .../upstreams/roslynk/skills/roslynk/SKILL.md | 4 +- .../skills/roslynk/references/tools.md | 33 +- external-sources/vendir.lock.yml | 20 +- 22 files changed, 2150 insertions(+), 1064 deletions(-) create mode 100644 catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/CoverageData.ps1 create mode 100644 external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/CoverageData.ps1 diff --git a/catalog/Libraries/Official-DotNet-Data/skills/optimizing-ef-core-queries/SKILL.md b/catalog/Libraries/Official-DotNet-Data/skills/optimizing-ef-core-queries/SKILL.md index 7892ee5e..84f69884 100644 --- a/catalog/Libraries/Official-DotNet-Data/skills/optimizing-ef-core-queries/SKILL.md +++ b/catalog/Libraries/Official-DotNet-Data/skills/optimizing-ef-core-queries/SKILL.md @@ -1,12 +1,12 @@ --- name: optimizing-ef-core-queries -description: "Optimize and improve the performance of slow Entity Framework Core (EF Core) queries: make them generate less SQL, make fewer database round-trips, and return results faster. Use whenever an EF Core or DbContext query or data-access path is slow or should be made faster — whether or not EF Core owns the database schema. For EF Core, not Dapper or raw ADO.NET." +description: "Review and optimize application data access that uses Entity Framework Core (EF Core). Use for EF Core performance and correctness reviews of services, repositories, imports, and background jobs: slow or memory-heavy queries, excessive round-trips, hidden SaveChanges, premature materialization, dynamic query construction, result-graph tracking, bulk writes, DbContext lifetime or overlapping operations, search semantics, paging, raw SQL, and model/index coverage. Use whether or not EF Core owns the schema. For EF Core, not Dapper or raw ADO.NET." license: MIT --- # Optimizing EF Core Queries -Diagnose and fix slow Entity Framework Core (EF Core) queries. Start from the generated SQL/logs, apply the smallest change that removes the bottleneck, and confirm the fix by re-reading the SQL and the query count. Prefer changes that reduce round-trips, duplicated rows, scans, or per-call translation cost over micro-optimizations. Apply one change at a time and re-measure. +Diagnose and fix slow or incorrect application data access built on EF Core. In a source-only review, trace the code without executing it and separate proven code behavior from runtime assumptions. When SQL, logs, or plans are available, use them to confirm the bottleneck. Prefer changes that reduce round-trips, duplicated rows, scans, or per-call translation cost over micro-optimizations. ## When to Use @@ -21,9 +21,31 @@ Diagnose and fix slow Entity Framework Core (EF Core) queries. Start from the ge - **The code uses Dapper or raw ADO.NET, not EF Core.** Answer the SQL/indexing/query-plan question directly; do not introduce a `DbContext` or recommend `AsNoTracking`, `Include`, `AsSplitQuery`, or other EF Core APIs. -## First: capture the generated SQL +## Source review workflow -You cannot optimize what you cannot see. Turn on command logging and read the SQL and query count before changing anything: +1. Find the EF boundary: `DbContext`, `DbSet`, repository implementation, unit of work, query helper, or model-building code. Follow the implementation rather than assuming what an interface method does. +2. Trace each query to execution or enumeration. Check whether operators bind to `Queryable` or `Enumerable`, including inside helpers; an `IEnumerable` can remain a deferred database sequence. +3. Count database operations and actual transaction boundaries along the calling path, including immediate writes and repeated enumeration. Also track which operations share the same `DbContext` instance +4. For every execution path, independently inspect predicates, expression construction, projection contents, result bounds, ordering, Includes, tracking, SQL parameterization, and model-building code. +5. Preserve the contract: result membership and order, random-page access, generated keys, retries, transaction and partial-failure behavior, concurrency handling, and domain/interceptor side effects. +6. Report prioritized findings with locations, the affected call path, concrete impact, and a safe remedy. Briefly identify sound counterparts and state what was not validated. Do not equate source reasoning with measured plans or speedups. + +For a source-only request, recommend runtime validation as a next step. + +For raw SQL, prefer EF Core's interpolated overloads (`FromSql` / `FromSqlInterpolated` and `ExecuteSql[Async]`) so values are sent as parameters. + +## Gate source-only findings on evidence + +When the user requests a source-only review, distinguish a demonstrated defect from an optimization that still needs workload or deployment evidence: + +- A load-mutate loop followed by one `SaveChanges[Async]` is not `SaveChanges` in a loop. Do not call it a per-row round-trip defect; suggest a bulk API only when the operation is demonstrably large and bypassing tracking semantics is safe. +- Missing `HasIndex` calls in one supplied file do not prove that the deployed database lacks indexes. Use relevant model-building code, migrations, or a query plan as evidence. Do not pad a source review with generic conditional index advice when that evidence is absent. +- Missing `AsNoTracking` is actionable only on proven read-only entity results. Scalar-only projections have nothing to track, but an entity inside a DTO/record is still tracked by default. +- If every reviewed pattern is sound, say that no actionable changes are needed rather than manufacturing a finding. + +## Capture SQL and query counts + +Use supplied logs and plans first. For an executable investigation, enable command logging to verify query shape and count before and after a change: ```csharp optionsBuilder.LogTo(Console.WriteLine, LogLevel.Information); @@ -34,31 +56,50 @@ Tag a query with `.TagWith("...")` to find it in the log. Count how many stateme ## Fixes -### Keep predicates sargable — never wrap an indexed column in a function +### Trace materialization and result bounds + +Materialization inside a helper is still an execution boundary. A list or array has already loaded rows; later reductions run in memory. An `IEnumerable` can instead hide a deferred EF query: its `Enumerable` operators process rows locally, and each enumeration may execute the database query again. Trace implementation and static types, not just method names. Keep translatable filters, projections, and aggregates on `IQueryable` until execution: + +```csharp +// Loads rows, then filters in memory. +var candidates = await repository.LoadCandidatesAsync(cancellationToken); +return candidates.Where(o => o.CreatedAt < cutoff).ToList(); + +// Composes and filters in SQL. +return await repository.QueryCandidates() + .Where(o => o.CreatedAt < cutoff) + .ToListAsync(cancellationToken); +``` + +Use `AnyAsync` for boolean existence, but preserve `CountAsync` when the numeric total is required. Combine repeated full-sequence summary calculations into server-side aggregates rather than caching an unbounded list or parallelizing queries on one context. -An index can only be used when the indexed column appears **bare** on one side of the comparison. Wrapping it in a function or arithmetic — `CreatedAt.Year == y`, `CreatedAt.Date == d`, `ToLower(Name) == n`, `Price * 1.1 > x`, or a leading-wildcard `LIKE '%foo'` — forces a per-row computation the index cannot satisfy, so the query **scans the whole table even though the index exists**. Adding another index changes nothing. Rewrite the predicate so the column stays bare, usually as a half-open range: +Separately review result cardinality. Page or cap interactive results; for complete exports, reduce buffering with streaming/chunks without truncating the contract. + +### Keep predicates compatible with indexes + +Functions/arithmetic on a column, or unanchored substring matching, commonly prevent a seek on an ordinary column index. Prefer a bare-column comparison, often a half-open range. Do not assert a full-table scan from LINQ alone: provider translations, computed/expression indexes, and other predicates can change the plan. ```csharp -// Non-sargable: a function is computed for every row → full scan +// Computes a year expression instead of comparing the indexed date. db.Logs.Where(l => l.CreatedAt.Year == year); -// Sargable: bare column compared to constants → index seek +// Range compatible with an ordinary date index. var start = new DateTime(year, 1, 1); db.Logs.Where(l => l.CreatedAt >= start && l.CreatedAt < start.AddYears(1)); ``` The same rule covers several common shapes: -- **Case-insensitive text** — compare a stored normalized column instead of `ToLower(...)`/`ToUpper(...)`. +- **Case-insensitive text** — use an existing indexed normalized/computed value or a compatible column collation instead of wrapping the column. Query-specific collation changes can also defeat an index with a different collation. Preserve normalization and comparison semantics. `StringComparison` overloads are generally not translated by relational providers; current EF Core throws for an untranslatable predicate rather than silently filtering locally. - **Computed expressions** — compare against the precomputed constant, not `column * k > x`. -- **Converting the column to another type** — a predicate over `column.ToString()` (for example matching the *text form* of a number or date, `total.ToString().StartsWith(p)`) applies a function to every row and often can't be translated to SQL at all, forcing a client-side evaluation that pulls the whole table into memory. Filter on the typed column with a real comparison or range instead. -- **Substring search** — `name.Contains(term)` becomes an unanchored `LIKE '%term%'` that can't seek an index and scans the table; a trailing-wildcard prefix (`name.StartsWith(term)` → `'term%'`) can seek. Anchor the search when a prefix match is acceptable — this changes which rows match, so confirm the behavior first — and put real substring or fuzzy search behind a full-text index on large tables. +- **Converting the column to another type** — a predicate over `column.ToString()` (for example matching the *text form* of a number or date, `total.ToString().StartsWith(p)`) applies a function to every row and may not translate. Current EF Core reports a translation failure rather than silently evaluating an untranslatable predicate on the client. Filter on the typed column with a real comparison or range instead. +- **Substring search** — `name.Contains(term)` is generally not seekable with an ordinary B-tree index; prefix matching may be. Neither prefix matching nor token-based full-text search is a drop-in replacement for arbitrary substring matching. Preserve the required search mode or explicitly disclose the change; consider a provider-appropriate substring index/search design. -**Verify:** the plan shows a seek/index instead of a scan and duration drops. If the column genuinely has no index, add one (see below) — but only after the predicate is sargable. +**Verify:** when runtime validation is requested, inspect the resulting predicate and plan. An additional ordinary index does not by itself repair an incompatible expression. ### Compile hot, frequently-executed queries -On a very hot path that runs the *same* query shape thousands of times over a reused context, EF Core re-parses the LINQ expression tree and probes its query cache on every call. When the query is already minimal (an indexed lookup or a small projection) and read-only tweaks such as `AsNoTracking` buy nothing, that per-call translation is the remaining cost. Compile the query once with `EF.CompileQuery` / `EF.CompileAsyncQuery` and reuse the delegate: +EF Core already caches translation for matching query shapes. On a very hot, otherwise minimal query, expression processing and cache lookup can still matter. `EF.CompileQuery` / `EF.CompileAsyncQuery` reuse a delegate that bypasses that lookup; they do not make database I/O cheaper: ```csharp private static readonly Func GetProduct = @@ -74,6 +115,50 @@ The delegate is `static` (compiled once) and takes the `DbContext` plus each par **Verify:** the hot loop's mean time drops with identical results. +### Control SQL parameterization and database plan reuse + +Distinguish EF's query-translation cache from the database's execution-plan cache. Captured LINQ values normally become SQL parameters, while fixed literals remain constants. Parameters keep SQL text stable across values and usually promote database plan reuse; literals expose specific values to the optimizer and can improve selectivity estimates, but many distinct literal values can fragment the server plan cache. Keep the defaults unless workload evidence supports an override: + +| Override | SQL effect | Performance decision | +|----------|------------|----------------------| +| `EF.Parameter(value)` (EF Core 9+) | Forces a parameter, including for a literal | Use when reusable SQL across values matters; already captured arguments usually need no override | +| `EF.Constant(value)` (EF Core 8.0.2+) | Forces an inline constant, including for a captured value | Reserve for evidenced value-specific plan benefits that outweigh extra SQL variants | + +```csharp +db.Orders.Where(order => order.Status == EF.Parameter("Pending")); +db.Orders.Where(order => order.CustomerId == EF.Constant(customerId)); +``` + +The first query parameterizes a fixed status; the second specializes SQL for each customer ID. Neither is inherently faster. Do not parameterize every structural literal, or inline high-cardinality request values without evidence. Provider behavior, parameter-sensitive plans, data skew, and collection translations affect the tradeoff. + +In EF Core 9, these overrides are processed later than query-shape caching: `EF.Constant` can reuse EF's translation while still generating different SQL literals. It is not equivalent to embedding each request value with `Expression.Constant` in a dynamic tree. Compiled delegates do not fix server plan-cache fragmentation; EF Core 9 also does not support these overrides inside `EF.CompileQuery` / `EF.CompileAsyncQuery`. + +**Verify:** compare generated SQL, server compilation/plan reuse, estimates, and representative latency when execution is available. A source review can identify the override and likely tradeoff, not prove a faster plan. + +### Stabilize dynamically constructed query shapes + +Follow request values into expression builders. `Expression.Constant(requestValue)` inside a comparison can produce a new EF cache entry and different SQL for each value. Represent changing values through correctly typed captured-value expressions instead: + +```csharp +Expression> valueAccess = () => filterValue; +var comparison = Expression.Equal(propertyExpression, valueAccess.Body); +``` + +Creating the entity's `ParameterExpression` does not parameterize its comparison values. Ordinary captured LINQ arguments are already parameterizable; fixed structural constants are not automatically defects. Preserve null/conversion semantics, and canonicalize equivalent filter order only when it is safe. Making the builder static, caching a delegate per value, or compiling every new dynamic expression does not stabilize the shape. Report potential compilation/cache churn, not a proven unbounded leak. + +### Choose tracking for the actual result graph + +| Result and consumer | Decision | +|---------------------|----------| +| Scalar/value-only projection | No entity tracking to remove; do not add `AsNoTracking` as a finding | +| Entity or entity-valued DTO used for later edits | Preserve tracking and its identity semantics | +| Read-only entity graph | Consider no tracking; narrow the projection first if full entities are unnecessary | +| Read-only graph with heavily shared entities or reference-identity consumers | Consider `AsNoTrackingWithIdentityResolution`, or key-based DTO processing | + +Plain `AsNoTracking` does **not** resolve repeated keys to one object. Identity-resolution mode uses a query-local map without attaching results to the context; it costs bookkeeping and does not reduce joined SQL payload or deduplicate across separate queries. + +For filtered Includes, inspect earlier/later loads on the same tracking context. Navigation fixup can add already tracked children that fail the Include predicate; the filtered navigation is also marked loaded. Use a detached projection with its own child filter, an appropriate no-tracking read, or a fresh read context when the subset matters. Do not clear a write tracker with pending work just to clean up a read. + ### Remove N+1 and avoid lazy loading The same `SELECT` repeated once per row (a navigation accessed inside a loop) is an N+1. Load the related data in one round-trip — project the aggregates with `Select`, or eager-load with `Include`: @@ -88,15 +173,17 @@ Prefer projection or `Include` over lazy loading: lazy loading is a leading caus **Verify:** a fixed, small query count regardless of row count. -### Split multiple collection Includes +### Separate join multiplication from wide-row duplication -`Include`ing two or more collection navigations in one query multiplies rows (a Cartesian explosion) and duplicates parent data. Use `AsSplitQuery()` so each collection loads in its own statement; add `OrderBy` on a unique key so rows stitch together: +Sibling collection Includes can multiply each other's rows; a nested collection chain is not the same cross-product, and references do not count as additional collections. Even one collection can repeat a wide parent blob/text column many times. Prefer a narrow projection when the consumer does not need that column or graph. Otherwise consider `AsSplitQuery` or targeted loading: ```csharp db.Blogs.Include(b => b.Posts).Include(b => b.Contributors).AsSplitQuery(); ``` -**Verify:** rows per statement drop sharply and total duration improves. +Splitting adds statements/round-trips and can buffer results. Preserve deliberate `AsSingleQuery` and consistency requirements; stable multi-statement reads may need an appropriate snapshot/serializable transaction. Paged results need unique ordering (especially split pagination before EF Core 10); ordering is not what establishes entity identity. Do not prescribe splitting merely by counting Includes. + +**Verify:** lower payload/row duplication outweighs extra round-trips with the required consistency. ### Filter and paginate; prefer keyset over offset @@ -106,13 +193,20 @@ Constrain large result sets with `Where`, and page with **keyset (seek)** pagina db.Orders.Where(o => o.Id > lastSeenId).OrderBy(o => o.Id).Take(pageSize); ``` -Order by a unique, stable, indexed key (add tie-breakers if the sort column isn't unique). Keyset pages by the *last key seen* rather than a page number, so it changes the method's inputs; when a fixed signature rules out an in-place switch, still flag the deep-offset scan and recommend keyset. +Order by a unique, stable key with suitable index coverage. A descending `(CreatedAt, Id)` cursor requires `CreatedAt < lastDate || (CreatedAt == lastDate && Id < lastId)`, not independent bounds on both fields. Include every ordering key with its direction; IDs need not follow business timestamps. + +Trace ordering through query helpers and execution boundaries: + +- A second `OrderBy`/`OrderByDescending` on an `IQueryable` replaces the earlier primary ordering; use `ThenBy` for a secondary key. +- `OrderBy(...).Skip(...).Take(...).AsEnumerable().OrderBy(...)` first chooses a stable database window and then reorders only that bounded window in memory. That can be intentional and is not an unordered page. +- `Skip`/`Take` without a stable unique order is nondeterministic. Deep offset paging is a separate scaling concern, and keyset pagination is not behavior-preserving for random page-number access. +- Even fully ordered offset pages can shift under concurrent inserts/deletes; adding an order fixes nondeterminism, not cross-request snapshot consistency. **Verify:** page latency stays roughly constant from early to deep pages. ### Add missing indexes -Moving a filter into SQL or making a predicate sargable stops the *client-side* waste, but a `WHERE` or `ORDER BY` on a column with no index still scans the whole table inside the database — and a frequently-run query then re-scans it on every call. So audit index coverage separately from the query rewrite: for each query, check whether its filter and sort columns are backed by an index. Entity keys and foreign keys are indexed by convention, but other columns — status flags, state/enum fields, timestamps, names — usually are **not** unless the model configures it. When a hot query filters or sorts on such an unindexed column, recommend adding an index and say so explicitly, even when the rewritten query already returns the right rows: the index is a separate fix the code change alone doesn't deliver. (If the predicate isn't sargable, fix that first — a new index can't help a scan caused by a function on the column.) +Audit workload-specific coverage separately from moving filters into SQL. Inspect applied model configurations and conventions. Derive each dependent FK's ordered properties from `HasForeignKey`; EF normally supplies an index on that tuple unless an existing index/key covers the ordered prefix and required uniqueness, or the convention was removed. Do not count only `HasIndex` calls or confuse a principal key with the dependent table's coverage. Status, timestamps, and other non-key columns may need explicit coverage. Recommend evidenced gaps for hot queries, not generic indexes in every review. If EF Core owns the schema, add the index in the model and migrate: @@ -123,36 +217,64 @@ modelBuilder.Entity() Then create the migration with `dotnet ef migrations add ...`. **Do not apply it** with `dotnet ef database update` (or any equivalent that writes to the database) without explicit user approval — applying a migration mutates the database, so add the migration, show it to the user, and let them run the update once they've reviewed it. If EF Core does not own the schema, recommend the same index to whoever manages the database. Don't over-index — every index slows writes. +Match equality columns, then range/sort keys. An unconstrained intermediate key can block the desired range/order even if an earlier equality column is covered. When all initial keys use equality, more than one order may work; do not condemn a reversed composite index from relationship metadata alone. A FK index can cover the relationship but not a different date-ordered workload. Consider an additional workload index rather than replacing one that supports another path. Model code proves intended metadata, not current physical indexes; weigh write cost and provider-specific coverage. + **Verify:** the plan uses a seek/index instead of a scan. +### Trace persistence through repositories + +A service loop can hide `SaveChanges` behind a repository, audit journal, or progress hook. One final unit-of-work commit does not undo earlier writes; establish whether an enclosing transaction exists before claiming per-item commits. A save can itself require multiple SQL batches. + +Stage bounded batches and save together when the contract permits. Preserve audit/outbox records, generated keys, relationship fixup, ordering, concurrency, and domain events. Capture IDs from the input-ordered entity references after saving; do not assume numerical ID allocation order. + +For retried feeds, reason about what already committed and what the caller can replay. Preserve or design durable row/feed identity with uniqueness and replay handling, or acknowledged checkpoints. Chunking alone is not idempotency, and an unprotected existence-check-then-insert is not race-safe. + +### Bound change detection in growing imports + +Successfully saved entities remain tracked. With automatic detection enabled, `ChangeTracker.Entries()`, `HasChanges`, and `SaveChanges` inspect tracked state; filtering entries by type does not limit the preceding scan. Repeated per-row inspection of a growing graph can create superlinear CPU work even when saves are chunked. + +Use import-owned progress counters instead of repeated tracker enumeration, and rotate a context or clear completed work only when that context is owned by the batch. Preserve pending changes and references. EF Core `Add` and `AddRange` do **not** automatically call `DetectChanges`; replacing one with the other is not a detection fix. Suppress automatic detection only for a demonstrated redundant scan, restore its previous setting in `finally`, and explicitly detect mutations when required before persistence/auditing. + ### Set-based bulk updates and deletes -Replace a load-mutate-`SaveChanges` loop with `ExecuteUpdateAsync`/`ExecuteDeleteAsync` (EF Core 7+) — one statement, no entities materialized: +For demonstrably large, simple updates/deletes that need no per-entity processing, consider `ExecuteUpdateAsync`/`ExecuteDeleteAsync` (EF Core 7+ and a supporting provider). They execute immediately without materialization: ```csharp await db.Products.Where(p => p.LastSoldDate < cutoff) .ExecuteUpdateAsync(s => s.SetProperty(p => p.IsActive, false)); ``` -These bypass the change tracker and EF-side cascade behavior — apply related changes explicitly. +Check the entire write workflow: + +- Immediate writes do not update loaded entity values or snapshots. Later mutations of the same properties can overwrite the SQL change **even in one sequential request**. Use one coherent tracked save or an explicitly synchronized/untracked set-based path; a transaction alone does not refresh tracked state. +- Configured concurrency tokens are not automatically checked by these APIs. Preserve token predicates, token updates, and affected-row handling when optimistic concurrency is required. +- Each call executes separately; a later `SaveChanges` neither batches nor makes them atomic with audit/outbox writes. Use one save unit or an explicit shared transaction where required. +- Preserve client-side cascades and SaveChanges-based hooks. Command interceptors are different and can still observe the SQL. Do not bypass domain behavior merely to reduce query count. **Verify:** a single `UPDATE`/`DELETE` with a `WHERE` and no preceding `SELECT`. +### Keep DbContext operations non-overlapping + +EF Core does not support overlapping operations on one `DbContext`, including through distinct repository objects constructed with that context. Starting both calls before awaiting either is enough; `Task.WhenAll` does not make sharing safe. Await sequentially, or create/dispose one independent context per concurrent operation. Separate factory-created contexts are safe to overlap. A static or singleton-held context also creates lifetime/thread-safety hazards. + ## Common Pitfalls | Pitfall | Fix | |---------|-----| -| Wrapping an indexed column in `.Year`/`.Date`/`ToLower`/arithmetic | Rewrite to a sargable range/comparison on the bare column | -| Adding an index to fix a scan on a non-sargable predicate | Fix the predicate first; the index can't help until the column is bare | -| Rewriting a filter into SQL but leaving a hot query on an unindexed column | The server-side scan is still a scan — recommend an index on the filter/sort column too | +| Function-wrapped predicate incompatible with an ordinary index | Use a range or an existing indexed normalized/computed value | +| Fixing query composition but ignoring workload index coverage | Match the actual equality/range/order prefix, including convention-created indexes | | Compiling a query that runs only occasionally | Compile only genuinely hot, high-frequency query shapes | -| Lazy loading (proxies / `virtual` navigations) causing N+1 and forced sync I/O | Eager-load (`Include`) or project; keep queries async | -| `ToList()`/`AsEnumerable()` before `Where`/`Select` | Keep the query `IQueryable` so filtering/projection run in SQL | +| Confusing EF translation reuse with database plan reuse | Inspect SQL parameter/literal choices and justify overrides with workload evidence | +| DTO contains entities but is assumed untracked | Inspect the materialized graph, not the outer return type | +| Bulk write followed by tracked arithmetic on stale values | Unify writes and preserve concurrency/atomicity; do not just add a transaction | +| Repeated enumeration of a repository's deferred `IEnumerable` | Compose database aggregates before crossing to local operators | ## References - [Efficient querying — EF Core](https://learn.microsoft.com/en-us/ef/core/performance/efficient-querying) - [Compiled queries — EF Core](https://learn.microsoft.com/en-us/ef/core/performance/advanced-performance-topics#compiled-queries) +- [Force or prevent query parameterization — EF Core 9](https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-9.0/whatsnew#force-or-prevent-query-parameterization) +- [Parameterization overrides inside compiled queries — EF Core 9](https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-9.0/breaking-changes) - [Efficient updating (ExecuteUpdate/ExecuteDelete) — EF Core](https://learn.microsoft.com/en-us/ef/core/performance/efficient-updating) - [Single vs. split queries](https://learn.microsoft.com/en-us/ef/core/querying/single-split-queries) - [Pagination](https://learn.microsoft.com/en-us/ef/core/querying/pagination) diff --git a/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/Compute-CrapScores.ps1 b/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/Compute-CrapScores.ps1 index b0c8d9f3..121b54e4 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/Compute-CrapScores.ps1 +++ b/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/Compute-CrapScores.ps1 @@ -1,165 +1,80 @@ -# Compute-CrapScores.ps1 -# -# Reads a Cobertura XML coverage file and calculates CRAP scores per method. -# Uses Alberto Savoia's original CRAP formula: -# CRAP(m) = comp(m)^2 * (1 - cov(m))^3 + comp(m) -# -# Usage: -# .\Compute-CrapScores.ps1 -CoberturaPath ,,... [-CrapThreshold ] [-TopN ] -# -# Outputs: -# - OVERALL_LINE_COVERAGE: (aggregate line coverage across input files, as percent) -# - OVERALL_BRANCH_COVERAGE: (aggregate branch coverage across input files, as percent) -# - TOTAL_METHODS: -# - FLAGGED_METHODS: -# - HOTSPOTS: (top N by CRAP score) - +<# +.SYNOPSIS +Compute CRAP = complexity^2 * (1 - line coverage)^3 + complexity. +.DESCRIPTION +Missing complexity or executable lines yield unknown, never complexity=1/coverage=0. +Legacy output retains OVERALL_*, TOTAL_METHODS, FLAGGED_METHODS and HOTSPOTS markers +only when all requested measurements are representable. -OutputFormat json-v1 +emits one provenance-bearing JSON document. Diagnostics go to stderr. +Thresholds and hotspot ordering use raw scores; only displayed scores are rounded. +Exit codes: 0 complete, 2 invalid input, 3 incomplete/unrepresentable evidence. +#> param( [Parameter(Mandatory)][string[]]$CoberturaPath, - [int]$CrapThreshold = 30, - [int]$TopN = 10 + [ValidateRange(0,[int]::MaxValue)][int]$CrapThreshold = 30, + [ValidateRange(1,[int]::MaxValue)][int]$TopN = 10, + [ValidateSet('legacy', 'json-v1')][string]$OutputFormat = 'legacy' ) -# Merge methods across all Cobertura files using a stable key (Class|Method|Signature|File). -# Line hits are accumulated so a line is counted as covered if any input coverage file covered it. -$methodMap = @{} -$overallLineRate = 0.0 -$overallBranchRate = 0.0 -$totalLinesCovered = 0 -$totalLinesValid = 0 -$totalBranchesCovered = 0 -$totalBranchesValid = 0 -$fallbackLineRates = [System.Collections.Generic.List[double]]::new() -$fallbackBranchRates = [System.Collections.Generic.List[double]]::new() - -foreach ($filePath in $CoberturaPath) { - if (-not (Test-Path $filePath)) { - Write-Error "Cobertura file not found: $filePath" - exit 2 - } - - try { - [xml]$cobertura = Get-Content $filePath -Encoding UTF8 -ErrorAction Stop - } catch { - Write-Error "Failed to parse Cobertura XML: $filePath. $_" - exit 2 - } - - # Prefer aggregate numerator/denominator attributes when present. - if ($null -ne $cobertura.coverage.'lines-covered' -and $null -ne $cobertura.coverage.'lines-valid') { - $totalLinesCovered += [double]$cobertura.coverage.'lines-covered' - $totalLinesValid += [double]$cobertura.coverage.'lines-valid' - } elseif ($cobertura.coverage.'line-rate') { - $fallbackLineRates.Add([double]$cobertura.coverage.'line-rate') - } - if ($null -ne $cobertura.coverage.'branches-covered' -and $null -ne $cobertura.coverage.'branches-valid') { - $totalBranchesCovered += [double]$cobertura.coverage.'branches-covered' - $totalBranchesValid += [double]$cobertura.coverage.'branches-valid' - } elseif ($cobertura.coverage.'branch-rate') { - $fallbackBranchRates.Add([double]$cobertura.coverage.'branch-rate') - } - - foreach ($package in $cobertura.coverage.packages.package) { - foreach ($class in $package.classes.class) { - $className = $class.name - $fileName = $class.filename - - foreach ($method in $class.methods.method) { - $key = "$className|$($method.name)|$($method.signature)|$fileName" - - # Cyclomatic complexity is stored as an XML attribute in Cobertura format - $complexity = if ($null -ne $method.complexity) { [int]$method.complexity } else { 1 } - if ($complexity -lt 1) { $complexity = 1 } - - if (-not $methodMap.ContainsKey($key)) { - $methodMap[$key] = @{ - Class = $className - Method = $method.name - Signature = $method.signature - File = $fileName - Complexity = $complexity - LineHits = @{} - } - } - - # Accumulate hit counts per line number across files - foreach ($line in $method.lines.line) { - $lineNo = $line.number - $hits = [int]$line.hits - if ($methodMap[$key].LineHits.ContainsKey($lineNo)) { - $methodMap[$key].LineHits[$lineNo] += $hits - } else { - $methodMap[$key].LineHits[$lineNo] = $hits - } - } - } - } +try { + . "$PSScriptRoot\CoverageData.ps1" +} catch { + $diagnostic = @{ code = 'DEPENDENCY_UNAVAILABLE'; severity = 'error'; message = $_.Exception.Message; path = "$PSScriptRoot\CoverageData.ps1" } + [Console]::Error.WriteLine("$($diagnostic.code): $($diagnostic.message)") + if ($OutputFormat -eq 'json-v1') { + @{ + schema_version = '1'; tool = 'Compute-CrapScores'; tool_version = '1.0'; status = 'error' + scope = @{}; inputs = @(); diagnostics = @($diagnostic); data = @{} + } | ConvertTo-Json -Depth 5 } + exit 2 } - -$results = [System.Collections.Generic.List[PSCustomObject]]::new() - -foreach ($entry in $methodMap.Values) { - $totalLines = $entry.LineHits.Count - $coveredLines = ($entry.LineHits.Values | Where-Object { $_ -gt 0 } | Measure-Object).Count - $lineCoverage = if ($totalLines -gt 0) { $coveredLines / $totalLines } else { 0.0 } - - $complexity = $entry.Complexity - - # Alberto Savoia's CRAP formula: comp^2 * (1 - cov)^3 + comp - # The cubic exponent on (1-cov) sharply penalizes low coverage: - # at 0% coverage the risk multiplier is 1.0; at 50% it drops to 0.125. - # Higher scores = more complex AND less covered = riskier to change - $uncovered = 1.0 - $lineCoverage - $crapScore = [Math]::Round(($complexity * $complexity * [Math]::Pow($uncovered, 3)) + $complexity, 2) - - $results.Add([PSCustomObject]@{ - Class = $entry.Class - Method = $entry.Method - Signature = $entry.Signature - File = $entry.File - TotalLines = $totalLines - CoveredLines = $coveredLines - LineCoverage = [Math]::Round($lineCoverage * 100, 1) - Complexity = $complexity - CrapScore = $crapScore - }) +$scope = @{ identity = 'reported-module-file-class-signature'; source_revision = $null; top = $TopN; crap_threshold = $CrapThreshold } +try { + $coverage = Get-CoverageData $CoberturaPath +} catch { + $diagnostics = @([pscustomobject]@{ code = 'INVALID_INPUT'; severity = 'error'; message = $_.Exception.Message }) + Write-CoverageDiagnostics $diagnostics + if ($OutputFormat -eq 'json-v1') { Write-CoverageJson 'Compute-CrapScores' 'error' $scope @() $diagnostics @{} } + exit 2 } - -$hotspots = $results | Sort-Object CrapScore -Descending | Select-Object -First $TopN -$flagged = $results | Where-Object { $_.CrapScore -gt $CrapThreshold } - -if ($totalLinesValid -gt 0) { - $overallLineRate = $totalLinesCovered / $totalLinesValid -} else { - # Fallback approximation when Cobertura aggregate counters and per-file rates are unavailable. - # This uses merged method line totals and may under/over-estimate if Cobertura - # includes executable lines outside method nodes. - $mergedTotalLines = ($results | Measure-Object -Property TotalLines -Sum).Sum - $mergedCoveredLines = ($results | Measure-Object -Property CoveredLines -Sum).Sum - if ($mergedTotalLines -gt 0) { - $overallLineRate = [double]$mergedCoveredLines / [double]$mergedTotalLines - } elseif ($fallbackLineRates.Count -gt 0) { - $overallLineRate = ($fallbackLineRates | Measure-Object -Average).Average - } else { - $overallLineRate = 0.0 - } +foreach ($method in $coverage.Methods) { + $score = if ($null -ne $method.Complexity -and $method.LineCoverageStatus -eq 'exact') { + [math]::Pow($method.Complexity, 2) * [math]::Pow(1 - $method.CoveredLines / [double]$method.TotalLines, 3) + $method.Complexity + } else { $null } + $method | Add-Member -NotePropertyName CrapScore -NotePropertyValue $score } - -if ($totalBranchesValid -gt 0) { - $overallBranchRate = $totalBranchesCovered / $totalBranchesValid -} elseif ($fallbackBranchRates.Count -gt 0) { - $overallBranchRate = ($fallbackBranchRates | Measure-Object -Average).Average -} else { - $overallBranchRate = 0.0 +$ranked = @($coverage.Methods | Where-Object { $null -ne $_.CrapScore } | + Sort-Object -Property @{Expression='CrapScore';Descending=$true}, Class, Method, File, Signature) +$hotspots = @($ranked | Select-Object -First $TopN) +$unknown = @($coverage.Methods | Where-Object { $null -eq $_.CrapScore }).Count +$coverage.Summary.flagged_methods = @($ranked | Where-Object { $_.CrapScore -gt $CrapThreshold }).Count +$coverage.Summary.unknown_crap_methods = $unknown +foreach ($method in $ranked) { + $method.CrapScore = [math]::Round($method.CrapScore, 2) } - -Write-Host "OVERALL_LINE_COVERAGE:$([Math]::Round($overallLineRate * 100, 1))" -Write-Host "OVERALL_BRANCH_COVERAGE:$([Math]::Round($overallBranchRate * 100, 1))" -Write-Host "TOTAL_METHODS:$($results.Count)" -Write-Host "FLAGGED_METHODS:$($flagged.Count)" -if ($hotspots) { - Write-Output "HOTSPOTS:$(@($hotspots) | ConvertTo-Json -Compress)" -} else { - Write-Output "HOTSPOTS:[]" +foreach ($metric in @('line', 'branch')) { + if ($coverage.Summary["overall_${metric}_coverage_status"] -in @('unknown', 'bounded')) { + Add-CoverageDiagnostic $coverage.Diagnostics 'AGGREGATE_UNKNOWN' "Overall $metric coverage cannot be derived from overlapping report counters." '' + } +} +$partial = -not $coverage.Methods.Count -or $unknown -gt 0 -or + $coverage.Summary.overall_line_coverage_status -in @('unknown', 'bounded') -or + $coverage.Summary.overall_branch_coverage_status -in @('unknown', 'bounded') +Write-CoverageDiagnostics $coverage.Diagnostics +if ($OutputFormat -eq 'json-v1') { + Write-CoverageJson 'Compute-CrapScores' $(if ($partial) { 'partial' } else { 'complete' }) $scope $coverage.Inputs $coverage.Diagnostics @{ + methods = $coverage.Methods; hotspots = $hotspots; summary = $coverage.Summary + } +} elseif (-not $partial) { + $line = if ($coverage.Summary.overall_line_coverage_status -eq 'not-applicable') { 0 } else { $coverage.Summary.overall_line_coverage } + $branch = if ($coverage.Summary.overall_branch_coverage_status -eq 'not-applicable') { 0 } else { $coverage.Summary.overall_branch_coverage } + Write-Output "OVERALL_LINE_COVERAGE:$line" + Write-Output "OVERALL_BRANCH_COVERAGE:$branch" + Write-Output "TOTAL_METHODS:$($coverage.Methods.Count)" + Write-Output "FLAGGED_METHODS:$($coverage.Summary.flagged_methods)" + $legacy = @($hotspots | Select-Object Class, Method, Signature, File, TotalLines, CoveredLines, LineCoverage, Complexity, CrapScore) + Write-Output "HOTSPOTS:$(ConvertTo-Json -InputObject $legacy -Compress)" } +if ($partial) { exit 3 } +exit 0 diff --git a/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/CoverageData.ps1 b/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/CoverageData.ps1 new file mode 100644 index 00000000..25a343d1 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/CoverageData.ps1 @@ -0,0 +1,272 @@ +# Shared, report-identity-scoped Cobertura normalization. This does not attest +# that a report was collected from the current checkout. + +function New-CoverageMap { + return [System.Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal) +} + +function Add-CoverageDiagnostic($Diagnostics, [string]$Code, [string]$Message, [string]$Path) { + $Diagnostics.Add([pscustomobject]@{ code = $Code; severity = 'warning'; message = $Message; path = $Path }) +} + +function Read-CoverageInteger([string]$Text, [string]$Name, [long]$Minimum = 0) { + $value = 0L + if (-not [long]::TryParse($Text, [Globalization.NumberStyles]::Integer, + [Globalization.CultureInfo]::InvariantCulture, [ref]$value) -or $value -lt $Minimum) { + throw [FormatException]::new("Invalid $Name value '$Text'.") + } + return $value +} + +function Add-CoverageLine($Map, $Node, [string]$Key, [string]$ReportId) { + $number = Read-CoverageInteger $Node.GetAttribute('number') 'line number' 1 + $hits = Read-CoverageInteger $Node.GetAttribute('hits') 'line hits' + $lineKey = "$Key`n$number" + if (-not $Map.ContainsKey($lineKey)) { + $Map[$lineKey] = @{ + Covered = $false + NonBranch = $false + ConflictingBranches = $false + Branches = [System.Collections.Generic.List[object]]::new() + Observations = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + } + } + $entry = $Map[$lineKey] + $entry.Covered = $entry.Covered -or $hits -gt 0 + if (-not $entry.Observations.Add($ReportId)) { return } + $branch = $Node.GetAttribute('branch') + if ($branch -and $branch -notin @('true', 'false')) { + throw [FormatException]::new("Invalid branch flag '$branch'.") + } + if ($branch -ne 'true') { + $entry.NonBranch = $true + if ($entry.Branches.Count) { $entry.ConflictingBranches = $true } + return + } + if ($entry.NonBranch) { $entry.ConflictingBranches = $true } + $raw = $Node.GetAttribute('condition-coverage') + if ($raw -notmatch '^\s*[\d.]+%\s*\((\d+)/(\d+)\)\s*$') { + $entry.Branches.Add(@{ Total = $null; Covered = $null; Conditions = $null }) + return + } + $covered = Read-CoverageInteger $Matches[1] 'covered branches' + $total = Read-CoverageInteger $Matches[2] 'total branches' 1 + if ($covered -gt $total) { throw [FormatException]::new('Covered branches exceed total branches.') } + + # Cobertura jump conditions describe two outcomes, not which outcome a + # 50% observation took. Keep that ambiguity even when condition ids match. + $conditions = New-CoverageMap + $nodes = @($Node.SelectNodes('conditions/condition')) + $conditionSum = 0 + $usable = $nodes.Count -gt 0 -and $total -eq 2 * $nodes.Count + foreach ($condition in $nodes) { + $id = $condition.GetAttribute('number') + $coverage = $condition.GetAttribute('coverage') + if ($condition.GetAttribute('type') -ne 'jump' -or -not $id -or + $conditions.ContainsKey($id) -or $coverage -notmatch '^(0|50|100)(?:\.0+)?%$') { + $usable = $false + continue + } + $count = [int]$Matches[1] / 50 + $conditions[$id] = $count + $conditionSum += $count + } + if (-not $usable -or $conditionSum -ne $covered) { $conditions = $null } + $entry.Branches.Add(@{ Total = $total; Covered = $covered; Conditions = $conditions }) +} + +function Get-CoverageMeasure($Lines) { + $coveredLines = @($Lines.Values | Where-Object Covered).Count + $linePercent = if ($Lines.Count) { [math]::Round(100.0 * $coveredLines / $Lines.Count, 1) } else { $null } + $lower = 0L + $upper = 0L + $total = 0L + $unknown = $Lines.Count -eq 0 + foreach ($line in $Lines.Values) { + $observations = @($line.Branches) + if (-not $observations.Count) { continue } + if ($line.ConflictingBranches -or @($observations | Where-Object { $null -eq $_.Total }).Count) { + $unknown = $true + continue + } + $totals = @($observations | ForEach-Object { $_.Total } | Sort-Object -Unique) + if ($totals.Count -ne 1 -or $null -eq $totals[0]) { $unknown = $true; continue } + $capacity = $totals[0] + $total += $capacity + $useConditions = $true + $conditionIds = $null + foreach ($observation in $observations) { + if ($null -eq $observation.Conditions) { $useConditions = $false; break } + $ids = @($observation.Conditions.Keys | Sort-Object -CaseSensitive) + $identity = ConvertTo-Json -InputObject $ids -Compress + if ($null -ne $conditionIds -and $conditionIds -cne $identity) { $useConditions = $false; break } + $conditionIds = $identity + } + if ($useConditions) { + foreach ($id in $observations[0].Conditions.Keys) { + $counts = @($observations | ForEach-Object { $_.Conditions[$id] }) + $lower += ($counts | Measure-Object -Maximum).Maximum + $upper += [math]::Min(2, ($counts | Measure-Object -Sum).Sum) + } + } else { + $counts = @($observations | ForEach-Object { $_.Covered }) + $lower += ($counts | Measure-Object -Maximum).Maximum + $upper += [math]::Min($capacity, ($counts | Measure-Object -Sum).Sum) + } + } + $branchStatus = if ($unknown) { 'unknown' } elseif (-not $total) { 'not-applicable' } elseif ($lower -eq $upper) { 'exact' } else { 'bounded' } + return [pscustomobject]@{ + TotalLines = $Lines.Count + CoveredLines = $coveredLines + LineCoverage = $linePercent + LineCoverageStatus = $(if ($Lines.Count) { 'exact' } else { 'unknown' }) + TotalBranches = $(if ($unknown) { $null } else { $total }) + CoveredBranches = $(if ($branchStatus -in @('exact', 'not-applicable')) { [long]$lower } else { $null }) + BranchCoverage = $(if ($branchStatus -eq 'exact') { [math]::Round(100.0 * $lower / $total, 1) } else { $null }) + BranchCoverageStatus = $branchStatus + CoveredBranchesLower = $(if (-not $unknown) { [long]$lower } else { $null }) + CoveredBranchesUpper = $(if (-not $unknown) { [long]$upper } else { $null }) + BranchCoverageLower = $(if (-not $unknown -and $total) { [math]::Round(100.0 * $lower / $total, 1) } else { $null }) + BranchCoverageUpper = $(if (-not $unknown -and $total) { [math]::Round(100.0 * $upper / $total, 1) } else { $null }) + } +} + +function Get-CoverageData([string[]]$Paths) { + if (-not $Paths.Count) { throw [ArgumentException]::new('At least one Cobertura report is required.') } + $methods = New-CoverageMap + $classLines = New-CoverageMap + $methodLines = New-CoverageMap + $inputs = [System.Collections.Generic.List[object]]::new() + $reports = [System.Collections.Generic.List[object]]::new() + $diagnostics = [System.Collections.Generic.List[object]]::new() + $seen = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + $allClassesHaveLines = $true + foreach ($path in ($Paths | Sort-Object -Unique -CaseSensitive)) { + $resolved = (Get-Item -LiteralPath $path -ErrorAction Stop).FullName + $bytes = [IO.File]::ReadAllBytes($resolved) + $hash = [Convert]::ToHexString([Security.Cryptography.SHA256]::HashData($bytes)).ToLowerInvariant() + $inputs.Add([pscustomobject]@{ path = $resolved; sha256 = $hash }) + if (-not $seen.Add($hash)) { continue } + $xml = [xml]::new() + $xml.XmlResolver = $null + $stream = [IO.MemoryStream]::new($bytes, $false) + try { $xml.Load($stream) } finally { $stream.Dispose() } + if ($xml.DocumentElement.Name -ne 'coverage') { throw [FormatException]::new("Not a Cobertura report: $path") } + $coverageRoot = $xml.DocumentElement + $report = @{ line = $null; branch = $null } + foreach ($metric in @('line', 'branch')) { + $prefix = if ($metric -eq 'line') { 'lines' } else { 'branches' } + $valid = $coverageRoot.GetAttribute("$prefix-valid") + $covered = $coverageRoot.GetAttribute("$prefix-covered") + if ($valid -or $covered) { + $denominator = Read-CoverageInteger $valid "$prefix-valid" + $numerator = Read-CoverageInteger $covered "$prefix-covered" + if ($numerator -gt $denominator) { throw [FormatException]::new("$prefix-covered exceeds $prefix-valid.") } + $report[$metric] = @{ Total = $denominator; Covered = $numerator } + } + } + $reports.Add($report) + $reportMethods = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + foreach ($class in $xml.SelectNodes('/coverage/packages/package/classes/class')) { + $filename = $class.GetAttribute('filename').Replace('\', '/') + $className = $class.GetAttribute('name') + $module = $class.ParentNode.ParentNode.GetAttribute('name') + if (-not $filename -or -not $className) { throw [FormatException]::new('Class name and filename are required.') } + $sourceKey = ConvertTo-Json -InputObject @($module, $filename) -Compress + $classNodes = @($class.SelectNodes('lines/line')) + if (-not $classNodes.Count) { $allClassesHaveLines = $false } + foreach ($line in $classNodes) { Add-CoverageLine $classLines $line $sourceKey $hash } + foreach ($method in $class.SelectNodes('methods/method')) { + $name = $method.GetAttribute('name') + if (-not $name) { throw [FormatException]::new('Method name is required.') } + $signature = $method.GetAttribute('signature') + $key = ConvertTo-Json -InputObject @($module, $filename, $className, $name, $signature) -Compress + if (-not $reportMethods.Add($key)) { + throw [FormatException]::new("Duplicate method identity for $className.$name; signatures or source mapping are ambiguous.") + } + if (-not $methods.ContainsKey($key)) { + $methods[$key] = @{ + Module = $module; Class = $className; Method = $name; Signature = $signature; File = $filename + Lines = (New-CoverageMap) + Complexities = [System.Collections.Generic.HashSet[long]]::new() + } + } + $entry = $methods[$key] + $complexity = $method.GetAttribute('complexity') + if ($complexity) { [void]$entry.Complexities.Add((Read-CoverageInteger $complexity 'complexity' 1)) } + foreach ($line in $method.SelectNodes('lines/line')) { + Add-CoverageLine $entry.Lines $line '' $hash + Add-CoverageLine $methodLines $line $sourceKey $hash + } + } + } + } + $result = [System.Collections.Generic.List[object]]::new() + foreach ($entry in $methods.Values) { + $measure = Get-CoverageMeasure $entry.Lines + $complexity = if ($entry.Complexities.Count -eq 1) { @($entry.Complexities)[0] } else { $null } + $item = [ordered]@{ + Class = $entry.Class; Method = $entry.Method; Signature = $entry.Signature; File = $entry.File + Module = $entry.Module; Complexity = $complexity + ComplexityStatus = $(if ($null -ne $complexity) { 'exact' } else { 'unknown' }) + } + foreach ($property in $measure.PSObject.Properties) { $item[$property.Name] = $property.Value } + $item.UncoveredLines = $measure.TotalLines - $measure.CoveredLines + if ($null -eq $complexity) { + Add-CoverageDiagnostic $diagnostics 'COMPLEXITY_UNKNOWN' "Missing or conflicting complexity for $($entry.Class).$($entry.Method)." $entry.File + } + if ($measure.LineCoverageStatus -eq 'unknown') { + Add-CoverageDiagnostic $diagnostics 'METHOD_LINES_UNKNOWN' "No executable-line evidence for $($entry.Class).$($entry.Method)." $entry.File + } + if ($measure.BranchCoverageStatus -in @('unknown', 'bounded')) { + Add-CoverageDiagnostic $diagnostics 'BRANCH_UNION_UNCERTAIN' "Branch outcomes cannot be uniquely merged for $($entry.Class).$($entry.Method)." $entry.File + } + $result.Add([pscustomobject]$item) + } + $aggregate = Get-CoverageMeasure $classLines + $aggregateScope = 'class-lines' + if (-not $allClassesHaveLines -or -not $classLines.Count) { + $aggregateScope = if ($reports.Count -eq 1) { 'report' } else { 'method-lines-only' } + $aggregate = [pscustomobject]@{ LineCoverage = $null; LineCoverageStatus = 'unknown'; BranchCoverage = $null; BranchCoverageStatus = 'unknown' } + if ($reports.Count -eq 1) { + foreach ($metric in @('line', 'branch')) { + $counter = $reports[0][$metric] + $name = if ($metric -eq 'line') { 'LineCoverage' } else { 'BranchCoverage' } + if ($null -ne $counter) { + $aggregate.$name = if ($counter.Total) { [math]::Round(100.0 * $counter.Covered / $counter.Total, 1) } else { $null } + $aggregate."${name}Status" = if ($counter.Total) { 'exact' } else { 'not-applicable' } + } + } + } + } + if (-not $result.Count) { + Add-CoverageDiagnostic $diagnostics 'METHODS_UNAVAILABLE' 'The report has no method-level coverage evidence.' '' + } + return [pscustomobject]@{ + Methods = @($result | Sort-Object -CaseSensitive Module, File, Class, Method, Signature) + Inputs = @($inputs) + Diagnostics = $diagnostics + Summary = [ordered]@{ + overall_line_coverage = $aggregate.LineCoverage + overall_line_coverage_status = $aggregate.LineCoverageStatus + overall_branch_coverage = $aggregate.BranchCoverage + overall_branch_coverage_status = $aggregate.BranchCoverageStatus + coverage_scope = $aggregateScope + method_line_coverage = (Get-CoverageMeasure $methodLines).LineCoverage + total_methods = $result.Count + } + } +} + +function Write-CoverageJson([string]$Tool, [string]$Status, $Scope, $Inputs, $Diagnostics, $Data) { + [ordered]@{ + schema_version = '1'; tool = $Tool; tool_version = '1.0'; status = $Status + scope = $Scope; inputs = @($Inputs); diagnostics = @($Diagnostics); data = $Data + } | ConvertTo-Json -Depth 30 +} + +function Write-CoverageDiagnostics($Diagnostics) { + foreach ($diagnostic in $Diagnostics) { + [Console]::Error.WriteLine("$($diagnostic.code): $($diagnostic.message)") + } +} diff --git a/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/Extract-MethodCoverage.ps1 b/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/Extract-MethodCoverage.ps1 index 999a8273..4e2f68fb 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/Extract-MethodCoverage.ps1 +++ b/catalog/Testing/Official-DotNet-Test/skills/coverage-analysis/scripts/Extract-MethodCoverage.ps1 @@ -1,193 +1,65 @@ -param( - [Parameter(Mandatory=$true)] - [string[]]$CoberturaPath, - - [Parameter(Mandatory=$false)] - [int]$CoverageThreshold = 80, - - [Parameter(Mandatory=$false)] - [int]$BranchThreshold = 70, - - [Parameter(Mandatory=$false)] - [ValidateSet('uncovered', 'below-threshold', 'all')] - [string]$Filter = 'all' -) - <# .SYNOPSIS -Extract method-level coverage from Cobertura XML and output as JSON. - +Extract Cobertura method coverage without inventing missing measurements. .DESCRIPTION -Parses one or more Cobertura code coverage XML files and extracts per-method coverage metrics: -- Method name and class -- Line coverage percentage -- Branch coverage percentage -- Lines covered / total -- Branches covered / total -- Complexity (if available) - -When multiple files are provided, line hits are merged across files so a line is counted -as covered if any test project covered it. - -Filters by coverage status (uncovered, below threshold, or all). -Output is JSON for easy post-processing into tables, CSV, or other formats. - -.PARAMETER CoberturaPath -Path(s) to Cobertura coverage.cobertura.xml file(s). Accepts multiple paths for multi-test-project merging. - -.PARAMETER CoverageThreshold -Minimum acceptable line coverage percentage. Methods below this threshold are flagged (default: 80). - -.PARAMETER BranchThreshold -Minimum acceptable branch coverage percentage for methods that contain branches (default: 70). - -.PARAMETER Filter -Which methods to include: - 'uncovered' - methods with 0% coverage only - 'below-threshold' - methods with line coverage < CoverageThreshold OR branch coverage < BranchThreshold (for methods with branches) - 'all' - all methods (default) - -.EXAMPLE -PS> & .\Extract-MethodCoverage.ps1 -CoberturaPath "coverage.cobertura.xml" -CoverageThreshold 80 -BranchThreshold 70 -Filter uncovered -Outputs a JSON array of uncovered methods. - -.EXAMPLE -PS> & .\Extract-MethodCoverage.ps1 -CoberturaPath @("tests1/coverage.cobertura.xml","tests2/coverage.cobertura.xml") -Merges coverage from multiple test projects and outputs combined method-level metrics. - -.OUTPUTS -Writes JSON array to stdout. -Sets exit code 0 on success, 2 on missing/invalid file. +Legacy output is a JSON array; diagnostics and summaries go to stderr. +Use -OutputFormat json-v1 for provenance, null/unknown metrics and branch bounds. +Merging is scoped to reported module/file/class/signature identities, not an +attestation of source revision. Duplicate report bytes are idempotent. +Exit codes: 0 complete, 2 invalid input, 3 incomplete/unrepresentable evidence. #> +param( + [Parameter(Mandatory=$true)][string[]]$CoberturaPath, + [ValidateRange(0,100)][int]$CoverageThreshold = 80, + [ValidateRange(0,100)][int]$BranchThreshold = 70, + [ValidateSet('uncovered', 'below-threshold', 'all')][string]$Filter = 'all', + [ValidateSet('legacy', 'json-v1')][string]$OutputFormat = 'legacy' +) -foreach ($p in $CoberturaPath) { - if (-not (Test-Path $p)) { - Write-Error "Cobertura file not found: $p" - exit 2 +try { + . "$PSScriptRoot\CoverageData.ps1" +} catch { + $diagnostic = @{ code = 'DEPENDENCY_UNAVAILABLE'; severity = 'error'; message = $_.Exception.Message; path = "$PSScriptRoot\CoverageData.ps1" } + [Console]::Error.WriteLine("$($diagnostic.code): $($diagnostic.message)") + if ($OutputFormat -eq 'json-v1') { + @{ + schema_version = '1'; tool = 'Extract-MethodCoverage'; tool_version = '1.0'; status = 'error' + scope = @{}; inputs = @(); diagnostics = @($diagnostic); data = @{} + } | ConvertTo-Json -Depth 5 } + exit 2 } - -# Merge methods across all Cobertura files using a stable key (Class|Method|Signature|File). -# Line hits and branch data are accumulated so coverage reflects all test projects. -$methodMap = @{} - -foreach ($p in $CoberturaPath) { - try { - [xml]$xml = Get-Content $p -Encoding UTF8 -ErrorAction Stop - } catch { - Write-Error "Failed to parse Cobertura XML: $_" - exit 2 - } - - foreach ($package in $xml.coverage.packages.package) { - foreach ($class in $package.classes.class) { - $className = $class.name - $classFilename = $class.filename - - foreach ($method in $class.methods.method) { - $key = "$className|$($method.name)|$($method.signature)|$classFilename" - - if (-not $methodMap.ContainsKey($key)) { - $complexity = if ($null -ne $method.complexity) { [int]$method.complexity } else { 1 } - if ($complexity -lt 1) { $complexity = 1 } - $methodMap[$key] = @{ - Class = $className - Method = $method.name - Signature = $method.signature - File = $classFilename - Complexity = $complexity - LineHits = @{} - BranchData = @{} - } - } - - # Accumulate line hits across files - foreach ($line in $method.lines.line) { - $lineNo = $line.number - $hits = [int]$line.hits - if ($methodMap[$key].LineHits.ContainsKey($lineNo)) { - $methodMap[$key].LineHits[$lineNo] += $hits - } else { - $methodMap[$key].LineHits[$lineNo] = $hits - } - - # Accumulate branch data - if ($line.branch -eq 'true' -and $line.'condition-coverage') { - if ($line.'condition-coverage' -match '\((\d+)/(\d+)\)') { - $covered = [int]$Matches[1] - $total = [int]$Matches[2] - if ($methodMap[$key].BranchData.ContainsKey($lineNo)) { - # Merge branch coverage across files by accumulating covered branches (capped at total) - $existingCovered = $methodMap[$key].BranchData[$lineNo].Covered - $existingTotal = $methodMap[$key].BranchData[$lineNo].Total - if ($existingTotal -ne $total) { - Write-Warning ("Branch total mismatch for {0} at line {1}: {2} vs {3}" -f $key, $lineNo, $existingTotal, $total) - } - $mergedTotal = [Math]::Max($existingTotal, $total) - $mergedCovered = [Math]::Min($existingCovered + $covered, $mergedTotal) - $methodMap[$key].BranchData[$lineNo] = @{ Covered = $mergedCovered; Total = $mergedTotal } - } else { - $methodMap[$key].BranchData[$lineNo] = @{ Covered = $covered; Total = $total } - } - } - } - } - } - } - } +$scope = @{ identity = 'reported-module-file-class-signature'; source_revision = $null; filter = $Filter } +try { + $coverage = Get-CoverageData $CoberturaPath +} catch { + $diagnostics = @([pscustomobject]@{ code = 'INVALID_INPUT'; severity = 'error'; message = $_.Exception.Message }) + Write-CoverageDiagnostics $diagnostics + if ($OutputFormat -eq 'json-v1') { Write-CoverageJson 'Extract-MethodCoverage' 'error' $scope @() $diagnostics @{} } + exit 2 } - -$methods = [System.Collections.Generic.List[PSCustomObject]]::new() - -foreach ($entry in $methodMap.Values) { - $totalLines = $entry.LineHits.Count - $coveredLineCount = ($entry.LineHits.Values | Where-Object { $_ -gt 0 } | Measure-Object).Count - $lineCoveragePercent = if ($totalLines -gt 0) { [math]::Round(($coveredLineCount / $totalLines) * 100, 1) } else { 0 } - - $branchesTotal = 0 - $branchesCovered = 0 - foreach ($bd in $entry.BranchData.Values) { - $branchesCovered += $bd.Covered - $branchesTotal += $bd.Total - } - $branchCoveragePercent = if ($branchesTotal -gt 0) { [math]::Round(($branchesCovered / $branchesTotal) * 100, 1) } else { 0 } - - # Apply filter - if ($Filter -eq 'uncovered' -and $lineCoveragePercent -gt 0) { continue } - if ($Filter -eq 'below-threshold') { - $lineOk = $lineCoveragePercent -ge $CoverageThreshold - $branchOk = ($branchesTotal -eq 0) -or ($branchCoveragePercent -ge $BranchThreshold) - if ($lineOk -and $branchOk) { continue } +$partial = -not $coverage.Methods.Count -or @($coverage.Methods | Where-Object { + $_.LineCoverageStatus -eq 'unknown' -or $_.BranchCoverageStatus -in @('unknown', 'bounded') -or $_.ComplexityStatus -eq 'unknown' +}).Count -gt 0 +$methods = @($coverage.Methods | Where-Object { + if ($Filter -eq 'all') { return $true } + if ($_.LineCoverageStatus -eq 'unknown' -or $_.BranchCoverageStatus -in @('unknown', 'bounded')) { return $true } + if ($Filter -eq 'uncovered') { return $_.LineCoverage -eq 0 } + return (100.0 * $_.CoveredLines / $_.TotalLines) -lt $CoverageThreshold -or + ($_.TotalBranches -gt 0 -and (100.0 * $_.CoveredBranches / $_.TotalBranches) -lt $BranchThreshold) +} | Sort-Object -Property @{Expression='UncoveredLines';Descending=$true}, @{Expression='LineCoverage';Descending=$false}, Class, Method, File, Signature) +Write-CoverageDiagnostics $coverage.Diagnostics +if ($OutputFormat -eq 'json-v1') { + Write-CoverageJson 'Extract-MethodCoverage' $(if ($partial) { 'partial' } else { 'complete' }) $scope $coverage.Inputs $coverage.Diagnostics @{ + methods = $methods; summary = $coverage.Summary } - - $methods.Add([PSCustomObject]@{ - Class = $entry.Class - Method = $entry.Method - Signature = $entry.Signature - File = $entry.File - Complexity = $entry.Complexity - LineCoverage = $lineCoveragePercent - BranchCoverage = $branchCoveragePercent - CoveredLines = $coveredLineCount - TotalLines = $totalLines - UncoveredLines = ($totalLines - $coveredLineCount) - CoveredBranches = $branchesCovered - TotalBranches = $branchesTotal - }) +} elseif (-not $partial) { + $legacy = @($methods | Select-Object Class, Method, Signature, File, Complexity, LineCoverage, + @{Name='BranchCoverage';Expression={if ($_.BranchCoverageStatus -eq 'not-applicable') { 0 } else { $_.BranchCoverage }}}, + CoveredLines, TotalLines, UncoveredLines, CoveredBranches, TotalBranches) + ConvertTo-Json -InputObject $legacy -Depth 5 + [Console]::Error.WriteLine("METHODS_FILTERED:$($methods.Count)") + [Console]::Error.WriteLine("UNCOVERED_METHODS:$(@($methods | Where-Object { $_.LineCoverageStatus -eq 'exact' -and $_.LineCoverage -eq 0 }).Count)") } -# Sort by uncovered lines descending, then by line coverage ascending -$sorted = $methods | Sort-Object -Property @{Expression='UncoveredLines';Descending=$true}, @{Expression='LineCoverage';Descending=$false}, Class, Method - -# Output as JSON (empty array guard for zero results) -if ($sorted.Count -eq 0) { - Write-Output "[]" -} else { - $json = @($sorted) | ConvertTo-Json - Write-Output $json -} - -# Summary -Write-Host "METHODS_FILTERED:$($methods.Count)" -ForegroundColor Green -$uncovered = $methods | Where-Object { $_.LineCoverage -eq 0 } | Measure-Object | Select-Object -ExpandProperty Count -Write-Host "UNCOVERED_METHODS:$uncovered" -ForegroundColor $(if ($uncovered -gt 0) { 'Yellow' } else { 'Green' }) +if ($partial) { exit 3 } exit 0 diff --git a/catalog/Testing/Official-DotNet-Test/skills/find-untested-sources/scripts/Find-UntestedSources.cs b/catalog/Testing/Official-DotNet-Test/skills/find-untested-sources/scripts/Find-UntestedSources.cs index 9feb7680..e2f28b03 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/find-untested-sources/scripts/Find-UntestedSources.cs +++ b/catalog/Testing/Official-DotNet-Test/skills/find-untested-sources/scripts/Find-UntestedSources.cs @@ -1,4 +1,5 @@ #:package Microsoft.CodeAnalysis.CSharp@4.14.0 +#:property PublishAot=false // Find-UntestedSources.cs // @@ -37,48 +38,63 @@ // } using System.Collections.Concurrent; +using System.Security.Cryptography; +using System.Text; using System.Text.Json; using Microsoft.CodeAnalysis; using Microsoft.CodeAnalysis.CSharp; using Microsoft.CodeAnalysis.CSharp.Syntax; +bool jsonV1 = args.Contains("json-v1", StringComparer.Ordinal); +int topN = int.MaxValue; +bool includeTested = false; if (args.Length < 1) { - Console.Error.WriteLine("usage: dotnet run Find-UntestedSources.cs -- [--top N]"); + Console.Error.WriteLine("usage: dotnet run Find-UntestedSources.cs -- [--top N] [--include-tested] [--output-format legacy|json-v1]"); return 1; } -string root = Path.GetFullPath(args[0]); -if (!Directory.Exists(root)) +for (int i = 1; i < args.Length; i++) { - Console.Error.WriteLine($"Not a directory: {root}"); - return 1; + if (args[i] == "--include-tested") { includeTested = true; continue; } + if (args[i] == "--output-format" && i + 1 < args.Length && args[i + 1] is "legacy" or "json-v1") + { + jsonV1 = args[++i] == "json-v1"; + continue; + } + if ((args[i] == "--top" || args[i] == "--limit-untested") && i + 1 < args.Length && + int.TryParse(args[i + 1], out int n) && n >= 0) + { + topN = n == 0 ? int.MaxValue : n; + i++; + continue; + } + return InputError($"Invalid argument: {args[i]}", jsonV1); } -int topN = int.MaxValue; -for (int i = 1; i < args.Length - 1; i++) +string root; +try { root = Path.GetFullPath(args[0]); } +catch (Exception ex) when (ex is ArgumentException or NotSupportedException or IOException) { - if (args[i] == "--top" && int.TryParse(args[i + 1], out var n)) - { - topN = n; - } + return InputError(ex.Message, jsonV1); } +if (!Directory.Exists(root)) { return InputError($"Not a directory: {root}", jsonV1); } var swTotal = System.Diagnostics.Stopwatch.StartNew(); // 1. Discover .cs files var allCs = new List(); -foreach (var f in EnumerateCsFiles(root)) +foreach (var f in EnumerateSourceFiles(root, "*.cs")) { allCs.Add(f); } Log($"Discovered {allCs.Count} .cs files"); // 2. Classify by walking up to nearest .csproj -var projectForDir = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); -var isTestProject = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); +var projectForDir = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); +var isTestProject = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); -string? ProjectFor(string filePath) +ProjectLookup ProjectFor(string filePath) { var dir = Path.GetDirectoryName(filePath)!; return projectForDir.GetOrAdd(dir, d => @@ -88,35 +104,51 @@ { string[] csprojs; try { csprojs = Directory.GetFiles(cur, "*.csproj"); } - catch { csprojs = Array.Empty(); } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Report("PROJECT_DISCOVERY_FAILED", cur, ex.Message); + return new ProjectLookup(null, false); + } if (csprojs.Length > 0) { - // Sort to keep selection deterministic when multiple .csproj - // files coexist in the same directory. - Array.Sort(csprojs, StringComparer.Ordinal); - return csprojs[0]; + if (csprojs.Length > 1) + { + Report("PROJECT_AMBIGUOUS", cur, "Multiple nearest project files; classification is unresolved."); + return new ProjectLookup(null, false); + } + return new ProjectLookup(csprojs[0], true); } cur = Path.GetDirectoryName(cur); } - return null; + return new ProjectLookup(null, true); }); } -bool IsTest(string filePath) +bool? IsTest(string filePath) { - var proj = ProjectFor(filePath); - if (proj is null) + var lookup = ProjectFor(filePath); + if (!lookup.IsResolved) + { + return null; + } + if (lookup.Path is null) { return false; } - return isTestProject.GetOrAdd(proj, ClassifyProjectAsTest); + return isTestProject.GetOrAdd(lookup.Path, ClassifyProjectAsTest); } var sourceFiles = new List(); var testFiles = new List(); foreach (var f in allCs) { - if (IsTest(f)) + var classification = IsTest(f); + if (classification is null) + { + Report("CLASSIFICATION_UNKNOWN", f, "Nearest project discovery or classification could not be completed."); + continue; + } + if (classification.Value) { testFiles.Add(f); } @@ -134,11 +166,8 @@ bool IsTest(string filePath) var swIndex = System.Diagnostics.Stopwatch.StartNew(); Parallel.ForEach(sourceFiles, file => { - string text; - try { text = File.ReadAllText(file); } - catch { return; } - var tree = CSharpSyntaxTree.ParseText(text); - var rootNode = tree.GetRoot(); + var rootNode = ReadSyntax(file); + if (rootNode is null) { return; } int count = 0; foreach (var node in rootNode.DescendantNodes()) { @@ -176,11 +205,8 @@ bool IsTest(string filePath) var swScan = System.Diagnostics.Stopwatch.StartNew(); Parallel.ForEach(testFiles, file => { - string text; - try { text = File.ReadAllText(file); } - catch { return; } - var tree = CSharpSyntaxTree.ParseText(text); - var rootNode = tree.GetRoot(); + var rootNode = ReadSyntax(file); + if (rootNode is null) { return; } var usings = new HashSet(StringComparer.Ordinal); foreach (var u in rootNode.DescendantNodes().OfType()) @@ -250,7 +276,7 @@ bool IsTest(string filePath) // 5. Build suggested test paths for untested files var untested = sourceToTests - .Where(kv => kv.Value.IsEmpty) + .Where(kv => kv.Value.IsEmpty && !AnalysisState.FailedFiles.ContainsKey(kv.Key)) .Select(kv => kv.Key) .OrderBy(s => s, StringComparer.Ordinal) .ToList(); @@ -260,8 +286,9 @@ bool IsTest(string filePath) string? SuggestTestPath(string srcFile) { - var proj = ProjectFor(srcFile); - if (proj is null) + var lookup = ProjectFor(srcFile); + var proj = lookup.Path; + if (!lookup.IsResolved || proj is null) { return null; } @@ -323,15 +350,80 @@ bool IsTest(string filePath) WriteIndented = true, TypeInfoResolver = new System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver(), }; -Console.WriteLine(JsonSerializer.Serialize(result, opts)); +var diagnostics = AnalysisState.Diagnostics + .OrderBy(d => d.path, StringComparer.Ordinal).ThenBy(d => d.code, StringComparer.Ordinal) + .ThenBy(d => d.message, StringComparer.Ordinal).Distinct().ToArray(); +foreach (var diagnostic in diagnostics) { Console.Error.WriteLine($"{diagnostic.code}: {diagnostic.path}: {diagnostic.message}"); } +if (jsonV1) +{ + var declarations = declsByShort.Values.SelectMany(values => values) + .GroupBy(d => ToRel(root, d.File), StringComparer.Ordinal) + .ToDictionary(g => g.Key, g => g.Select(d => d.ShortName).Distinct(StringComparer.Ordinal).Order(StringComparer.Ordinal).ToArray(), StringComparer.Ordinal); + var pairedTests = pairedReport.Values.SelectMany(tests => tests).ToHashSet(StringComparer.Ordinal); + var orphanTests = testFiles.Where(file => !AnalysisState.FailedFiles.ContainsKey(file) && !pairedTests.Contains(ToRel(root, file))) + .Order(StringComparer.Ordinal).Select(file => new { path = ToRel(root, file), language = "csharp" }).ToArray(); + var unassessed = allCs.Where(AnalysisState.FailedFiles.ContainsKey).Order(StringComparer.Ordinal) + .Select(file => new { path = ToRel(root, file), language = "csharp" }).ToArray(); + var data = new Dictionary + { + ["repo_root"] = root, + ["summary"] = new + { + files = allCs.Count, source_files = sourceFiles.Count, test_files = testFiles.Count, + tested_source_files = pairedReport.Count, untested_source_files = untested.Count, + orphan_test_files = orphanTests.Length, unassessed_files = unassessed.Length, + languages = allCs.Count > 0 ? new[] { "csharp" } : Array.Empty(), + }, + ["untested_sources"] = untestedReport.Select(entry => new + { + path = entry.source, language = "csharp", declaration_count = entry.decl_count, + declarations = declarations.GetValueOrDefault(entry.source, Array.Empty()), + suggested_test_path = entry.suggested_test_path, + }).ToArray(), + ["orphan_tests"] = orphanTests, + ["unassessed_files"] = unassessed, + ["test_regions"] = Array.Empty(), + }; + if (includeTested) + { + data["tested_sources"] = pairedReport.Select(entry => new + { + path = entry.Key, language = "csharp", + declaration_count = declsByFile.GetValueOrDefault(Path.GetFullPath(Path.Combine(root, entry.Key))), + declarations = declarations.GetValueOrDefault(entry.Key, Array.Empty()), + covering_tests = entry.Value, + }).ToArray(); + } + var envelope = new + { + schema_version = "1", tool = "Find-UntestedSources", tool_version = "1.0", + status = diagnostics.Length > 0 ? "partial" : "complete", + scope = new + { + root, source_revision = (string?)null, evidence = "static-reference-pairing", + engine = "Roslyn", engine_version = typeof(CSharpSyntaxTree).Assembly.GetName().Version?.ToString(), + declarations = "C# type and delegate syntax declarations; not behavioral coverage", + languages = new[] { "csharp" }, exclude_symlinks = true, + untested_limit = topN == int.MaxValue ? 0 : topN, truncated = untestedReport.Count < untested.Count, + }, + inputs = AnalysisState.Inputs.OrderBy(entry => entry.Key, StringComparer.Ordinal) + .Select(entry => new { path = ToRel(root, entry.Key), sha256 = entry.Value }).ToArray(), + diagnostics, data, + }; + Console.WriteLine(JsonSerializer.Serialize(envelope, opts)); +} +else if (diagnostics.Length == 0) +{ + Console.WriteLine(JsonSerializer.Serialize(result, opts)); +} Log($"Total time: {swTotal.ElapsedMilliseconds} ms"); -return 0; +return diagnostics.Length > 0 ? 3 : 0; // ============================================================================ // Helpers // ============================================================================ -static IEnumerable EnumerateCsFiles(string root) +static IEnumerable EnumerateSourceFiles(string root, string pattern) { var stack = new Stack(); stack.Push(root); @@ -339,10 +431,15 @@ static IEnumerable EnumerateCsFiles(string root) { var dir = stack.Pop(); IEnumerable subdirs; - try { subdirs = Directory.EnumerateDirectories(dir); } - catch { continue; } + try { subdirs = Directory.GetDirectories(dir); } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Report("WALK_FAILED", dir, ex.Message); + continue; + } foreach (var sub in subdirs) { + if (IsLink(sub)) { continue; } var name = Path.GetFileName(sub); if (name.Length == 0) { @@ -361,11 +458,15 @@ static IEnumerable EnumerateCsFiles(string root) stack.Push(sub); } IEnumerable files; - try { files = Directory.EnumerateFiles(dir, "*.cs"); } - catch { continue; } + try { files = Directory.GetFiles(dir, pattern); } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Report("WALK_FAILED", dir, ex.Message); + continue; + } foreach (var f in files) { - if (IsSkippedFile(f)) + if (IsSkippedFile(f) || IsLink(f)) { continue; } @@ -394,20 +495,14 @@ static bool IsSkippedFile(string path) return false; } -static bool ClassifyProjectAsTest(string csproj) +static bool? ClassifyProjectAsTest(string csproj) { - var name = Path.GetFileNameWithoutExtension(csproj); - string[] testSuffixes = { ".Tests", ".Test", ".UnitTests", ".IntegrationTests", ".E2E", ".EndToEnd", ".Spec", ".Specs" }; - foreach (var suf in testSuffixes) - { - if (name.EndsWith(suf, StringComparison.OrdinalIgnoreCase)) - { - return true; - } - } try { - var content = File.ReadAllText(csproj); + var content = ReadSnapshot(csproj); + var name = Path.GetFileNameWithoutExtension(csproj); + string[] testSuffixes = { ".Tests", ".Test", ".UnitTests", ".IntegrationTests", ".E2E", ".EndToEnd", ".Spec", ".Specs" }; + if (testSuffixes.Any(suffix => name.EndsWith(suffix, StringComparison.OrdinalIgnoreCase))) { return true; } if (content.Contains("Microsoft.NET.Test.Sdk", StringComparison.OrdinalIgnoreCase)) { return true; } if (content.Contains("MSTest.Sdk", StringComparison.OrdinalIgnoreCase)) { return true; } if (content.Contains("Microsoft.Testing.Platform", StringComparison.OrdinalIgnoreCase)) { return true; } @@ -416,8 +511,10 @@ static bool ClassifyProjectAsTest(string csproj) if (content.Contains("NUnit", StringComparison.OrdinalIgnoreCase)) { return true; } if (content.Contains("TUnit", StringComparison.OrdinalIgnoreCase)) { return true; } } - catch + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or DecoderFallbackException) { + Report("PROJECT_READ_FAILED", csproj, ex.Message); + return null; } return false; } @@ -459,28 +556,24 @@ static bool IsUsingPrefix(HashSet usings, string ns) static Dictionary BuildProductionToTestProjectMap(string root) { var map = new Dictionary(StringComparer.OrdinalIgnoreCase); - List allCsproj; - try - { - allCsproj = Directory.EnumerateFiles(root, "*.csproj", SearchOption.AllDirectories).ToList(); - } - catch - { - return map; - } + List allCsproj = EnumerateSourceFiles(root, "*.csproj").ToList(); // Sort so the deterministic "first write wins" rule below selects the // same test project across runs/machines regardless of filesystem // enumeration order. allCsproj.Sort(StringComparer.Ordinal); foreach (var testProj in allCsproj) { - if (!ClassifyProjectAsTest(testProj)) + if (ClassifyProjectAsTest(testProj) != true) { continue; } string content; - try { content = File.ReadAllText(testProj); } - catch { continue; } + try { content = ReadSnapshot(testProj); } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or DecoderFallbackException) + { + Report("PROJECT_READ_FAILED", testProj, ex.Message); + continue; + } var testDir = Path.GetDirectoryName(testProj)!; var matches = System.Text.RegularExpressions.Regex.Matches(content, @" BuildProductionToTestProjectMap(string root) static string ToRel(string root, string path) { - try { return Path.GetRelativePath(root, path).Replace('\\', '/'); } - catch { return path; } + return Path.GetRelativePath(root, path).Replace('\\', '/'); } static void Log(string msg) => Console.Error.WriteLine($"[find-untested-sources] {msg}"); +static int InputError(string message, bool jsonV1) +{ + Console.Error.WriteLine($"INVALID_INPUT: {message}"); + if (jsonV1) + { + Console.WriteLine(JsonSerializer.Serialize(new + { + schema_version = "1", tool = "Find-UntestedSources", tool_version = "1.0", status = "error", + scope = new { }, inputs = Array.Empty(), data = new { }, + diagnostics = new[] { new { code = "INVALID_INPUT", severity = "error", message } }, + })); + } + return jsonV1 ? 2 : 1; +} + +static void Report(string code, string path, string message) +{ + AnalysisState.Diagnostics.Enqueue(new DiagnosticRecord(code, "warning", path, message)); + AnalysisState.FailedFiles[path] = 0; +} + +static bool IsLink(string path) +{ + try { return (File.GetAttributes(path) & FileAttributes.ReparsePoint) != 0; } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Report("STAT_FAILED", path, ex.Message); + return true; + } +} + +static string ReadSnapshot(string path) +{ + byte[] bytes = File.ReadAllBytes(path); + string hash = Convert.ToHexString(SHA256.HashData(bytes)).ToLowerInvariant(); + if (AnalysisState.Inputs.TryGetValue(path, out string? previous) && previous != hash) + { + Report("INPUT_CHANGED", path, "Input changed during analysis."); + } + AnalysisState.Inputs[path] = hash; + using var stream = new MemoryStream(bytes, writable: false); + using var reader = new StreamReader(stream, new UTF8Encoding(false, true), detectEncodingFromByteOrderMarks: true); + return reader.ReadToEnd(); +} + +static SyntaxNode? ReadSyntax(string path) +{ + try + { + var tree = CSharpSyntaxTree.ParseText(ReadSnapshot(path)); + var errors = tree.GetDiagnostics().Where(d => d.Severity == DiagnosticSeverity.Error).Take(3).ToArray(); + if (errors.Length > 0) + { + Report("SYNTAX_ERROR", path, string.Join("; ", errors.Select(d => d.ToString()))); + return null; + } + return tree.GetRoot(); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or DecoderFallbackException) + { + Report("READ_FAILED", path, ex.Message); + return null; + } +} + internal readonly record struct Decl(string ShortName, string Namespace, string File); +internal readonly record struct ProjectLookup(string? Path, bool IsResolved); +internal sealed record DiagnosticRecord(string code, string severity, string path, string message); +internal static class AnalysisState +{ + internal static readonly ConcurrentQueue Diagnostics = new(); + internal static readonly ConcurrentDictionary Inputs = new(StringComparer.Ordinal); + internal static readonly ConcurrentDictionary FailedFiles = new(StringComparer.Ordinal); +} diff --git a/catalog/Testing/Official-DotNet-Test/skills/find-untested-sources/scripts/find_untested_sources.py b/catalog/Testing/Official-DotNet-Test/skills/find-untested-sources/scripts/find_untested_sources.py index 2ffd5e97..083446a2 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/find-untested-sources/scripts/find_untested_sources.py +++ b/catalog/Testing/Official-DotNet-Test/skills/find-untested-sources/scripts/find_untested_sources.py @@ -1,6 +1,6 @@ """Polyglot source-to-test pairing analyzer using tree-sitter. -Given a repo root, identifies which source files are NOT covered by any test +Given a repo root, identifies which source files are NOT referenced by any test file via two complementary heuristics: 1. Identifier overlap: a test file declares (or references) the same name @@ -12,17 +12,20 @@ TypeScript, JavaScript, Go, Java, Rust, C#, Ruby, Kotlin, Swift, PowerShell, C++, ...). -Output: JSON to stdout matching the schema used by the C# `find-untested-sources` -skill, so the same prompt patterns can consume both tools. +Output: legacy engine-specific JSON, or a shared envelope with --output-format +json-v1. Pairing is heuristic static evidence, not behavioral coverage. Dependencies: - pip install tree-sitter-language-pack + pip install tree-sitter-language-pack==1.8.1 """ from __future__ import annotations import argparse +import hashlib +import importlib.metadata import json +import posixpath import re import sys from dataclasses import dataclass, field @@ -33,6 +36,7 @@ from tree_sitter_language_pack import ( ProcessConfig, detect_language_from_path, + get_parser, process, ) except ImportError as e: @@ -41,6 +45,12 @@ "Run: pip install tree-sitter-language-pack", file=sys.stderr, ) + if "json-v1" in sys.argv or "--output-format=json-v1" in sys.argv: + print(json.dumps({ + "schema_version": "1", "tool": "find_untested_sources", "tool_version": "1.0", + "status": "error", "scope": {}, "inputs": [], "data": {}, + "diagnostics": [{"code": "DEPENDENCY_MISSING", "severity": "error", "message": str(e)}], + })) raise SystemExit(2) from e @@ -89,7 +99,7 @@ "venv", "env", ".nuget", - "TestResults", + "testresults", "coverage", ".next", ".nuxt", @@ -211,6 +221,10 @@ class FileInfo: declarations: set[str] = field(default_factory=set) referenced_identifiers: set[str] = field(default_factory=set) imports: list[str] = field(default_factory=list) + is_source: bool = True + complete: bool = True + sha256: str | None = None + test_regions: list[dict] = field(default_factory=list) def __hash__(self) -> int: return id(self) @@ -223,27 +237,42 @@ def should_skip_filename(name: str) -> bool: return any(p.search(name) for p in SKIP_FILENAME_PATTERNS) -def walk_files(root: Path, lang_filter: set[str] | None = None) -> Iterable[Path]: +def diagnostic(diagnostics: list[dict], code: str, path: str, error: object) -> None: + diagnostics.append({"code": code, "severity": "warning", "path": path, "message": str(error)}) + + +def language_family(lang: str) -> str: + return "javascript" if lang in {"typescript", "tsx", "javascript"} else lang + + +def walk_files( + root: Path, lang_filter: set[str] | None = None, diagnostics: list[dict] | None = None +) -> Iterable[Path]: """Yield files under root, pruning common build/vendor directories.""" stack = [root] + if diagnostics is None: + diagnostics = [] while stack: current = stack.pop() try: - children = list(current.iterdir()) - except (PermissionError, OSError): + children = sorted(current.iterdir()) + except OSError as exc: + diagnostic(diagnostics, "WALK_FAILED", str(current), exc) continue for child in children: try: if child.is_symlink(): continue - except OSError: + is_dir, is_file = child.is_dir(), child.is_file() + except OSError as exc: + diagnostic(diagnostics, "STAT_FAILED", str(child), exc) continue - if child.is_dir(): + if is_dir: if child.name.lower() in PRUNE_DIRS: continue stack.append(child) continue - if not child.is_file(): + if not is_file: continue if should_skip_filename(child.name): continue @@ -257,55 +286,125 @@ def walk_files(root: Path, lang_filter: set[str] | None = None) -> Iterable[Path yield child -# Word-character regex used to harvest identifiers from a test file's source -# for the "identifier overlap" pairing strategy. Tree-sitter's `symbols` output -# only gives us declared names; references (`new Foo()`, `Foo.bar()`) won't -# appear, so we fall back to a simple word scan over the file body. -IDENTIFIER_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") +def syntax_nodes(root): + stack = [root] + while stack: + node = stack.pop() + if "comment" in node.kind(): + continue + yield node + stack.extend(node.named_child(i) for i in reversed(range(node.named_child_count()))) -def harvest_identifiers(text: str, lang: str) -> set[str]: - if lang == "powershell": - return set(re.findall(r"[A-Za-z_][A-Za-z0-9-]*", text)) - return set(IDENTIFIER_RE.findall(text)) +def node_text(node, source: bytes) -> str: + return source[node.start_byte():node.end_byte()].decode("utf-8") + + +def harvest_identifiers(text: str, lang: str, root=None, regions=None) -> set[str]: + root = root if root is not None else get_parser(lang).parse(text).root_node() + source = text.encode("utf-8") + kinds = { + "identifier", "type_identifier", "field_identifier", "property_identifier", + "constant", "simple_name", "simple_identifier", "command_name", + } + result = set() + for node in syntax_nodes(root): + if node.kind() not in kinds: + continue + if regions is not None and not any(start <= node.start_byte() < end for start, end in regions): + continue + value = node_text(node, source) + if re.fullmatch(r"[A-Za-z_][A-Za-z0-9_-]*", value): + result.add(value) + return result -def harvest_declarations(text: str, lang: str) -> set[str]: +def harvest_declarations(text: str, lang: str, root=None) -> set[str]: """Recover common declarations when the language-pack symbol view is sparse.""" + if lang not in {"powershell", "cpp"}: + return set() + root = root if root is not None else get_parser(lang).parse(text).root_node() + source = bytearray(text.encode("utf-8")) + stack = [root] + while stack: + node = stack.pop() + if "comment" in node.kind() or "string" in node.kind(): + source[node.start_byte():node.end_byte()] = b" " * (node.end_byte() - node.start_byte()) + continue + stack.extend(node.named_child(i) for i in range(node.named_child_count())) + code = source.decode("utf-8") if lang == "powershell": - return set(re.findall(r"(?im)^\s*function\s+([A-Za-z_][A-Za-z0-9-]*)", text)) + return set(re.findall(r"(?im)^\s*function\s+([A-Za-z_][A-Za-z0-9-]*)", code)) if lang == "cpp": return set( re.findall( r"\b(?:class|struct|enum(?:\s+class)?)\s+([A-Za-z_][A-Za-z0-9_]*)", - text, + code, ) ) return set() -def parse_file(path: Path, root: Path) -> FileInfo | None: +def rust_regions(root, source: bytes) -> tuple[list[tuple[int, int]], list[dict]]: + excluded = [] + tests = [] + for parent in syntax_nodes(root): + attributes = [] + for i in range(parent.named_child_count()): + node = parent.named_child(i) + kind = node.kind() + if kind == "attribute_item": + attributes.append(node_text(node, source)) + continue + if kind in {"mod_item", "function_item"}: + if any(re.search(r"#\[\s*cfg\s*\(\s*test\s*\)\s*\]", attr) for attr in attributes): + excluded.append((node.start_byte(), node.end_byte())) + if kind == "function_item" and any( + re.fullmatch(r"#\[\s*(?:\w+::)*test(?:\s*\([^]]*\))?\s*\]", attr) + for attr in attributes + ): + excluded.append((node.start_byte(), node.end_byte())) + name = node.child_by_field_name("name") + tests.append({ + "name": node_text(name, source), + "start_line": source[:node.start_byte()].count(b"\n") + 1, + "end_line": source[:node.end_byte()].count(b"\n") + 1, + "start_byte": node.start_byte(), "end_byte": node.end_byte(), + "kind": "inline-test", + }) + attributes = [] + return excluded, tests + + +def parse_file(path: Path, root: Path, diagnostics: list[dict] | None = None) -> FileInfo | None: rel_str = path.relative_to(root).as_posix() rel = PurePosixPath(rel_str) lang = detect_language_from_path(str(path)) if lang is None or lang not in SUPPORTED_LANGUAGES: return None - try: - text = path.read_text(encoding="utf-8", errors="replace") - except OSError: - return None - + if diagnostics is None: + diagnostics = [] is_test = is_test_path(rel, lang) - info = FileInfo(path=path, rel=rel, lang=lang, is_test=is_test) + info = FileInfo(path=path, rel=rel, lang=lang, is_test=is_test, is_source=not is_test) + try: + raw = path.read_bytes() + info.sha256 = hashlib.sha256(raw).hexdigest() + text = raw.decode("utf-8-sig") + except (OSError, UnicodeError) as exc: + info.complete = False + diagnostic(diagnostics, "READ_FAILED", rel_str, exc) + return info try: + syntax = get_parser(lang).parse(text).root_node() + if syntax.has_error(): + info.complete = False + diagnostic(diagnostics, "SYNTAX_ERROR", rel_str, "Parser reported missing or erroneous syntax.") cfg = ProcessConfig(language=lang, structure=True, imports=True, symbols=True) result = process(text, cfg) except Exception as exc: - # Emit a diagnostic so users can see when tree-sitter parsing silently - # degrades results (e.g. 0-declaration sources because the parser - # blew up on an unusual construct). Keep it short — one line per file. - print(f"WARN: tree-sitter parse failed for {rel_str} ({lang}): {exc}", file=sys.stderr) + info.complete = False + diagnostic(diagnostics, "PARSE_FAILED", rel_str, exc) return info # Top-level declarations: union of structure + symbols. The two views are @@ -331,7 +430,30 @@ def _kind_str(item: object) -> str: name = getattr(sym, "name", None) if name: info.declarations.add(name) - info.declarations.update(harvest_declarations(text, lang)) + info.declarations.update(harvest_declarations(text, lang, syntax)) + + reference_regions = None + if lang == "rust": + source = text.encode("utf-8") + excluded, info.test_regions = rust_regions(syntax, source) + if raw.startswith(b"\xef\xbb\xbf"): + for region in info.test_regions: + region["start_byte"] += 3 + region["end_byte"] += 3 + info.declarations = set() + for node in syntax_nodes(syntax): + if node.kind() not in {"function_item", "struct_item", "enum_item", "trait_item", "type_item", "const_item", "static_item"}: + continue + if any(start <= node.start_byte() < end for start, end in excluded): + continue + name = node.child_by_field_name("name") + if name is not None: + info.declarations.add(node_text(name, source)) + if info.test_regions: + info.is_test = True + info.is_source = not is_test and bool(info.declarations) + if not is_test: + reference_regions = excluded # Imports: keep the raw `source` field; we'll normalize per language. if getattr(result, "imports", None): @@ -339,11 +461,11 @@ def _kind_str(item: object) -> str: src = getattr(imp, "source", None) if src: info.imports.append(src) + if lang == "rust" and info.is_source and info.is_test: + info.imports.clear() - # For test files only, scan all identifier-like tokens — caller uses these - # to pair with source declarations by name. - if is_test: - info.referenced_identifiers = harvest_identifiers(text, lang) + if info.is_test: + info.referenced_identifiers = harvest_identifiers(text, lang, syntax, reference_regions) return info @@ -367,24 +489,19 @@ def _resolve_relative_js(test_rel: PurePosixPath, target: str) -> set[PurePosixP if target.startswith("/"): joined = PurePosixPath(target.lstrip("/")) else: - joined_str = (test_rel.parent / base).as_posix() - # Collapse any number of "/../" pairs, including chained ones - # like "a/b/../../c" → "c". The previous regex only collapsed a single - # occurrence so chained "../../foo" imports stayed partially - # un-normalized and failed to match indexed paths. - prev = None - while prev != joined_str: - prev = joined_str - joined_str = re.sub(r"(?:^|/)[^/]+/\.\./", "/", joined_str) - joined_str = joined_str.lstrip("/") - joined = PurePosixPath(joined_str) + joined = PurePosixPath(posixpath.normpath((test_rel.parent / base).as_posix())) + if ".." in joined.parts: + return set() # Try various extensions and /index suffix. + extensions = (".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs") + if joined.suffix in extensions: + return {joined} candidates = set() stems = [str(joined)] if str(joined).endswith("/index"): stems.append(str(joined)[: -len("/index")]) for stem in stems: - for ext in (".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs"): + for ext in extensions: candidates.add(PurePosixPath(stem + ext)) candidates.add(PurePosixPath(stem + "/index.ts")) candidates.add(PurePosixPath(stem + "/index.tsx")) @@ -484,7 +601,7 @@ def _extract_go_import_targets(import_text: str) -> set[str]: def _build_indexes(sources: list[FileInfo]) -> dict: """Build lookup indexes used to resolve test imports to source files.""" - by_rel: dict[str, FileInfo] = {s.rel.as_posix().lower(): s for s in sources} + by_rel: dict[str, FileInfo] = {s.rel.as_posix(): s for s in sources} by_decl: dict[str, list[FileInfo]] = {} for s in sources: for d in s.declarations: @@ -495,12 +612,12 @@ def _build_indexes(sources: list[FileInfo]) -> dict: # instead of O(#sources) per import target. by_filename: dict[str, list[FileInfo]] = {} for s in sources: - p = s.rel.as_posix().lower() + p = s.rel.as_posix() parts = p.split("/") for i in range(len(parts)): suffix = "/".join(parts[i:]) by_path_suffix.setdefault(suffix, []).append(s) - by_filename.setdefault(s.rel.name.lower(), []).append(s) + by_filename.setdefault(s.rel.name, []).append(s) return { "by_rel": by_rel, "by_decl": by_decl, @@ -516,7 +633,7 @@ def _resolve_test_imports(test: FileInfo, indexes: dict, lang: str) -> set[FileI by_path_suffix = indexes["by_path_suffix"] def add_candidate(path: PurePosixPath) -> None: - key = path.as_posix().lower() + key = path.as_posix() if key in by_rel: found.add(by_rel[key]) return @@ -543,7 +660,7 @@ def add_candidate(path: PurePosixPath) -> None: if not segs: continue last = segs[-1] - key = (last + ".go").lower() + key = last + ".go" for info in by_filename.get(key, ()): if not info.is_test: found.add(info) @@ -597,10 +714,10 @@ def _resolve_test_by_identifiers( if len(ident) < 4: continue sources = by_decl.get(ident) - if not sources: + if not sources or len(sources) != 1: continue for s in sources: - if s.lang == test.lang: + if language_family(s.lang) == language_family(test.lang): found.add(s) return found @@ -611,8 +728,8 @@ def build_pairings( ) -> tuple[dict[FileInfo, set[FileInfo]], list[FileInfo]]: """Return (source -> covering tests) and the list of orphan test files.""" by_lang_indexes: dict[str, dict] = {} - for lang in SUPPORTED_LANGUAGES: - lang_sources = [s for s in sources if s.lang == lang] + for lang in {language_family(lang) for lang in SUPPORTED_LANGUAGES}: + lang_sources = [s for s in sources if language_family(s.lang) == lang and s.complete] if lang_sources: by_lang_indexes[lang] = _build_indexes(lang_sources) @@ -620,7 +737,9 @@ def build_pairings( orphans: list[FileInfo] = [] for t in tests: - idx = by_lang_indexes.get(t.lang) + if not t.complete: + continue + idx = by_lang_indexes.get(language_family(t.lang)) if idx is None: orphans.append(t) continue @@ -653,7 +772,7 @@ def _test_filename(source: FileInfo) -> str: if lang == "java": return f"{stem}Test.java" if lang == "rust": - return f"{stem}_test.rs" + return "" if lang == "csharp": return f"{stem}Tests.cs" if lang == "ruby": @@ -674,11 +793,15 @@ def _suggest_test_path( sibling_test_dirs: dict[tuple[str, PurePosixPath], PurePosixPath], ) -> str: filename = _test_filename(source) + if source.lang == "rust": + # A sibling *_test.rs is not a Cargo target. Suggest the source file + # for an inline #[cfg(test)] module without creating or editing it. + return source.rel.as_posix() if not filename: return "" - language_family = "typescript" if source.lang in {"typescript", "tsx"} else source.lang - parent = sibling_test_dirs.get((language_family, source.rel.parent), source.rel.parent) + family = language_family(source.lang) + parent = sibling_test_dirs.get((family, source.rel.parent), source.rel.parent) return (parent / filename).as_posix() @@ -693,8 +816,8 @@ def build_output( tested: list[dict] = [] sibling_test_dirs: dict[tuple[str, PurePosixPath], PurePosixPath] = {} for paired_source, covering_tests in source_to_tests.items(): - language_family = "typescript" if paired_source.lang in {"typescript", "tsx"} else paired_source.lang - key = (language_family, paired_source.rel.parent) + family = language_family(paired_source.lang) + key = (family, paired_source.rel.parent) for test in covering_tests: candidate = test.rel.parent current = sibling_test_dirs.get(key) @@ -702,6 +825,8 @@ def build_output( sibling_test_dirs[key] = candidate for s in sources: + if not s.complete: + continue covering = sorted(source_to_tests.get(s, set()), key=lambda t: t.rel.as_posix()) entry = { "path": s.rel.as_posix(), @@ -714,23 +839,37 @@ def build_output( tested.append(entry) else: entry["suggested_test_path"] = _suggest_test_path(s, sibling_test_dirs) + if s.lang == "rust": + entry["suggestion_kind"] = "inline-test-module" untested.append(entry) return { "repo_root": str(repo_root), "summary": { + "files": len({s.rel for s in sources + tests}), "source_files": len(sources), "test_files": len(tests), "tested_source_files": len(tested), "untested_source_files": len(untested), "orphan_test_files": len(orphans), "languages": sorted({s.lang for s in sources} | {t.lang for t in tests}), + "unassessed_files": len({s.rel for s in sources + tests if not s.complete}), }, "untested_sources": sorted(untested, key=lambda e: (-e["declaration_count"], e["path"])), "tested_sources": sorted(tested, key=lambda e: e["path"]), "orphan_tests": [ {"path": t.rel.as_posix(), "language": t.lang} for t in sorted(orphans, key=lambda t: t.rel.as_posix()) ], + "test_regions": [ + {"path": t.rel.as_posix(), **region} + for t in sorted(tests, key=lambda t: t.rel.as_posix()) + for region in t.test_regions + ], + "unassessed_files": [ + {"path": info.rel.as_posix(), "language": info.lang} + for info in sorted(set(sources + tests), key=lambda info: info.rel.as_posix()) + if not info.complete + ], } @@ -751,6 +890,10 @@ def main() -> int: default=0, help="If > 0, truncate the untested_sources list to N entries.", ) + parser.add_argument( + "--output-format", choices=("legacy", "json-v1"), default="legacy", + help="Use json-v1 for versioned provenance and explicit partial results.", + ) parser.add_argument( "--include-tested", action="store_true", @@ -761,22 +904,33 @@ def main() -> int: root = args.root.resolve() if not root.is_dir(): print(f"ERROR: not a directory: {root}", file=sys.stderr) + if args.output_format == "json-v1": + print(json.dumps(error_report("INVALID_INPUT", f"Not a directory: {root}"))) + return 2 return 1 + if args.limit_untested < 0: + if args.output_format == "json-v1": + message = "--limit-untested must be nonnegative" + print(f"INVALID_INPUT: {message}", file=sys.stderr) + print(json.dumps(error_report("INVALID_INPUT", message))) + return 2 + parser.error("--limit-untested must be nonnegative") lang_filter = set(args.lang) if args.lang else None print(f"Scanning {root}...", file=sys.stderr) sources: list[FileInfo] = [] tests: list[FileInfo] = [] + diagnostics: list[dict] = [] parsed_count = 0 - for path in walk_files(root, lang_filter=lang_filter): - info = parse_file(path, root) + for path in walk_files(root, lang_filter=lang_filter, diagnostics=diagnostics): + info = parse_file(path, root, diagnostics) if info is None: continue parsed_count += 1 if info.is_test: tests.append(info) - else: + if info.is_source: sources.append(info) print(f"Parsed {parsed_count} files: {len(sources)} source, {len(tests)} test.", file=sys.stderr) @@ -788,10 +942,49 @@ def main() -> int: if not args.include_tested: output.pop("tested_sources", None) + for item in diagnostics: + print(f"{item['code']}: {item['path']}: {item['message']}", file=sys.stderr) + if args.output_format == "json-v1": + files = sorted(set(sources + tests), key=lambda item: item.rel.as_posix()) + output = { + "schema_version": "1", "tool": "find_untested_sources", "tool_version": "1.0", + "status": "partial" if diagnostics else "complete", + "scope": { + "root": str(root), "source_revision": None, "evidence": "static-reference-pairing", + "engine": "tree-sitter-language-pack", + "engine_version": importlib.metadata.version("tree-sitter-language-pack"), + "declarations": "syntax declarations; language-dependent, not a behavioral metric", + "languages": sorted(lang_filter or SUPPORTED_LANGUAGES), + "excluded_directories": sorted(PRUNE_DIRS), "exclude_symlinks": True, + "untested_limit": args.limit_untested, + "truncated": len(output["untested_sources"]) < output["summary"]["untested_source_files"], + }, + "inputs": [{"path": item.rel.as_posix(), "sha256": item.sha256} for item in files], + "diagnostics": diagnostics, "data": output, + } + elif diagnostics: + return 3 json.dump(output, sys.stdout, indent=2) sys.stdout.write("\n") - return 0 + return 3 if diagnostics else 0 + + +def error_report(code: str, message: str) -> dict: + return { + "schema_version": "1", "tool": "find_untested_sources", "tool_version": "1.0", + "status": "error", "scope": {}, "inputs": [], "data": {}, + "diagnostics": [{"code": code, "severity": "error", "message": message}], + } if __name__ == "__main__": - sys.exit(main()) + try: + sys.exit(main()) + except KeyboardInterrupt: + print("CANCELLED: analysis interrupted", file=sys.stderr) + sys.exit(130) + except Exception as exc: + print(f"TOOL_ERROR: {exc}", file=sys.stderr) + if "json-v1" in sys.argv or "--output-format=json-v1" in sys.argv: + print(json.dumps(error_report("TOOL_ERROR", str(exc)))) + sys.exit(2) diff --git a/catalog/Tools/Official-DotNet-Agentic-Workflows/skills/agentic-workflows/SKILL.md b/catalog/Tools/Official-DotNet-Agentic-Workflows/skills/agentic-workflows/SKILL.md index 4c983cc5..a70ea478 100644 --- a/catalog/Tools/Official-DotNet-Agentic-Workflows/skills/agentic-workflows/SKILL.md +++ b/catalog/Tools/Official-DotNet-Agentic-Workflows/skills/agentic-workflows/SKILL.md @@ -31,6 +31,7 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/create-agentic-workflow.md` - `.github/aw/create-shared-agentic-workflow.md` - `.github/aw/debug-agentic-workflow.md` +- `.github/aw/debug-security-review.md` - `.github/aw/dependabot.md` - `.github/aw/deployment-status.md` - `.github/aw/designer-mappings.md` @@ -46,6 +47,7 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/instructions.md` - `.github/aw/intent.md` - `.github/aw/jobs.md` +- `.github/aw/ledger.md` - `.github/aw/linter-workflows.md` - `.github/aw/llms.md` - `.github/aw/loop.md` @@ -87,6 +89,7 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/update-agentic-workflow.md` - `.github/aw/upgrade-agentic-workflows.md` - `.github/aw/visual-regression.md` +- `.github/aw/work-queue.md` - `.github/aw/workflow-constraints.md` - `.github/aw/workflow-editing.md` - `.github/aw/workflow-patterns.md` diff --git a/catalog/Tools/Roslynk/skills/roslynk/SKILL.md b/catalog/Tools/Roslynk/skills/roslynk/SKILL.md index 8759a279..b5689822 100644 --- a/catalog/Tools/Roslynk/skills/roslynk/SKILL.md +++ b/catalog/Tools/Roslynk/skills/roslynk/SKILL.md @@ -32,7 +32,7 @@ Each tool's exact contract — parameters, output format, limits, error codes | Know what an expression means before editing it (type, overload chosen, nullability, constant, conversion) | `get_expression_info` | `var`, overloads, implicit conversions and nullable flow are invisible in the text | | Find implementations of an interface/abstract member | `find_implementations` | compiler's type graph | | See a type's members (and their local functions) / what a name refers to | `get_members` / `get_symbol` | compiler's view; correct across partial classes | -| Read a member's implementation (incl. a partial method's implementation part, and members only a generator declares) | `get_symbol_body` | returns the declaration verbatim, paged when long (`truncated=Y` → re-call with `startLine=`; a partial type's `omitted=Y` part is fetched with `part=`); a path marked `generated=Y` is virtual - read it with this tool, not from disk | +| Read a member's implementation (incl. a partial method's implementation part, and members only a generator declares) | `get_symbol_body` | returns the declaration verbatim, paged when long (`truncated=Y` → re-call with `startItem=`; a partial type's `omitted=Y` part is fetched with `part=`); a path marked `generated=Y` is virtual - read it with this tool, not from disk | | Explore base/derived types | `get_type_hierarchy` | includes referenced-assembly base types | | Find a symbol by partial name | `search_symbols` | compiler-declared symbols, generated code included | | Rename a symbol everywhere (incl. `.razor`/`.cshtml`) | `rename_symbol` | find-and-replace misses markup and same-named text; a symbol only a source generator declares is refused (`error=NotSupported`) - change the generator's input instead | @@ -50,7 +50,7 @@ All name arguments are fully-qualified (`Namespace.Type` or `Namespace.Type.Memb ## Combine tools, don't chain calls -When you need several facts before making a change, send them as **one `multi_query`** call instead of sequential calls: every operation runs against a single snapshot, so the facts can never disagree with each other. `multi_query` accepts only the read-only query tools, and each operation uses exactly that tool's own parameter names — a wrong key is `error=Invalid` naming the key, never silently ignored. The whole response stays within the server's response budget (about 80,000 characters by default; set `Roslynk:MaxResponseChars` to change it). Each operation gets what earlier slots left: `get_symbol_body` pages itself within its share (`truncated=Y`/`nextStartLine=` in its slot), and any other tool's oversized output is cut after its last whole line — its meta line gains `truncated=Y` and its body ends with a bracketed `[response budget exhausted: ...]` note naming the continuation. **Trust rule:** a slot whose *meta line* carries `truncated=Y` was cut by the batch budget, so its body is a prefix — do not derive tool-level paging (e.g. `nextStartLine`) from it; re-send that operation alone. Remaining slots are whole `error=Truncated` blocks with `truncatedSlots=`: re-send exactly those operations, passing the response's `snapshot=` value as `expectSnapshot`, so a continuation that would straddle a solution change is refused with `error=Stale` instead of mixing states. The envelope format is in the multi_query section of [references/tools.md](references/tools.md). +When you need several facts before making a change, send them as **one `multi_query`** call instead of sequential calls: every operation runs against a single snapshot, so the facts can never disagree with each other. `multi_query` accepts only the read-only query tools, and each operation uses exactly that tool's own parameter names — a wrong key is `error=Invalid` naming the key, never silently ignored. Every operation runs; results are then delivered whole, smallest first, until the response budget (about 80,000 characters) is filled, and operations whose result did not fit are named once in the header as `retry=:` (size = that result's cost as a ratio of the budget, rounded up to 0.1). Plan the continuation from the ratios: size ≥ 1.0 → re-send that operation alone (it pages with `nextStartItem=`); sizes summing to ≤ 1.0 → one continuation batch together; pass the response's `snapshot=` value as `expectSnapshot` so a continuation that would straddle a solution change is refused with `error=Stale` instead of mixing states. Delivered slots are always complete, so their own headers can be trusted. The envelope format is in the multi_query section of [references/tools.md](references/tools.md). **Impact analysis** — before renaming a symbol or changing a signature, and whenever someone asks "what uses X", "who calls X", "what breaks if I change X": batch `get_symbol` + `find_references` + `get_callers` + `get_callees` + `find_implementations` + `get_type_hierarchy` in one call, pointing each at the member or type you're changing (`get_callers` takes `methodName`, `get_callees` takes `memberName`, `get_type_hierarchy` takes `typeName`, the others `symbolName`). `get_callers` and `get_callees` are inverses — who calls this member, what does this member call — so alternating them walks a call chain one hop per query, including calls the compiler inserts (`foreach`/`await` plumbing, `using` disposal, operators and conversions); a caller inside an accessor is reported as its property, while `get_callees` names the accessor (`get_X`), so pass the property name back. Then follow each reference with `get_symbol_body` to read the calling convention before editing. Answer with Roslynk, never with grep over `*.cs`/`*.cshtml`/`*.razor` — text search both misses real usages and wrongly matches identically-named symbols. diff --git a/catalog/Tools/Roslynk/skills/roslynk/manifest.json b/catalog/Tools/Roslynk/skills/roslynk/manifest.json index 89fe1e58..e5654dd6 100644 --- a/catalog/Tools/Roslynk/skills/roslynk/manifest.json +++ b/catalog/Tools/Roslynk/skills/roslynk/manifest.json @@ -1,5 +1,5 @@ { - "version": "1.0.0+source.f4e02012aaae4c30", + "version": "1.0.0+source.28965e29d20f3192", "category": "Quality", "compatibility": "Requires a connected Roslynk MCP server and a .NET solution. Follow repository build and approval requirements.", "packages": [ diff --git a/catalog/Tools/Roslynk/skills/roslynk/references/tools.md b/catalog/Tools/Roslynk/skills/roslynk/references/tools.md index 1acb54e0..687ab44a 100644 --- a/catalog/Tools/Roslynk/skills/roslynk/references/tools.md +++ b/catalog/Tools/Roslynk/skills/roslynk/references/tools.md @@ -63,7 +63,7 @@ container's parameter list, e.g. `N.Parent.Widget.Method1(int).localMethod(strin that form is what an `Ambiguous` candidate looks like. A bare name such as `inner` also resolves. Outlines report a local function's kind as `localfunction` and nest it under its containing member. -**Truncation.** Known-total tools emit `count=` and `truncated=Y` when capped; unknown-total scans (`find_dead_code`) emit only `truncated=Y`. Line-paged output (`get_symbol_body`) emits `truncated=Y`, `totalLines=` and `nextStartLine=` instead: re-call with `startLine=` to continue. Any result that would still exceed the server-wide response budget is cut after its last whole line and starts with `outputTruncated=Y` and `fullOutputChars=` — narrow the request (a filter, `maxResults` or a paging parameter) to see the rest. The budget defaults to about 80,000 characters and is set with `Roslynk:MaxResponseChars` (or the `Roslynk__MaxResponseChars` environment variable) in the daemon's host; the daemon reads it once at startup, so changing it requires a restart, and a shared daemon serves every client with one value (pick the smallest cap your clients need). +**Paging and truncation.** Tools whose result can be too large page by item: an oversized result carries `count=`, `truncated=Y` and `nextStartItem=` — call again with `startItem=` and the same other arguments for the next page. An item is one leaf record (a reference, caller, callee, implementor, diagnostic, member, candidate, search hit; for get_symbol_body one source line). Every page repeats the container lines open above its first item, so it reads standalone; the preamble counts against `maxSize` but not against `count=`. `maxSize` (characters; default the response budget, about 80,000 — set with `Roslynk:MaxResponseChars` or the `Roslynk__MaxResponseChars` environment variable, read once at daemon start) bounds a page; an item longer than a whole page is delivered as far as it fits and named `partialItem=` — the rest of that item is not retrievable. `startItem` past the end, or a `maxSize` too small to carry one item beside its container lines, is `error=Invalid`. `maxResults` bounds the result SET (a scan halts at the cap); paging walks that capped set and `count=` names its size — omit `maxResults` and page when you want everything. A result that fits whole carries no paging headers at all. The tools that cannot page (get_symbol, find_definition, get_expression_info, get_solution_status) cut an oversized result after its last whole line, starting with `outputTruncated=Y` and `fullOutputChars=`. **Defaults.** Don't pass a value for a defaulted parameter unless you need the non-default behavior. @@ -128,9 +128,9 @@ Output is header only, in this order (optional keys only when applicable): `solutionId`, `symbolName` (FQN, any symbol kind including members). Unambiguous source match → `#project`/`#path`/`#loc` headers + body containing the verbatim declaration text cut before the body (brace/`=>` excluded). Metadata symbol → `#source=metadata`, `#kind`, `#signature`, `#assembly`. Ambiguous → `error=Ambiguous` with one `candidate=` per match. Preferred over reading a file to identify a symbol. ### get_symbol_body -`solutionId`, `symbolName` (FQN, any declared symbol kind), `includeLeadingTrivia` (default false), `startLine` (default 1), `maxLines` (default 400), `part` (default 0). Returns the symbol's source text including its body, verbatim — original indentation, line endings and spacing, never reformatted. Headers `project`/`path`/`loc`, a blank line, then the text. `includeLeadingTrivia=true` extends the text (and the `loc`) back over the declaration's comments, XML docs and directives. A partial type, or a partial method/property/event, returns `parts=` and one `part=,project=...,path=...,loc=...,lines=` line per block; a partial member's parts are its defining declaration then its implementation, including one supplied by a source generator (`[GeneratedRegex]`, `[LoggerMessage]`, `[LibraryImport]`), whose part line ends `,generated=Y,generator=`. +`solutionId`, `symbolName` (FQN, any declared symbol kind), `includeLeadingTrivia` (default false), `startItem` (default 1), `maxSize` (characters; default the response budget), `part` (default 0). Returns the symbol's source text including its body, verbatim — original indentation, line endings and spacing, never reformatted. Headers `project`/`path`/`loc`, a blank line, then the text. `includeLeadingTrivia=true` extends the text (and the `loc`) back over the declaration's comments, XML docs and directives. A partial type, or a partial method/property/event, returns `parts=` and one `part=,project=...,path=...,loc=...,lines=` line per block; a partial member's parts are its defining declaration then its implementation, including one supplied by a source generator (`[GeneratedRegex]`, `[LoggerMessage]`, `[LibraryImport]`), whose part line ends `,generated=Y,generator=`. -Long text is paged, never cut silently: a call returns at most `maxLines` lines and stays within the response budget. Lines are counted from 1 at the first line of the returned text (its first comment line with `includeLeadingTrivia`), not in the file; for a Razor member that text is the generated C#, while `loc` stays the `#line`-mapped display span — the two coordinates are deliberately unrelated. A page that is not the whole text adds `truncated=Y`, `totalLines=` and `nextStartLine=`: re-call with `startLine=`, keeping `includeLeadingTrivia` and `part` unchanged; a `startLine` past the end is `error=Invalid` naming `totalLines`. There is no unlimited mode (the budget is the ceiling); a single line longer than a whole page's budget is delivered as far as it fits and named `partialLine=` — the rest of that line is not retrievable by paging. When a partial type's parts do not all fit, the result is the whole-or-omitted overview: `parts=`, `truncated=Y`, `omittedParts=`, each part that fits shown whole, the rest listed with their `lines=` count and `,omitted=Y` — fetch one with `part=`, which returns that part alone, paged as above with `parts=`/`part=` headers. `startLine` pages one part; combining it with several parts and no `part=` is `error=Invalid`. +Long text is paged, never cut silently: a call returns at most `maxSize` characters and stays within the response budget. Items are the text's lines, counted from 1 at the first line of the returned text (its first comment line with `includeLeadingTrivia`), not in the file; for a Razor member that text is the generated C#, while `loc` stays the `#line`-mapped display span — the two coordinates are deliberately unrelated. A page that is not the whole text adds `truncated=Y`, `count=` and `nextStartItem=`: re-call with `startItem=`, keeping `includeLeadingTrivia` and `part` unchanged; a `startItem` past the end is `error=Invalid` naming `count`. There is no unlimited mode (the budget is the ceiling). A single line longer than a whole page's budget is delivered as far as it fits and named `partialItem=` — the rest of that line is not retrievable by paging. When a partial type's parts do not all fit, the result is the whole-or-omitted overview: `parts=`, `truncated=Y`, `omittedParts=`, each part that fits shown whole, the rest listed with their `lines=` count and `,omitted=Y` — fetch one with `part=`, which returns that part alone, paged as above with `parts=`/`part=` headers. `startItem` pages one part; combining it with several parts and no `part=` is `error=Invalid`. Overloads (or any name matching several distinct symbols) — `error=Ambiguous`, resolved by retrying with a candidate's parameter-type list; a metadata/referenced-assembly symbol — `error=NotSupported` (use `get_symbol` for its signature), but a metadata spelling (``Ns.Box`1``, `N.Outer+Inner`) of a type the solution declares in source returns that source. Prefer this over reading or grepping the file to see what a member does. @@ -138,15 +138,15 @@ Overloads (or any name matching several distinct symbols) — `error=Ambiguous`, `solutionId`, `typeName` (FQN), `includeInherited` (default false), `nameFilter` (default null; trailing `*` = prefix match, otherwise case-insensitive substring), `includeMethods`/`includeFields`/`includeProperties`/`includeEvents`/`includeNestedTypes` (all default true). Lists all members including private, grouped by declaring file (`` bucket for referenced-assembly types). Member lines: `kind,name,loc[,paramType|paramType|...]` (param list only for methods with parameters). Returns declarations + spans, not bodies — use `get_symbol_body` for a member's body. Each member's local functions are listed beneath it in declaration order, one level deeper (`localfunction,,,`), a local function declared inside another one level further. ### search_symbols -`solutionId`, `query` (case-insensitive substring), `maxResults` (default 50). Source-declared symbols only (no metadata), local functions included. Declarations a source generator emits are included and marked `,generated=Y` — a generated-only type, member, nested type, namespace or local function is found even when nothing hand-written matches; a partial type with a generated part is still one leaf at its first location (the hand-written part), with the generated part visible through its own members. Matched members nest under their type, and a local function under its containing member (`localfunction` kind); a type's `loc` appears only if the type itself matched. +`solutionId`, `query` (case-insensitive substring), `maxResults` (omit for every match - the result then pages with startItem/maxSize). Source-declared symbols only (no metadata), local functions included. Declarations a source generator emits are included and marked `,generated=Y` — a generated-only type, member, nested type, namespace or local function is found even when nothing hand-written matches; a partial type with a generated part is still one leaf at its first location (the hand-written part), with the generated part visible through its own members. Matched members nest under their type, and a local function under its containing member (`localfunction` kind); a type's `loc` appears only if the type itself matched. ## Relationships ### find_references -`solutionId`, `symbolName` (FQN), `maxResults` (default 100). Compiler-accurate: skips comments, strings, unrelated same-named members. References grouped file→namespace→type→member; pipe-delimited `loc`s on one line for multiple hits. Deduped across `#if` projections. `count=`/`truncated=Y` when capped. Prefer this over text search for usages in `*.cs`, `*.cshtml` and `*.razor`; combine with get_callers/find_implementations via multi_query for impact analysis. +`solutionId`, `symbolName` (FQN), `maxResults` (omit for every reference - the result then pages with startItem/maxSize). Compiler-accurate: skips comments, strings, unrelated same-named members. References grouped file→namespace→type→member; pipe-delimited `loc`s on one line for multiple hits. Deduped across `#if` projections. `count=`/`truncated=Y` when capped. Prefer this over text search for usages in `*.cs`, `*.cshtml` and `*.razor`; combine with get_callers/find_implementations via multi_query for impact analysis. ### find_reads / find_writes -`solutionId`, `symbolName`, `maxResults` (default 100). Target a field or property by FQN (`N.T.Field`), or a parameter as its containing method, constructor, indexer or local function's FQN + `:` + parameter name (`N.T.Method:value`, `N.T.Method(int):value`; the member part follows the usual overload grammar). Local variables are not supported. Same body as find_references, but one location per leaf with the access kind appended: `,,,`. +`solutionId`, `symbolName`, `maxResults` (omit for every location - the result then pages with startItem/maxSize). Target a field or property by FQN (`N.T.Field`), or a parameter as its containing method, constructor, indexer or local function's FQN + `:` + parameter name (`N.T.Method:value`, `N.T.Method(int):value`; the member part follows the usual overload grammar). Local variables are not supported. Same body as find_references, but one location per leaf with the access kind appended: `,,,`. `accessKind`: `read`, `assign` (`=`, including deconstruction targets), `compound` (`+=`, `??=`, ...), `increment` (`++`/`--`), `ref` (`ref x` argument, `ref x` expression, `&x`), `out`, `init` (field/property declaration initialiser, object or `with` initialiser, attribute named argument, or an assignment to the constructor's own field/property inside its constructor; a lambda inside the constructor is `assign`). find_reads returns `read`, `compound`, `increment`, `ref`; find_writes returns everything except `read`. **`compound`, `increment` and `ref` appear in both tools with the same kind** — do not add the two counts. `nameof(...)`, doc crefs and named arguments (`M(value: 1)`) are omitted from both. @@ -211,7 +211,7 @@ MyApp - `projectName`: only that project's diagnostics - exactly the name the outline prints (the project file name with the `.csproj` extension omitted; other extensions such as `.vbproj` are kept, and a trailing `.csproj` you pass is stripped). Case-insensitive, and it matches every loaded target framework of a multi-targeted project. - `filePath`: only diagnostics located in that file - absolute or relative to the solution folder (the same spellings `find_definition` accepts), case-insensitive. A `.razor`/`.cshtml` path also matches the C# diagnostics `#line`-mapped into it and the `RZ*` diagnostics reported against it. - `ids`: only those diagnostic ids, e.g. `["CS0246","CS0103"]`, case-insensitive. Omit, or pass an empty list, for every id. -- `maxResults`: caps the entries listed in the body (id rows in summary mode). When something is dropped, `count=` and `truncated=Y` precede the body; both are absent otherwise, so the body is then the complete set. The per-severity header counts always stay complete. +- `maxResults`: caps the result set (the body lists id rows in summary mode). When the cap drops something, `count=` and `truncated=Y` precede the body; both are absent otherwise. The per-severity header counts always stay complete. Omit maxResults to page through everything with startItem/maxSize. - `summaryOnly`: `true` replaces the body with one line per id - `,,,,,` - ordered by count descending then id, carrying each id's count, one example location (its first diagnostic in body order) and that example's message. It covers every severity unless an include flag is set. `maxResults` caps the id rows, with `count=` counting rows. Every active filter is echoed as a `filter=:` header before the counts, and the counts describe the **filtered** set - `errors=3` under a `filter=` line means 3 errors in that filtered scope, not 3 in the solution. An empty body with a `filter=` header means the filter matched nothing, NOT that the solution is clean. A `projectName` or `filePath` that is not in the solution is `error=NotFound` with `candidate=` values (one of which the tool would accept); one that exists but has no diagnostics returns zero counts, not an error. An `ids` entry that matches nothing is zero counts too - ids are open-ended and never validated. @@ -301,7 +301,7 @@ Razor: the compiler never reports CS8019 in generated code, so each `.razor`/`.c ## Dead code ### find_dead_code -`solutionId`, `scope` (optional FQN prefix, e.g. `MyApp.Services` — use it on large solutions; the scan runs find-references per symbol), `includePublic` (default false — public/protected API excluded unless set), `maxResults` (default 50). +`solutionId`, `scope` (optional FQN prefix, e.g. `MyApp.Services` — use it on large solutions; the scan runs find-references per symbol), `includePublic` (default false — public/protected API excluded unless set), `maxResults` (bounds the expensive scan; omit it to scan everything - the result then pages with startItem/maxSize). Leaf lines: `memberKind,memberName,loc,confidence,reason` with confidence `High|Medium`. Automatically excludes interface implementations, virtual/override/abstract chains, test-attributed members (xUnit/NUnit/MSTest), generated code, and DI/reflection-attributed members (`[Export]`, `[ImportingConstructor]`). The `loc` is the full declaration span, ready to pass to `apply_patch` for removal — this tool never deletes anything itself. Treat results as candidates, not verdicts. @@ -311,7 +311,7 @@ Leaf lines: `memberKind,memberName,loc,confidence,reason` with confidence `High| ## Batching ### multi_query -`solutionId`, `operations` (1-25 items), optional `expectSnapshot`. +`solutionId`, `operations` (1+ items; the limit is the framing itself — a batch whose retry line cannot fit the budget is refused with `error=Invalid` before anything runs), optional `expectSnapshot`. Each operation is `{ "tool": , "arguments": { ... } }` where `tool` is one of the 15 read-only query tools (get_symbol, get_symbol_body, get_members, find_definition, find_implementations, find_references, find_reads, find_writes, get_callers, get_callees, search_symbols, get_type_hierarchy, find_dead_code, find_dead_conditionals, get_expression_info) and `arguments` uses exactly that tool's single-call parameter names - unknown or misspelled keys are rejected (`error=Invalid` naming the key), never ignored; omitted parameters take the tool's declared defaults. The schema's `tool` enum lists the legal names, so a write tool, get_diagnostics or get_solution_status is unrepresentable and fails the whole call at binding (`error=Invalid` naming the offending value and the permitted set). @@ -323,21 +323,22 @@ Everything runs against ONE snapshot of the solution, so results inside one resp operations= snapshot=<32-hex id of the snapshot every slot was computed against> boundary=<32-hex, fresh per request> -budget= +budget= +retry=1:3.0,7:0.2 (present only when operations were dropped) -- -slot= tool= +slot=2 tool=get_type_hierarchy - + -- -slot= tool= truncated=Y +slot=3 tool=get_callees - + ---- ``` -A failing operation does not abort the batch: its slot carries its normal `error=` block (NotFound/Ambiguous/Invalid/...) and the other slots still deliver. The response is a hard ceiling on the whole envelope: framing is reserved up front and each slot gets exactly what remains. A slot whose output would not fit is cut after its last whole line - its meta line gains `truncated=Y` and its body ends with a bracketed `[response budget exhausted: slot () was cut at a line boundary; of lines of its output shown. ... pass expectSnapshot= ...]` note - or, when not even one whole line fits, refused whole as `error=Truncated`. `get_symbol_body` pages itself within its share instead (its `truncated=`/`nextStartLine=` headers are its own, not the batch's). **Trust rule:** a slot whose META LINE carries `truncated=Y` was cut by the batch budget, so its body is a prefix - do not derive tool-level paging (e.g. `nextStartLine`) from it; re-send that operation alone. Only a complete slot's own body headers may drive tool-level paging. +Every operation runs (in parallel); each result is then delivered WHOLE, smallest first, until the budget is filled, and delivered slots are emitted in request order. A slot is complete or absent - never cut - so only a delivered slot's own body headers may drive tool-level paging (`nextStartItem=`). Operations whose result did not fit are listed once in the `retry=` header, in request order, as `:` where size is that result's cost (frame + body) as a ratio of `budget=`, rounded up to 0.1 (0.1 is the smallest value reported, never 0.0). Plan the continuation from the ratios: size ≥ 1.0 can never fit beside anything - re-send that operation alone (it pages); sizes summing to at most 1.0 fit one continuation batch together; `ceil(size)` is roughly a solo call's page count. Ratios smaller than 0.1 of the budget are indistinguishable, so about ten such operations per continuation batch is the safe number. Ratios describe one snapshot: pass the response's `snapshot=` as `expectSnapshot` on the continuation, or a re-run against a moved solution re-measures and re-decides. Re-send exactly the listed operations with their original arguments from your own copy of the request - slot frames carry the index and tool name only, never the arguments. A failing operation is delivered as its normal `error=` block and can itself be dropped like any oversized result. -**Truncation and continuation.** More than 25 operations, or a total output over the response budget, truncates: unexecuted operations still get their numbered slot carrying `error=Truncated`, and the header carries `truncatedSlots=` (counting every slot that did not deliver a complete result: refused or cut - a self-paged `get_symbol_body` slot does not count, its page is a complete answer). To continue, re-send exactly the operations whose slots were Truncated - the slot meta line gives the index and tool name, but NOT the arguments, so rebuild the continuation from your own copy of the request (the server does not echo them back). Pass the first response's `snapshot=` value as `expectSnapshot` when you do: if the solution changed between the calls (another client, or the user saving a file) the continuation is refused with `error=Stale` naming both ids plus a `snapshot=` header - re-run the whole batch instead of stitching two generations. Consistency is guaranteed within one response, never across two. `get_diagnostics` and `get_solution_status` are deliberately not multi-queryable - call them directly. +**Truncation and continuation.** Pass the first response's `snapshot=` value as `expectSnapshot` when you continue: if the solution changed between the calls (another client, or the user saving a file) the continuation is refused with `error=Stale` naming both ids plus a `snapshot=` header - re-run the whole batch instead of stitching two generations. Consistency is guaranteed within one response, never across two. `get_diagnostics` and `get_solution_status` are deliberately not multi-queryable - call them directly. diff --git a/external-sources/upstreams/astro/packages/astro/package.json b/external-sources/upstreams/astro/packages/astro/package.json index 609136bf..d8cdc46a 100644 --- a/external-sources/upstreams/astro/packages/astro/package.json +++ b/external-sources/upstreams/astro/packages/astro/package.json @@ -191,7 +191,7 @@ "devDependencies": { "@astrojs/check": "workspace:*", "@astrojs/markdown-remark": "workspace:*", - "@playwright/test": "1.63.0", + "@playwright/test": "1.64.0", "@types/aria-query": "^5.0.4", "@types/html-escaper": "3.0.4", "@types/http-cache-semantics": "^4.2.0", diff --git a/external-sources/upstreams/dotnet-skills/.github/skills/agentic-workflows/SKILL.md b/external-sources/upstreams/dotnet-skills/.github/skills/agentic-workflows/SKILL.md index 4c983cc5..a70ea478 100644 --- a/external-sources/upstreams/dotnet-skills/.github/skills/agentic-workflows/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/.github/skills/agentic-workflows/SKILL.md @@ -31,6 +31,7 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/create-agentic-workflow.md` - `.github/aw/create-shared-agentic-workflow.md` - `.github/aw/debug-agentic-workflow.md` +- `.github/aw/debug-security-review.md` - `.github/aw/dependabot.md` - `.github/aw/deployment-status.md` - `.github/aw/designer-mappings.md` @@ -46,6 +47,7 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/instructions.md` - `.github/aw/intent.md` - `.github/aw/jobs.md` +- `.github/aw/ledger.md` - `.github/aw/linter-workflows.md` - `.github/aw/llms.md` - `.github/aw/loop.md` @@ -87,6 +89,7 @@ Load these files from `github/gh-aw` (they are not available locally). - `.github/aw/update-agentic-workflow.md` - `.github/aw/upgrade-agentic-workflows.md` - `.github/aw/visual-regression.md` +- `.github/aw/work-queue.md` - `.github/aw/workflow-constraints.md` - `.github/aw/workflow-editing.md` - `.github/aw/workflow-patterns.md` diff --git a/external-sources/upstreams/dotnet-skills/plugins/dotnet-data/skills/optimizing-ef-core-queries/SKILL.md b/external-sources/upstreams/dotnet-skills/plugins/dotnet-data/skills/optimizing-ef-core-queries/SKILL.md index 7892ee5e..84f69884 100644 --- a/external-sources/upstreams/dotnet-skills/plugins/dotnet-data/skills/optimizing-ef-core-queries/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/plugins/dotnet-data/skills/optimizing-ef-core-queries/SKILL.md @@ -1,12 +1,12 @@ --- name: optimizing-ef-core-queries -description: "Optimize and improve the performance of slow Entity Framework Core (EF Core) queries: make them generate less SQL, make fewer database round-trips, and return results faster. Use whenever an EF Core or DbContext query or data-access path is slow or should be made faster — whether or not EF Core owns the database schema. For EF Core, not Dapper or raw ADO.NET." +description: "Review and optimize application data access that uses Entity Framework Core (EF Core). Use for EF Core performance and correctness reviews of services, repositories, imports, and background jobs: slow or memory-heavy queries, excessive round-trips, hidden SaveChanges, premature materialization, dynamic query construction, result-graph tracking, bulk writes, DbContext lifetime or overlapping operations, search semantics, paging, raw SQL, and model/index coverage. Use whether or not EF Core owns the schema. For EF Core, not Dapper or raw ADO.NET." license: MIT --- # Optimizing EF Core Queries -Diagnose and fix slow Entity Framework Core (EF Core) queries. Start from the generated SQL/logs, apply the smallest change that removes the bottleneck, and confirm the fix by re-reading the SQL and the query count. Prefer changes that reduce round-trips, duplicated rows, scans, or per-call translation cost over micro-optimizations. Apply one change at a time and re-measure. +Diagnose and fix slow or incorrect application data access built on EF Core. In a source-only review, trace the code without executing it and separate proven code behavior from runtime assumptions. When SQL, logs, or plans are available, use them to confirm the bottleneck. Prefer changes that reduce round-trips, duplicated rows, scans, or per-call translation cost over micro-optimizations. ## When to Use @@ -21,9 +21,31 @@ Diagnose and fix slow Entity Framework Core (EF Core) queries. Start from the ge - **The code uses Dapper or raw ADO.NET, not EF Core.** Answer the SQL/indexing/query-plan question directly; do not introduce a `DbContext` or recommend `AsNoTracking`, `Include`, `AsSplitQuery`, or other EF Core APIs. -## First: capture the generated SQL +## Source review workflow -You cannot optimize what you cannot see. Turn on command logging and read the SQL and query count before changing anything: +1. Find the EF boundary: `DbContext`, `DbSet`, repository implementation, unit of work, query helper, or model-building code. Follow the implementation rather than assuming what an interface method does. +2. Trace each query to execution or enumeration. Check whether operators bind to `Queryable` or `Enumerable`, including inside helpers; an `IEnumerable` can remain a deferred database sequence. +3. Count database operations and actual transaction boundaries along the calling path, including immediate writes and repeated enumeration. Also track which operations share the same `DbContext` instance +4. For every execution path, independently inspect predicates, expression construction, projection contents, result bounds, ordering, Includes, tracking, SQL parameterization, and model-building code. +5. Preserve the contract: result membership and order, random-page access, generated keys, retries, transaction and partial-failure behavior, concurrency handling, and domain/interceptor side effects. +6. Report prioritized findings with locations, the affected call path, concrete impact, and a safe remedy. Briefly identify sound counterparts and state what was not validated. Do not equate source reasoning with measured plans or speedups. + +For a source-only request, recommend runtime validation as a next step. + +For raw SQL, prefer EF Core's interpolated overloads (`FromSql` / `FromSqlInterpolated` and `ExecuteSql[Async]`) so values are sent as parameters. + +## Gate source-only findings on evidence + +When the user requests a source-only review, distinguish a demonstrated defect from an optimization that still needs workload or deployment evidence: + +- A load-mutate loop followed by one `SaveChanges[Async]` is not `SaveChanges` in a loop. Do not call it a per-row round-trip defect; suggest a bulk API only when the operation is demonstrably large and bypassing tracking semantics is safe. +- Missing `HasIndex` calls in one supplied file do not prove that the deployed database lacks indexes. Use relevant model-building code, migrations, or a query plan as evidence. Do not pad a source review with generic conditional index advice when that evidence is absent. +- Missing `AsNoTracking` is actionable only on proven read-only entity results. Scalar-only projections have nothing to track, but an entity inside a DTO/record is still tracked by default. +- If every reviewed pattern is sound, say that no actionable changes are needed rather than manufacturing a finding. + +## Capture SQL and query counts + +Use supplied logs and plans first. For an executable investigation, enable command logging to verify query shape and count before and after a change: ```csharp optionsBuilder.LogTo(Console.WriteLine, LogLevel.Information); @@ -34,31 +56,50 @@ Tag a query with `.TagWith("...")` to find it in the log. Count how many stateme ## Fixes -### Keep predicates sargable — never wrap an indexed column in a function +### Trace materialization and result bounds + +Materialization inside a helper is still an execution boundary. A list or array has already loaded rows; later reductions run in memory. An `IEnumerable` can instead hide a deferred EF query: its `Enumerable` operators process rows locally, and each enumeration may execute the database query again. Trace implementation and static types, not just method names. Keep translatable filters, projections, and aggregates on `IQueryable` until execution: + +```csharp +// Loads rows, then filters in memory. +var candidates = await repository.LoadCandidatesAsync(cancellationToken); +return candidates.Where(o => o.CreatedAt < cutoff).ToList(); + +// Composes and filters in SQL. +return await repository.QueryCandidates() + .Where(o => o.CreatedAt < cutoff) + .ToListAsync(cancellationToken); +``` + +Use `AnyAsync` for boolean existence, but preserve `CountAsync` when the numeric total is required. Combine repeated full-sequence summary calculations into server-side aggregates rather than caching an unbounded list or parallelizing queries on one context. -An index can only be used when the indexed column appears **bare** on one side of the comparison. Wrapping it in a function or arithmetic — `CreatedAt.Year == y`, `CreatedAt.Date == d`, `ToLower(Name) == n`, `Price * 1.1 > x`, or a leading-wildcard `LIKE '%foo'` — forces a per-row computation the index cannot satisfy, so the query **scans the whole table even though the index exists**. Adding another index changes nothing. Rewrite the predicate so the column stays bare, usually as a half-open range: +Separately review result cardinality. Page or cap interactive results; for complete exports, reduce buffering with streaming/chunks without truncating the contract. + +### Keep predicates compatible with indexes + +Functions/arithmetic on a column, or unanchored substring matching, commonly prevent a seek on an ordinary column index. Prefer a bare-column comparison, often a half-open range. Do not assert a full-table scan from LINQ alone: provider translations, computed/expression indexes, and other predicates can change the plan. ```csharp -// Non-sargable: a function is computed for every row → full scan +// Computes a year expression instead of comparing the indexed date. db.Logs.Where(l => l.CreatedAt.Year == year); -// Sargable: bare column compared to constants → index seek +// Range compatible with an ordinary date index. var start = new DateTime(year, 1, 1); db.Logs.Where(l => l.CreatedAt >= start && l.CreatedAt < start.AddYears(1)); ``` The same rule covers several common shapes: -- **Case-insensitive text** — compare a stored normalized column instead of `ToLower(...)`/`ToUpper(...)`. +- **Case-insensitive text** — use an existing indexed normalized/computed value or a compatible column collation instead of wrapping the column. Query-specific collation changes can also defeat an index with a different collation. Preserve normalization and comparison semantics. `StringComparison` overloads are generally not translated by relational providers; current EF Core throws for an untranslatable predicate rather than silently filtering locally. - **Computed expressions** — compare against the precomputed constant, not `column * k > x`. -- **Converting the column to another type** — a predicate over `column.ToString()` (for example matching the *text form* of a number or date, `total.ToString().StartsWith(p)`) applies a function to every row and often can't be translated to SQL at all, forcing a client-side evaluation that pulls the whole table into memory. Filter on the typed column with a real comparison or range instead. -- **Substring search** — `name.Contains(term)` becomes an unanchored `LIKE '%term%'` that can't seek an index and scans the table; a trailing-wildcard prefix (`name.StartsWith(term)` → `'term%'`) can seek. Anchor the search when a prefix match is acceptable — this changes which rows match, so confirm the behavior first — and put real substring or fuzzy search behind a full-text index on large tables. +- **Converting the column to another type** — a predicate over `column.ToString()` (for example matching the *text form* of a number or date, `total.ToString().StartsWith(p)`) applies a function to every row and may not translate. Current EF Core reports a translation failure rather than silently evaluating an untranslatable predicate on the client. Filter on the typed column with a real comparison or range instead. +- **Substring search** — `name.Contains(term)` is generally not seekable with an ordinary B-tree index; prefix matching may be. Neither prefix matching nor token-based full-text search is a drop-in replacement for arbitrary substring matching. Preserve the required search mode or explicitly disclose the change; consider a provider-appropriate substring index/search design. -**Verify:** the plan shows a seek/index instead of a scan and duration drops. If the column genuinely has no index, add one (see below) — but only after the predicate is sargable. +**Verify:** when runtime validation is requested, inspect the resulting predicate and plan. An additional ordinary index does not by itself repair an incompatible expression. ### Compile hot, frequently-executed queries -On a very hot path that runs the *same* query shape thousands of times over a reused context, EF Core re-parses the LINQ expression tree and probes its query cache on every call. When the query is already minimal (an indexed lookup or a small projection) and read-only tweaks such as `AsNoTracking` buy nothing, that per-call translation is the remaining cost. Compile the query once with `EF.CompileQuery` / `EF.CompileAsyncQuery` and reuse the delegate: +EF Core already caches translation for matching query shapes. On a very hot, otherwise minimal query, expression processing and cache lookup can still matter. `EF.CompileQuery` / `EF.CompileAsyncQuery` reuse a delegate that bypasses that lookup; they do not make database I/O cheaper: ```csharp private static readonly Func GetProduct = @@ -74,6 +115,50 @@ The delegate is `static` (compiled once) and takes the `DbContext` plus each par **Verify:** the hot loop's mean time drops with identical results. +### Control SQL parameterization and database plan reuse + +Distinguish EF's query-translation cache from the database's execution-plan cache. Captured LINQ values normally become SQL parameters, while fixed literals remain constants. Parameters keep SQL text stable across values and usually promote database plan reuse; literals expose specific values to the optimizer and can improve selectivity estimates, but many distinct literal values can fragment the server plan cache. Keep the defaults unless workload evidence supports an override: + +| Override | SQL effect | Performance decision | +|----------|------------|----------------------| +| `EF.Parameter(value)` (EF Core 9+) | Forces a parameter, including for a literal | Use when reusable SQL across values matters; already captured arguments usually need no override | +| `EF.Constant(value)` (EF Core 8.0.2+) | Forces an inline constant, including for a captured value | Reserve for evidenced value-specific plan benefits that outweigh extra SQL variants | + +```csharp +db.Orders.Where(order => order.Status == EF.Parameter("Pending")); +db.Orders.Where(order => order.CustomerId == EF.Constant(customerId)); +``` + +The first query parameterizes a fixed status; the second specializes SQL for each customer ID. Neither is inherently faster. Do not parameterize every structural literal, or inline high-cardinality request values without evidence. Provider behavior, parameter-sensitive plans, data skew, and collection translations affect the tradeoff. + +In EF Core 9, these overrides are processed later than query-shape caching: `EF.Constant` can reuse EF's translation while still generating different SQL literals. It is not equivalent to embedding each request value with `Expression.Constant` in a dynamic tree. Compiled delegates do not fix server plan-cache fragmentation; EF Core 9 also does not support these overrides inside `EF.CompileQuery` / `EF.CompileAsyncQuery`. + +**Verify:** compare generated SQL, server compilation/plan reuse, estimates, and representative latency when execution is available. A source review can identify the override and likely tradeoff, not prove a faster plan. + +### Stabilize dynamically constructed query shapes + +Follow request values into expression builders. `Expression.Constant(requestValue)` inside a comparison can produce a new EF cache entry and different SQL for each value. Represent changing values through correctly typed captured-value expressions instead: + +```csharp +Expression> valueAccess = () => filterValue; +var comparison = Expression.Equal(propertyExpression, valueAccess.Body); +``` + +Creating the entity's `ParameterExpression` does not parameterize its comparison values. Ordinary captured LINQ arguments are already parameterizable; fixed structural constants are not automatically defects. Preserve null/conversion semantics, and canonicalize equivalent filter order only when it is safe. Making the builder static, caching a delegate per value, or compiling every new dynamic expression does not stabilize the shape. Report potential compilation/cache churn, not a proven unbounded leak. + +### Choose tracking for the actual result graph + +| Result and consumer | Decision | +|---------------------|----------| +| Scalar/value-only projection | No entity tracking to remove; do not add `AsNoTracking` as a finding | +| Entity or entity-valued DTO used for later edits | Preserve tracking and its identity semantics | +| Read-only entity graph | Consider no tracking; narrow the projection first if full entities are unnecessary | +| Read-only graph with heavily shared entities or reference-identity consumers | Consider `AsNoTrackingWithIdentityResolution`, or key-based DTO processing | + +Plain `AsNoTracking` does **not** resolve repeated keys to one object. Identity-resolution mode uses a query-local map without attaching results to the context; it costs bookkeeping and does not reduce joined SQL payload or deduplicate across separate queries. + +For filtered Includes, inspect earlier/later loads on the same tracking context. Navigation fixup can add already tracked children that fail the Include predicate; the filtered navigation is also marked loaded. Use a detached projection with its own child filter, an appropriate no-tracking read, or a fresh read context when the subset matters. Do not clear a write tracker with pending work just to clean up a read. + ### Remove N+1 and avoid lazy loading The same `SELECT` repeated once per row (a navigation accessed inside a loop) is an N+1. Load the related data in one round-trip — project the aggregates with `Select`, or eager-load with `Include`: @@ -88,15 +173,17 @@ Prefer projection or `Include` over lazy loading: lazy loading is a leading caus **Verify:** a fixed, small query count regardless of row count. -### Split multiple collection Includes +### Separate join multiplication from wide-row duplication -`Include`ing two or more collection navigations in one query multiplies rows (a Cartesian explosion) and duplicates parent data. Use `AsSplitQuery()` so each collection loads in its own statement; add `OrderBy` on a unique key so rows stitch together: +Sibling collection Includes can multiply each other's rows; a nested collection chain is not the same cross-product, and references do not count as additional collections. Even one collection can repeat a wide parent blob/text column many times. Prefer a narrow projection when the consumer does not need that column or graph. Otherwise consider `AsSplitQuery` or targeted loading: ```csharp db.Blogs.Include(b => b.Posts).Include(b => b.Contributors).AsSplitQuery(); ``` -**Verify:** rows per statement drop sharply and total duration improves. +Splitting adds statements/round-trips and can buffer results. Preserve deliberate `AsSingleQuery` and consistency requirements; stable multi-statement reads may need an appropriate snapshot/serializable transaction. Paged results need unique ordering (especially split pagination before EF Core 10); ordering is not what establishes entity identity. Do not prescribe splitting merely by counting Includes. + +**Verify:** lower payload/row duplication outweighs extra round-trips with the required consistency. ### Filter and paginate; prefer keyset over offset @@ -106,13 +193,20 @@ Constrain large result sets with `Where`, and page with **keyset (seek)** pagina db.Orders.Where(o => o.Id > lastSeenId).OrderBy(o => o.Id).Take(pageSize); ``` -Order by a unique, stable, indexed key (add tie-breakers if the sort column isn't unique). Keyset pages by the *last key seen* rather than a page number, so it changes the method's inputs; when a fixed signature rules out an in-place switch, still flag the deep-offset scan and recommend keyset. +Order by a unique, stable key with suitable index coverage. A descending `(CreatedAt, Id)` cursor requires `CreatedAt < lastDate || (CreatedAt == lastDate && Id < lastId)`, not independent bounds on both fields. Include every ordering key with its direction; IDs need not follow business timestamps. + +Trace ordering through query helpers and execution boundaries: + +- A second `OrderBy`/`OrderByDescending` on an `IQueryable` replaces the earlier primary ordering; use `ThenBy` for a secondary key. +- `OrderBy(...).Skip(...).Take(...).AsEnumerable().OrderBy(...)` first chooses a stable database window and then reorders only that bounded window in memory. That can be intentional and is not an unordered page. +- `Skip`/`Take` without a stable unique order is nondeterministic. Deep offset paging is a separate scaling concern, and keyset pagination is not behavior-preserving for random page-number access. +- Even fully ordered offset pages can shift under concurrent inserts/deletes; adding an order fixes nondeterminism, not cross-request snapshot consistency. **Verify:** page latency stays roughly constant from early to deep pages. ### Add missing indexes -Moving a filter into SQL or making a predicate sargable stops the *client-side* waste, but a `WHERE` or `ORDER BY` on a column with no index still scans the whole table inside the database — and a frequently-run query then re-scans it on every call. So audit index coverage separately from the query rewrite: for each query, check whether its filter and sort columns are backed by an index. Entity keys and foreign keys are indexed by convention, but other columns — status flags, state/enum fields, timestamps, names — usually are **not** unless the model configures it. When a hot query filters or sorts on such an unindexed column, recommend adding an index and say so explicitly, even when the rewritten query already returns the right rows: the index is a separate fix the code change alone doesn't deliver. (If the predicate isn't sargable, fix that first — a new index can't help a scan caused by a function on the column.) +Audit workload-specific coverage separately from moving filters into SQL. Inspect applied model configurations and conventions. Derive each dependent FK's ordered properties from `HasForeignKey`; EF normally supplies an index on that tuple unless an existing index/key covers the ordered prefix and required uniqueness, or the convention was removed. Do not count only `HasIndex` calls or confuse a principal key with the dependent table's coverage. Status, timestamps, and other non-key columns may need explicit coverage. Recommend evidenced gaps for hot queries, not generic indexes in every review. If EF Core owns the schema, add the index in the model and migrate: @@ -123,36 +217,64 @@ modelBuilder.Entity() Then create the migration with `dotnet ef migrations add ...`. **Do not apply it** with `dotnet ef database update` (or any equivalent that writes to the database) without explicit user approval — applying a migration mutates the database, so add the migration, show it to the user, and let them run the update once they've reviewed it. If EF Core does not own the schema, recommend the same index to whoever manages the database. Don't over-index — every index slows writes. +Match equality columns, then range/sort keys. An unconstrained intermediate key can block the desired range/order even if an earlier equality column is covered. When all initial keys use equality, more than one order may work; do not condemn a reversed composite index from relationship metadata alone. A FK index can cover the relationship but not a different date-ordered workload. Consider an additional workload index rather than replacing one that supports another path. Model code proves intended metadata, not current physical indexes; weigh write cost and provider-specific coverage. + **Verify:** the plan uses a seek/index instead of a scan. +### Trace persistence through repositories + +A service loop can hide `SaveChanges` behind a repository, audit journal, or progress hook. One final unit-of-work commit does not undo earlier writes; establish whether an enclosing transaction exists before claiming per-item commits. A save can itself require multiple SQL batches. + +Stage bounded batches and save together when the contract permits. Preserve audit/outbox records, generated keys, relationship fixup, ordering, concurrency, and domain events. Capture IDs from the input-ordered entity references after saving; do not assume numerical ID allocation order. + +For retried feeds, reason about what already committed and what the caller can replay. Preserve or design durable row/feed identity with uniqueness and replay handling, or acknowledged checkpoints. Chunking alone is not idempotency, and an unprotected existence-check-then-insert is not race-safe. + +### Bound change detection in growing imports + +Successfully saved entities remain tracked. With automatic detection enabled, `ChangeTracker.Entries()`, `HasChanges`, and `SaveChanges` inspect tracked state; filtering entries by type does not limit the preceding scan. Repeated per-row inspection of a growing graph can create superlinear CPU work even when saves are chunked. + +Use import-owned progress counters instead of repeated tracker enumeration, and rotate a context or clear completed work only when that context is owned by the batch. Preserve pending changes and references. EF Core `Add` and `AddRange` do **not** automatically call `DetectChanges`; replacing one with the other is not a detection fix. Suppress automatic detection only for a demonstrated redundant scan, restore its previous setting in `finally`, and explicitly detect mutations when required before persistence/auditing. + ### Set-based bulk updates and deletes -Replace a load-mutate-`SaveChanges` loop with `ExecuteUpdateAsync`/`ExecuteDeleteAsync` (EF Core 7+) — one statement, no entities materialized: +For demonstrably large, simple updates/deletes that need no per-entity processing, consider `ExecuteUpdateAsync`/`ExecuteDeleteAsync` (EF Core 7+ and a supporting provider). They execute immediately without materialization: ```csharp await db.Products.Where(p => p.LastSoldDate < cutoff) .ExecuteUpdateAsync(s => s.SetProperty(p => p.IsActive, false)); ``` -These bypass the change tracker and EF-side cascade behavior — apply related changes explicitly. +Check the entire write workflow: + +- Immediate writes do not update loaded entity values or snapshots. Later mutations of the same properties can overwrite the SQL change **even in one sequential request**. Use one coherent tracked save or an explicitly synchronized/untracked set-based path; a transaction alone does not refresh tracked state. +- Configured concurrency tokens are not automatically checked by these APIs. Preserve token predicates, token updates, and affected-row handling when optimistic concurrency is required. +- Each call executes separately; a later `SaveChanges` neither batches nor makes them atomic with audit/outbox writes. Use one save unit or an explicit shared transaction where required. +- Preserve client-side cascades and SaveChanges-based hooks. Command interceptors are different and can still observe the SQL. Do not bypass domain behavior merely to reduce query count. **Verify:** a single `UPDATE`/`DELETE` with a `WHERE` and no preceding `SELECT`. +### Keep DbContext operations non-overlapping + +EF Core does not support overlapping operations on one `DbContext`, including through distinct repository objects constructed with that context. Starting both calls before awaiting either is enough; `Task.WhenAll` does not make sharing safe. Await sequentially, or create/dispose one independent context per concurrent operation. Separate factory-created contexts are safe to overlap. A static or singleton-held context also creates lifetime/thread-safety hazards. + ## Common Pitfalls | Pitfall | Fix | |---------|-----| -| Wrapping an indexed column in `.Year`/`.Date`/`ToLower`/arithmetic | Rewrite to a sargable range/comparison on the bare column | -| Adding an index to fix a scan on a non-sargable predicate | Fix the predicate first; the index can't help until the column is bare | -| Rewriting a filter into SQL but leaving a hot query on an unindexed column | The server-side scan is still a scan — recommend an index on the filter/sort column too | +| Function-wrapped predicate incompatible with an ordinary index | Use a range or an existing indexed normalized/computed value | +| Fixing query composition but ignoring workload index coverage | Match the actual equality/range/order prefix, including convention-created indexes | | Compiling a query that runs only occasionally | Compile only genuinely hot, high-frequency query shapes | -| Lazy loading (proxies / `virtual` navigations) causing N+1 and forced sync I/O | Eager-load (`Include`) or project; keep queries async | -| `ToList()`/`AsEnumerable()` before `Where`/`Select` | Keep the query `IQueryable` so filtering/projection run in SQL | +| Confusing EF translation reuse with database plan reuse | Inspect SQL parameter/literal choices and justify overrides with workload evidence | +| DTO contains entities but is assumed untracked | Inspect the materialized graph, not the outer return type | +| Bulk write followed by tracked arithmetic on stale values | Unify writes and preserve concurrency/atomicity; do not just add a transaction | +| Repeated enumeration of a repository's deferred `IEnumerable` | Compose database aggregates before crossing to local operators | ## References - [Efficient querying — EF Core](https://learn.microsoft.com/en-us/ef/core/performance/efficient-querying) - [Compiled queries — EF Core](https://learn.microsoft.com/en-us/ef/core/performance/advanced-performance-topics#compiled-queries) +- [Force or prevent query parameterization — EF Core 9](https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-9.0/whatsnew#force-or-prevent-query-parameterization) +- [Parameterization overrides inside compiled queries — EF Core 9](https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-9.0/breaking-changes) - [Efficient updating (ExecuteUpdate/ExecuteDelete) — EF Core](https://learn.microsoft.com/en-us/ef/core/performance/efficient-updating) - [Single vs. split queries](https://learn.microsoft.com/en-us/ef/core/querying/single-split-queries) - [Pagination](https://learn.microsoft.com/en-us/ef/core/querying/pagination) diff --git a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/Compute-CrapScores.ps1 b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/Compute-CrapScores.ps1 index b0c8d9f3..121b54e4 100644 --- a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/Compute-CrapScores.ps1 +++ b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/Compute-CrapScores.ps1 @@ -1,165 +1,80 @@ -# Compute-CrapScores.ps1 -# -# Reads a Cobertura XML coverage file and calculates CRAP scores per method. -# Uses Alberto Savoia's original CRAP formula: -# CRAP(m) = comp(m)^2 * (1 - cov(m))^3 + comp(m) -# -# Usage: -# .\Compute-CrapScores.ps1 -CoberturaPath ,,... [-CrapThreshold ] [-TopN ] -# -# Outputs: -# - OVERALL_LINE_COVERAGE: (aggregate line coverage across input files, as percent) -# - OVERALL_BRANCH_COVERAGE: (aggregate branch coverage across input files, as percent) -# - TOTAL_METHODS: -# - FLAGGED_METHODS: -# - HOTSPOTS: (top N by CRAP score) - +<# +.SYNOPSIS +Compute CRAP = complexity^2 * (1 - line coverage)^3 + complexity. +.DESCRIPTION +Missing complexity or executable lines yield unknown, never complexity=1/coverage=0. +Legacy output retains OVERALL_*, TOTAL_METHODS, FLAGGED_METHODS and HOTSPOTS markers +only when all requested measurements are representable. -OutputFormat json-v1 +emits one provenance-bearing JSON document. Diagnostics go to stderr. +Thresholds and hotspot ordering use raw scores; only displayed scores are rounded. +Exit codes: 0 complete, 2 invalid input, 3 incomplete/unrepresentable evidence. +#> param( [Parameter(Mandatory)][string[]]$CoberturaPath, - [int]$CrapThreshold = 30, - [int]$TopN = 10 + [ValidateRange(0,[int]::MaxValue)][int]$CrapThreshold = 30, + [ValidateRange(1,[int]::MaxValue)][int]$TopN = 10, + [ValidateSet('legacy', 'json-v1')][string]$OutputFormat = 'legacy' ) -# Merge methods across all Cobertura files using a stable key (Class|Method|Signature|File). -# Line hits are accumulated so a line is counted as covered if any input coverage file covered it. -$methodMap = @{} -$overallLineRate = 0.0 -$overallBranchRate = 0.0 -$totalLinesCovered = 0 -$totalLinesValid = 0 -$totalBranchesCovered = 0 -$totalBranchesValid = 0 -$fallbackLineRates = [System.Collections.Generic.List[double]]::new() -$fallbackBranchRates = [System.Collections.Generic.List[double]]::new() - -foreach ($filePath in $CoberturaPath) { - if (-not (Test-Path $filePath)) { - Write-Error "Cobertura file not found: $filePath" - exit 2 - } - - try { - [xml]$cobertura = Get-Content $filePath -Encoding UTF8 -ErrorAction Stop - } catch { - Write-Error "Failed to parse Cobertura XML: $filePath. $_" - exit 2 - } - - # Prefer aggregate numerator/denominator attributes when present. - if ($null -ne $cobertura.coverage.'lines-covered' -and $null -ne $cobertura.coverage.'lines-valid') { - $totalLinesCovered += [double]$cobertura.coverage.'lines-covered' - $totalLinesValid += [double]$cobertura.coverage.'lines-valid' - } elseif ($cobertura.coverage.'line-rate') { - $fallbackLineRates.Add([double]$cobertura.coverage.'line-rate') - } - if ($null -ne $cobertura.coverage.'branches-covered' -and $null -ne $cobertura.coverage.'branches-valid') { - $totalBranchesCovered += [double]$cobertura.coverage.'branches-covered' - $totalBranchesValid += [double]$cobertura.coverage.'branches-valid' - } elseif ($cobertura.coverage.'branch-rate') { - $fallbackBranchRates.Add([double]$cobertura.coverage.'branch-rate') - } - - foreach ($package in $cobertura.coverage.packages.package) { - foreach ($class in $package.classes.class) { - $className = $class.name - $fileName = $class.filename - - foreach ($method in $class.methods.method) { - $key = "$className|$($method.name)|$($method.signature)|$fileName" - - # Cyclomatic complexity is stored as an XML attribute in Cobertura format - $complexity = if ($null -ne $method.complexity) { [int]$method.complexity } else { 1 } - if ($complexity -lt 1) { $complexity = 1 } - - if (-not $methodMap.ContainsKey($key)) { - $methodMap[$key] = @{ - Class = $className - Method = $method.name - Signature = $method.signature - File = $fileName - Complexity = $complexity - LineHits = @{} - } - } - - # Accumulate hit counts per line number across files - foreach ($line in $method.lines.line) { - $lineNo = $line.number - $hits = [int]$line.hits - if ($methodMap[$key].LineHits.ContainsKey($lineNo)) { - $methodMap[$key].LineHits[$lineNo] += $hits - } else { - $methodMap[$key].LineHits[$lineNo] = $hits - } - } - } - } +try { + . "$PSScriptRoot\CoverageData.ps1" +} catch { + $diagnostic = @{ code = 'DEPENDENCY_UNAVAILABLE'; severity = 'error'; message = $_.Exception.Message; path = "$PSScriptRoot\CoverageData.ps1" } + [Console]::Error.WriteLine("$($diagnostic.code): $($diagnostic.message)") + if ($OutputFormat -eq 'json-v1') { + @{ + schema_version = '1'; tool = 'Compute-CrapScores'; tool_version = '1.0'; status = 'error' + scope = @{}; inputs = @(); diagnostics = @($diagnostic); data = @{} + } | ConvertTo-Json -Depth 5 } + exit 2 } - -$results = [System.Collections.Generic.List[PSCustomObject]]::new() - -foreach ($entry in $methodMap.Values) { - $totalLines = $entry.LineHits.Count - $coveredLines = ($entry.LineHits.Values | Where-Object { $_ -gt 0 } | Measure-Object).Count - $lineCoverage = if ($totalLines -gt 0) { $coveredLines / $totalLines } else { 0.0 } - - $complexity = $entry.Complexity - - # Alberto Savoia's CRAP formula: comp^2 * (1 - cov)^3 + comp - # The cubic exponent on (1-cov) sharply penalizes low coverage: - # at 0% coverage the risk multiplier is 1.0; at 50% it drops to 0.125. - # Higher scores = more complex AND less covered = riskier to change - $uncovered = 1.0 - $lineCoverage - $crapScore = [Math]::Round(($complexity * $complexity * [Math]::Pow($uncovered, 3)) + $complexity, 2) - - $results.Add([PSCustomObject]@{ - Class = $entry.Class - Method = $entry.Method - Signature = $entry.Signature - File = $entry.File - TotalLines = $totalLines - CoveredLines = $coveredLines - LineCoverage = [Math]::Round($lineCoverage * 100, 1) - Complexity = $complexity - CrapScore = $crapScore - }) +$scope = @{ identity = 'reported-module-file-class-signature'; source_revision = $null; top = $TopN; crap_threshold = $CrapThreshold } +try { + $coverage = Get-CoverageData $CoberturaPath +} catch { + $diagnostics = @([pscustomobject]@{ code = 'INVALID_INPUT'; severity = 'error'; message = $_.Exception.Message }) + Write-CoverageDiagnostics $diagnostics + if ($OutputFormat -eq 'json-v1') { Write-CoverageJson 'Compute-CrapScores' 'error' $scope @() $diagnostics @{} } + exit 2 } - -$hotspots = $results | Sort-Object CrapScore -Descending | Select-Object -First $TopN -$flagged = $results | Where-Object { $_.CrapScore -gt $CrapThreshold } - -if ($totalLinesValid -gt 0) { - $overallLineRate = $totalLinesCovered / $totalLinesValid -} else { - # Fallback approximation when Cobertura aggregate counters and per-file rates are unavailable. - # This uses merged method line totals and may under/over-estimate if Cobertura - # includes executable lines outside method nodes. - $mergedTotalLines = ($results | Measure-Object -Property TotalLines -Sum).Sum - $mergedCoveredLines = ($results | Measure-Object -Property CoveredLines -Sum).Sum - if ($mergedTotalLines -gt 0) { - $overallLineRate = [double]$mergedCoveredLines / [double]$mergedTotalLines - } elseif ($fallbackLineRates.Count -gt 0) { - $overallLineRate = ($fallbackLineRates | Measure-Object -Average).Average - } else { - $overallLineRate = 0.0 - } +foreach ($method in $coverage.Methods) { + $score = if ($null -ne $method.Complexity -and $method.LineCoverageStatus -eq 'exact') { + [math]::Pow($method.Complexity, 2) * [math]::Pow(1 - $method.CoveredLines / [double]$method.TotalLines, 3) + $method.Complexity + } else { $null } + $method | Add-Member -NotePropertyName CrapScore -NotePropertyValue $score } - -if ($totalBranchesValid -gt 0) { - $overallBranchRate = $totalBranchesCovered / $totalBranchesValid -} elseif ($fallbackBranchRates.Count -gt 0) { - $overallBranchRate = ($fallbackBranchRates | Measure-Object -Average).Average -} else { - $overallBranchRate = 0.0 +$ranked = @($coverage.Methods | Where-Object { $null -ne $_.CrapScore } | + Sort-Object -Property @{Expression='CrapScore';Descending=$true}, Class, Method, File, Signature) +$hotspots = @($ranked | Select-Object -First $TopN) +$unknown = @($coverage.Methods | Where-Object { $null -eq $_.CrapScore }).Count +$coverage.Summary.flagged_methods = @($ranked | Where-Object { $_.CrapScore -gt $CrapThreshold }).Count +$coverage.Summary.unknown_crap_methods = $unknown +foreach ($method in $ranked) { + $method.CrapScore = [math]::Round($method.CrapScore, 2) } - -Write-Host "OVERALL_LINE_COVERAGE:$([Math]::Round($overallLineRate * 100, 1))" -Write-Host "OVERALL_BRANCH_COVERAGE:$([Math]::Round($overallBranchRate * 100, 1))" -Write-Host "TOTAL_METHODS:$($results.Count)" -Write-Host "FLAGGED_METHODS:$($flagged.Count)" -if ($hotspots) { - Write-Output "HOTSPOTS:$(@($hotspots) | ConvertTo-Json -Compress)" -} else { - Write-Output "HOTSPOTS:[]" +foreach ($metric in @('line', 'branch')) { + if ($coverage.Summary["overall_${metric}_coverage_status"] -in @('unknown', 'bounded')) { + Add-CoverageDiagnostic $coverage.Diagnostics 'AGGREGATE_UNKNOWN' "Overall $metric coverage cannot be derived from overlapping report counters." '' + } +} +$partial = -not $coverage.Methods.Count -or $unknown -gt 0 -or + $coverage.Summary.overall_line_coverage_status -in @('unknown', 'bounded') -or + $coverage.Summary.overall_branch_coverage_status -in @('unknown', 'bounded') +Write-CoverageDiagnostics $coverage.Diagnostics +if ($OutputFormat -eq 'json-v1') { + Write-CoverageJson 'Compute-CrapScores' $(if ($partial) { 'partial' } else { 'complete' }) $scope $coverage.Inputs $coverage.Diagnostics @{ + methods = $coverage.Methods; hotspots = $hotspots; summary = $coverage.Summary + } +} elseif (-not $partial) { + $line = if ($coverage.Summary.overall_line_coverage_status -eq 'not-applicable') { 0 } else { $coverage.Summary.overall_line_coverage } + $branch = if ($coverage.Summary.overall_branch_coverage_status -eq 'not-applicable') { 0 } else { $coverage.Summary.overall_branch_coverage } + Write-Output "OVERALL_LINE_COVERAGE:$line" + Write-Output "OVERALL_BRANCH_COVERAGE:$branch" + Write-Output "TOTAL_METHODS:$($coverage.Methods.Count)" + Write-Output "FLAGGED_METHODS:$($coverage.Summary.flagged_methods)" + $legacy = @($hotspots | Select-Object Class, Method, Signature, File, TotalLines, CoveredLines, LineCoverage, Complexity, CrapScore) + Write-Output "HOTSPOTS:$(ConvertTo-Json -InputObject $legacy -Compress)" } +if ($partial) { exit 3 } +exit 0 diff --git a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/CoverageData.ps1 b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/CoverageData.ps1 new file mode 100644 index 00000000..25a343d1 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/CoverageData.ps1 @@ -0,0 +1,272 @@ +# Shared, report-identity-scoped Cobertura normalization. This does not attest +# that a report was collected from the current checkout. + +function New-CoverageMap { + return [System.Collections.Generic.Dictionary[string, object]]::new([StringComparer]::Ordinal) +} + +function Add-CoverageDiagnostic($Diagnostics, [string]$Code, [string]$Message, [string]$Path) { + $Diagnostics.Add([pscustomobject]@{ code = $Code; severity = 'warning'; message = $Message; path = $Path }) +} + +function Read-CoverageInteger([string]$Text, [string]$Name, [long]$Minimum = 0) { + $value = 0L + if (-not [long]::TryParse($Text, [Globalization.NumberStyles]::Integer, + [Globalization.CultureInfo]::InvariantCulture, [ref]$value) -or $value -lt $Minimum) { + throw [FormatException]::new("Invalid $Name value '$Text'.") + } + return $value +} + +function Add-CoverageLine($Map, $Node, [string]$Key, [string]$ReportId) { + $number = Read-CoverageInteger $Node.GetAttribute('number') 'line number' 1 + $hits = Read-CoverageInteger $Node.GetAttribute('hits') 'line hits' + $lineKey = "$Key`n$number" + if (-not $Map.ContainsKey($lineKey)) { + $Map[$lineKey] = @{ + Covered = $false + NonBranch = $false + ConflictingBranches = $false + Branches = [System.Collections.Generic.List[object]]::new() + Observations = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + } + } + $entry = $Map[$lineKey] + $entry.Covered = $entry.Covered -or $hits -gt 0 + if (-not $entry.Observations.Add($ReportId)) { return } + $branch = $Node.GetAttribute('branch') + if ($branch -and $branch -notin @('true', 'false')) { + throw [FormatException]::new("Invalid branch flag '$branch'.") + } + if ($branch -ne 'true') { + $entry.NonBranch = $true + if ($entry.Branches.Count) { $entry.ConflictingBranches = $true } + return + } + if ($entry.NonBranch) { $entry.ConflictingBranches = $true } + $raw = $Node.GetAttribute('condition-coverage') + if ($raw -notmatch '^\s*[\d.]+%\s*\((\d+)/(\d+)\)\s*$') { + $entry.Branches.Add(@{ Total = $null; Covered = $null; Conditions = $null }) + return + } + $covered = Read-CoverageInteger $Matches[1] 'covered branches' + $total = Read-CoverageInteger $Matches[2] 'total branches' 1 + if ($covered -gt $total) { throw [FormatException]::new('Covered branches exceed total branches.') } + + # Cobertura jump conditions describe two outcomes, not which outcome a + # 50% observation took. Keep that ambiguity even when condition ids match. + $conditions = New-CoverageMap + $nodes = @($Node.SelectNodes('conditions/condition')) + $conditionSum = 0 + $usable = $nodes.Count -gt 0 -and $total -eq 2 * $nodes.Count + foreach ($condition in $nodes) { + $id = $condition.GetAttribute('number') + $coverage = $condition.GetAttribute('coverage') + if ($condition.GetAttribute('type') -ne 'jump' -or -not $id -or + $conditions.ContainsKey($id) -or $coverage -notmatch '^(0|50|100)(?:\.0+)?%$') { + $usable = $false + continue + } + $count = [int]$Matches[1] / 50 + $conditions[$id] = $count + $conditionSum += $count + } + if (-not $usable -or $conditionSum -ne $covered) { $conditions = $null } + $entry.Branches.Add(@{ Total = $total; Covered = $covered; Conditions = $conditions }) +} + +function Get-CoverageMeasure($Lines) { + $coveredLines = @($Lines.Values | Where-Object Covered).Count + $linePercent = if ($Lines.Count) { [math]::Round(100.0 * $coveredLines / $Lines.Count, 1) } else { $null } + $lower = 0L + $upper = 0L + $total = 0L + $unknown = $Lines.Count -eq 0 + foreach ($line in $Lines.Values) { + $observations = @($line.Branches) + if (-not $observations.Count) { continue } + if ($line.ConflictingBranches -or @($observations | Where-Object { $null -eq $_.Total }).Count) { + $unknown = $true + continue + } + $totals = @($observations | ForEach-Object { $_.Total } | Sort-Object -Unique) + if ($totals.Count -ne 1 -or $null -eq $totals[0]) { $unknown = $true; continue } + $capacity = $totals[0] + $total += $capacity + $useConditions = $true + $conditionIds = $null + foreach ($observation in $observations) { + if ($null -eq $observation.Conditions) { $useConditions = $false; break } + $ids = @($observation.Conditions.Keys | Sort-Object -CaseSensitive) + $identity = ConvertTo-Json -InputObject $ids -Compress + if ($null -ne $conditionIds -and $conditionIds -cne $identity) { $useConditions = $false; break } + $conditionIds = $identity + } + if ($useConditions) { + foreach ($id in $observations[0].Conditions.Keys) { + $counts = @($observations | ForEach-Object { $_.Conditions[$id] }) + $lower += ($counts | Measure-Object -Maximum).Maximum + $upper += [math]::Min(2, ($counts | Measure-Object -Sum).Sum) + } + } else { + $counts = @($observations | ForEach-Object { $_.Covered }) + $lower += ($counts | Measure-Object -Maximum).Maximum + $upper += [math]::Min($capacity, ($counts | Measure-Object -Sum).Sum) + } + } + $branchStatus = if ($unknown) { 'unknown' } elseif (-not $total) { 'not-applicable' } elseif ($lower -eq $upper) { 'exact' } else { 'bounded' } + return [pscustomobject]@{ + TotalLines = $Lines.Count + CoveredLines = $coveredLines + LineCoverage = $linePercent + LineCoverageStatus = $(if ($Lines.Count) { 'exact' } else { 'unknown' }) + TotalBranches = $(if ($unknown) { $null } else { $total }) + CoveredBranches = $(if ($branchStatus -in @('exact', 'not-applicable')) { [long]$lower } else { $null }) + BranchCoverage = $(if ($branchStatus -eq 'exact') { [math]::Round(100.0 * $lower / $total, 1) } else { $null }) + BranchCoverageStatus = $branchStatus + CoveredBranchesLower = $(if (-not $unknown) { [long]$lower } else { $null }) + CoveredBranchesUpper = $(if (-not $unknown) { [long]$upper } else { $null }) + BranchCoverageLower = $(if (-not $unknown -and $total) { [math]::Round(100.0 * $lower / $total, 1) } else { $null }) + BranchCoverageUpper = $(if (-not $unknown -and $total) { [math]::Round(100.0 * $upper / $total, 1) } else { $null }) + } +} + +function Get-CoverageData([string[]]$Paths) { + if (-not $Paths.Count) { throw [ArgumentException]::new('At least one Cobertura report is required.') } + $methods = New-CoverageMap + $classLines = New-CoverageMap + $methodLines = New-CoverageMap + $inputs = [System.Collections.Generic.List[object]]::new() + $reports = [System.Collections.Generic.List[object]]::new() + $diagnostics = [System.Collections.Generic.List[object]]::new() + $seen = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + $allClassesHaveLines = $true + foreach ($path in ($Paths | Sort-Object -Unique -CaseSensitive)) { + $resolved = (Get-Item -LiteralPath $path -ErrorAction Stop).FullName + $bytes = [IO.File]::ReadAllBytes($resolved) + $hash = [Convert]::ToHexString([Security.Cryptography.SHA256]::HashData($bytes)).ToLowerInvariant() + $inputs.Add([pscustomobject]@{ path = $resolved; sha256 = $hash }) + if (-not $seen.Add($hash)) { continue } + $xml = [xml]::new() + $xml.XmlResolver = $null + $stream = [IO.MemoryStream]::new($bytes, $false) + try { $xml.Load($stream) } finally { $stream.Dispose() } + if ($xml.DocumentElement.Name -ne 'coverage') { throw [FormatException]::new("Not a Cobertura report: $path") } + $coverageRoot = $xml.DocumentElement + $report = @{ line = $null; branch = $null } + foreach ($metric in @('line', 'branch')) { + $prefix = if ($metric -eq 'line') { 'lines' } else { 'branches' } + $valid = $coverageRoot.GetAttribute("$prefix-valid") + $covered = $coverageRoot.GetAttribute("$prefix-covered") + if ($valid -or $covered) { + $denominator = Read-CoverageInteger $valid "$prefix-valid" + $numerator = Read-CoverageInteger $covered "$prefix-covered" + if ($numerator -gt $denominator) { throw [FormatException]::new("$prefix-covered exceeds $prefix-valid.") } + $report[$metric] = @{ Total = $denominator; Covered = $numerator } + } + } + $reports.Add($report) + $reportMethods = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + foreach ($class in $xml.SelectNodes('/coverage/packages/package/classes/class')) { + $filename = $class.GetAttribute('filename').Replace('\', '/') + $className = $class.GetAttribute('name') + $module = $class.ParentNode.ParentNode.GetAttribute('name') + if (-not $filename -or -not $className) { throw [FormatException]::new('Class name and filename are required.') } + $sourceKey = ConvertTo-Json -InputObject @($module, $filename) -Compress + $classNodes = @($class.SelectNodes('lines/line')) + if (-not $classNodes.Count) { $allClassesHaveLines = $false } + foreach ($line in $classNodes) { Add-CoverageLine $classLines $line $sourceKey $hash } + foreach ($method in $class.SelectNodes('methods/method')) { + $name = $method.GetAttribute('name') + if (-not $name) { throw [FormatException]::new('Method name is required.') } + $signature = $method.GetAttribute('signature') + $key = ConvertTo-Json -InputObject @($module, $filename, $className, $name, $signature) -Compress + if (-not $reportMethods.Add($key)) { + throw [FormatException]::new("Duplicate method identity for $className.$name; signatures or source mapping are ambiguous.") + } + if (-not $methods.ContainsKey($key)) { + $methods[$key] = @{ + Module = $module; Class = $className; Method = $name; Signature = $signature; File = $filename + Lines = (New-CoverageMap) + Complexities = [System.Collections.Generic.HashSet[long]]::new() + } + } + $entry = $methods[$key] + $complexity = $method.GetAttribute('complexity') + if ($complexity) { [void]$entry.Complexities.Add((Read-CoverageInteger $complexity 'complexity' 1)) } + foreach ($line in $method.SelectNodes('lines/line')) { + Add-CoverageLine $entry.Lines $line '' $hash + Add-CoverageLine $methodLines $line $sourceKey $hash + } + } + } + } + $result = [System.Collections.Generic.List[object]]::new() + foreach ($entry in $methods.Values) { + $measure = Get-CoverageMeasure $entry.Lines + $complexity = if ($entry.Complexities.Count -eq 1) { @($entry.Complexities)[0] } else { $null } + $item = [ordered]@{ + Class = $entry.Class; Method = $entry.Method; Signature = $entry.Signature; File = $entry.File + Module = $entry.Module; Complexity = $complexity + ComplexityStatus = $(if ($null -ne $complexity) { 'exact' } else { 'unknown' }) + } + foreach ($property in $measure.PSObject.Properties) { $item[$property.Name] = $property.Value } + $item.UncoveredLines = $measure.TotalLines - $measure.CoveredLines + if ($null -eq $complexity) { + Add-CoverageDiagnostic $diagnostics 'COMPLEXITY_UNKNOWN' "Missing or conflicting complexity for $($entry.Class).$($entry.Method)." $entry.File + } + if ($measure.LineCoverageStatus -eq 'unknown') { + Add-CoverageDiagnostic $diagnostics 'METHOD_LINES_UNKNOWN' "No executable-line evidence for $($entry.Class).$($entry.Method)." $entry.File + } + if ($measure.BranchCoverageStatus -in @('unknown', 'bounded')) { + Add-CoverageDiagnostic $diagnostics 'BRANCH_UNION_UNCERTAIN' "Branch outcomes cannot be uniquely merged for $($entry.Class).$($entry.Method)." $entry.File + } + $result.Add([pscustomobject]$item) + } + $aggregate = Get-CoverageMeasure $classLines + $aggregateScope = 'class-lines' + if (-not $allClassesHaveLines -or -not $classLines.Count) { + $aggregateScope = if ($reports.Count -eq 1) { 'report' } else { 'method-lines-only' } + $aggregate = [pscustomobject]@{ LineCoverage = $null; LineCoverageStatus = 'unknown'; BranchCoverage = $null; BranchCoverageStatus = 'unknown' } + if ($reports.Count -eq 1) { + foreach ($metric in @('line', 'branch')) { + $counter = $reports[0][$metric] + $name = if ($metric -eq 'line') { 'LineCoverage' } else { 'BranchCoverage' } + if ($null -ne $counter) { + $aggregate.$name = if ($counter.Total) { [math]::Round(100.0 * $counter.Covered / $counter.Total, 1) } else { $null } + $aggregate."${name}Status" = if ($counter.Total) { 'exact' } else { 'not-applicable' } + } + } + } + } + if (-not $result.Count) { + Add-CoverageDiagnostic $diagnostics 'METHODS_UNAVAILABLE' 'The report has no method-level coverage evidence.' '' + } + return [pscustomobject]@{ + Methods = @($result | Sort-Object -CaseSensitive Module, File, Class, Method, Signature) + Inputs = @($inputs) + Diagnostics = $diagnostics + Summary = [ordered]@{ + overall_line_coverage = $aggregate.LineCoverage + overall_line_coverage_status = $aggregate.LineCoverageStatus + overall_branch_coverage = $aggregate.BranchCoverage + overall_branch_coverage_status = $aggregate.BranchCoverageStatus + coverage_scope = $aggregateScope + method_line_coverage = (Get-CoverageMeasure $methodLines).LineCoverage + total_methods = $result.Count + } + } +} + +function Write-CoverageJson([string]$Tool, [string]$Status, $Scope, $Inputs, $Diagnostics, $Data) { + [ordered]@{ + schema_version = '1'; tool = $Tool; tool_version = '1.0'; status = $Status + scope = $Scope; inputs = @($Inputs); diagnostics = @($Diagnostics); data = $Data + } | ConvertTo-Json -Depth 30 +} + +function Write-CoverageDiagnostics($Diagnostics) { + foreach ($diagnostic in $Diagnostics) { + [Console]::Error.WriteLine("$($diagnostic.code): $($diagnostic.message)") + } +} diff --git a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/Extract-MethodCoverage.ps1 b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/Extract-MethodCoverage.ps1 index 999a8273..4e2f68fb 100644 --- a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/Extract-MethodCoverage.ps1 +++ b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/coverage-analysis/scripts/Extract-MethodCoverage.ps1 @@ -1,193 +1,65 @@ -param( - [Parameter(Mandatory=$true)] - [string[]]$CoberturaPath, - - [Parameter(Mandatory=$false)] - [int]$CoverageThreshold = 80, - - [Parameter(Mandatory=$false)] - [int]$BranchThreshold = 70, - - [Parameter(Mandatory=$false)] - [ValidateSet('uncovered', 'below-threshold', 'all')] - [string]$Filter = 'all' -) - <# .SYNOPSIS -Extract method-level coverage from Cobertura XML and output as JSON. - +Extract Cobertura method coverage without inventing missing measurements. .DESCRIPTION -Parses one or more Cobertura code coverage XML files and extracts per-method coverage metrics: -- Method name and class -- Line coverage percentage -- Branch coverage percentage -- Lines covered / total -- Branches covered / total -- Complexity (if available) - -When multiple files are provided, line hits are merged across files so a line is counted -as covered if any test project covered it. - -Filters by coverage status (uncovered, below threshold, or all). -Output is JSON for easy post-processing into tables, CSV, or other formats. - -.PARAMETER CoberturaPath -Path(s) to Cobertura coverage.cobertura.xml file(s). Accepts multiple paths for multi-test-project merging. - -.PARAMETER CoverageThreshold -Minimum acceptable line coverage percentage. Methods below this threshold are flagged (default: 80). - -.PARAMETER BranchThreshold -Minimum acceptable branch coverage percentage for methods that contain branches (default: 70). - -.PARAMETER Filter -Which methods to include: - 'uncovered' - methods with 0% coverage only - 'below-threshold' - methods with line coverage < CoverageThreshold OR branch coverage < BranchThreshold (for methods with branches) - 'all' - all methods (default) - -.EXAMPLE -PS> & .\Extract-MethodCoverage.ps1 -CoberturaPath "coverage.cobertura.xml" -CoverageThreshold 80 -BranchThreshold 70 -Filter uncovered -Outputs a JSON array of uncovered methods. - -.EXAMPLE -PS> & .\Extract-MethodCoverage.ps1 -CoberturaPath @("tests1/coverage.cobertura.xml","tests2/coverage.cobertura.xml") -Merges coverage from multiple test projects and outputs combined method-level metrics. - -.OUTPUTS -Writes JSON array to stdout. -Sets exit code 0 on success, 2 on missing/invalid file. +Legacy output is a JSON array; diagnostics and summaries go to stderr. +Use -OutputFormat json-v1 for provenance, null/unknown metrics and branch bounds. +Merging is scoped to reported module/file/class/signature identities, not an +attestation of source revision. Duplicate report bytes are idempotent. +Exit codes: 0 complete, 2 invalid input, 3 incomplete/unrepresentable evidence. #> +param( + [Parameter(Mandatory=$true)][string[]]$CoberturaPath, + [ValidateRange(0,100)][int]$CoverageThreshold = 80, + [ValidateRange(0,100)][int]$BranchThreshold = 70, + [ValidateSet('uncovered', 'below-threshold', 'all')][string]$Filter = 'all', + [ValidateSet('legacy', 'json-v1')][string]$OutputFormat = 'legacy' +) -foreach ($p in $CoberturaPath) { - if (-not (Test-Path $p)) { - Write-Error "Cobertura file not found: $p" - exit 2 +try { + . "$PSScriptRoot\CoverageData.ps1" +} catch { + $diagnostic = @{ code = 'DEPENDENCY_UNAVAILABLE'; severity = 'error'; message = $_.Exception.Message; path = "$PSScriptRoot\CoverageData.ps1" } + [Console]::Error.WriteLine("$($diagnostic.code): $($diagnostic.message)") + if ($OutputFormat -eq 'json-v1') { + @{ + schema_version = '1'; tool = 'Extract-MethodCoverage'; tool_version = '1.0'; status = 'error' + scope = @{}; inputs = @(); diagnostics = @($diagnostic); data = @{} + } | ConvertTo-Json -Depth 5 } + exit 2 } - -# Merge methods across all Cobertura files using a stable key (Class|Method|Signature|File). -# Line hits and branch data are accumulated so coverage reflects all test projects. -$methodMap = @{} - -foreach ($p in $CoberturaPath) { - try { - [xml]$xml = Get-Content $p -Encoding UTF8 -ErrorAction Stop - } catch { - Write-Error "Failed to parse Cobertura XML: $_" - exit 2 - } - - foreach ($package in $xml.coverage.packages.package) { - foreach ($class in $package.classes.class) { - $className = $class.name - $classFilename = $class.filename - - foreach ($method in $class.methods.method) { - $key = "$className|$($method.name)|$($method.signature)|$classFilename" - - if (-not $methodMap.ContainsKey($key)) { - $complexity = if ($null -ne $method.complexity) { [int]$method.complexity } else { 1 } - if ($complexity -lt 1) { $complexity = 1 } - $methodMap[$key] = @{ - Class = $className - Method = $method.name - Signature = $method.signature - File = $classFilename - Complexity = $complexity - LineHits = @{} - BranchData = @{} - } - } - - # Accumulate line hits across files - foreach ($line in $method.lines.line) { - $lineNo = $line.number - $hits = [int]$line.hits - if ($methodMap[$key].LineHits.ContainsKey($lineNo)) { - $methodMap[$key].LineHits[$lineNo] += $hits - } else { - $methodMap[$key].LineHits[$lineNo] = $hits - } - - # Accumulate branch data - if ($line.branch -eq 'true' -and $line.'condition-coverage') { - if ($line.'condition-coverage' -match '\((\d+)/(\d+)\)') { - $covered = [int]$Matches[1] - $total = [int]$Matches[2] - if ($methodMap[$key].BranchData.ContainsKey($lineNo)) { - # Merge branch coverage across files by accumulating covered branches (capped at total) - $existingCovered = $methodMap[$key].BranchData[$lineNo].Covered - $existingTotal = $methodMap[$key].BranchData[$lineNo].Total - if ($existingTotal -ne $total) { - Write-Warning ("Branch total mismatch for {0} at line {1}: {2} vs {3}" -f $key, $lineNo, $existingTotal, $total) - } - $mergedTotal = [Math]::Max($existingTotal, $total) - $mergedCovered = [Math]::Min($existingCovered + $covered, $mergedTotal) - $methodMap[$key].BranchData[$lineNo] = @{ Covered = $mergedCovered; Total = $mergedTotal } - } else { - $methodMap[$key].BranchData[$lineNo] = @{ Covered = $covered; Total = $total } - } - } - } - } - } - } - } +$scope = @{ identity = 'reported-module-file-class-signature'; source_revision = $null; filter = $Filter } +try { + $coverage = Get-CoverageData $CoberturaPath +} catch { + $diagnostics = @([pscustomobject]@{ code = 'INVALID_INPUT'; severity = 'error'; message = $_.Exception.Message }) + Write-CoverageDiagnostics $diagnostics + if ($OutputFormat -eq 'json-v1') { Write-CoverageJson 'Extract-MethodCoverage' 'error' $scope @() $diagnostics @{} } + exit 2 } - -$methods = [System.Collections.Generic.List[PSCustomObject]]::new() - -foreach ($entry in $methodMap.Values) { - $totalLines = $entry.LineHits.Count - $coveredLineCount = ($entry.LineHits.Values | Where-Object { $_ -gt 0 } | Measure-Object).Count - $lineCoveragePercent = if ($totalLines -gt 0) { [math]::Round(($coveredLineCount / $totalLines) * 100, 1) } else { 0 } - - $branchesTotal = 0 - $branchesCovered = 0 - foreach ($bd in $entry.BranchData.Values) { - $branchesCovered += $bd.Covered - $branchesTotal += $bd.Total - } - $branchCoveragePercent = if ($branchesTotal -gt 0) { [math]::Round(($branchesCovered / $branchesTotal) * 100, 1) } else { 0 } - - # Apply filter - if ($Filter -eq 'uncovered' -and $lineCoveragePercent -gt 0) { continue } - if ($Filter -eq 'below-threshold') { - $lineOk = $lineCoveragePercent -ge $CoverageThreshold - $branchOk = ($branchesTotal -eq 0) -or ($branchCoveragePercent -ge $BranchThreshold) - if ($lineOk -and $branchOk) { continue } +$partial = -not $coverage.Methods.Count -or @($coverage.Methods | Where-Object { + $_.LineCoverageStatus -eq 'unknown' -or $_.BranchCoverageStatus -in @('unknown', 'bounded') -or $_.ComplexityStatus -eq 'unknown' +}).Count -gt 0 +$methods = @($coverage.Methods | Where-Object { + if ($Filter -eq 'all') { return $true } + if ($_.LineCoverageStatus -eq 'unknown' -or $_.BranchCoverageStatus -in @('unknown', 'bounded')) { return $true } + if ($Filter -eq 'uncovered') { return $_.LineCoverage -eq 0 } + return (100.0 * $_.CoveredLines / $_.TotalLines) -lt $CoverageThreshold -or + ($_.TotalBranches -gt 0 -and (100.0 * $_.CoveredBranches / $_.TotalBranches) -lt $BranchThreshold) +} | Sort-Object -Property @{Expression='UncoveredLines';Descending=$true}, @{Expression='LineCoverage';Descending=$false}, Class, Method, File, Signature) +Write-CoverageDiagnostics $coverage.Diagnostics +if ($OutputFormat -eq 'json-v1') { + Write-CoverageJson 'Extract-MethodCoverage' $(if ($partial) { 'partial' } else { 'complete' }) $scope $coverage.Inputs $coverage.Diagnostics @{ + methods = $methods; summary = $coverage.Summary } - - $methods.Add([PSCustomObject]@{ - Class = $entry.Class - Method = $entry.Method - Signature = $entry.Signature - File = $entry.File - Complexity = $entry.Complexity - LineCoverage = $lineCoveragePercent - BranchCoverage = $branchCoveragePercent - CoveredLines = $coveredLineCount - TotalLines = $totalLines - UncoveredLines = ($totalLines - $coveredLineCount) - CoveredBranches = $branchesCovered - TotalBranches = $branchesTotal - }) +} elseif (-not $partial) { + $legacy = @($methods | Select-Object Class, Method, Signature, File, Complexity, LineCoverage, + @{Name='BranchCoverage';Expression={if ($_.BranchCoverageStatus -eq 'not-applicable') { 0 } else { $_.BranchCoverage }}}, + CoveredLines, TotalLines, UncoveredLines, CoveredBranches, TotalBranches) + ConvertTo-Json -InputObject $legacy -Depth 5 + [Console]::Error.WriteLine("METHODS_FILTERED:$($methods.Count)") + [Console]::Error.WriteLine("UNCOVERED_METHODS:$(@($methods | Where-Object { $_.LineCoverageStatus -eq 'exact' -and $_.LineCoverage -eq 0 }).Count)") } -# Sort by uncovered lines descending, then by line coverage ascending -$sorted = $methods | Sort-Object -Property @{Expression='UncoveredLines';Descending=$true}, @{Expression='LineCoverage';Descending=$false}, Class, Method - -# Output as JSON (empty array guard for zero results) -if ($sorted.Count -eq 0) { - Write-Output "[]" -} else { - $json = @($sorted) | ConvertTo-Json - Write-Output $json -} - -# Summary -Write-Host "METHODS_FILTERED:$($methods.Count)" -ForegroundColor Green -$uncovered = $methods | Where-Object { $_.LineCoverage -eq 0 } | Measure-Object | Select-Object -ExpandProperty Count -Write-Host "UNCOVERED_METHODS:$uncovered" -ForegroundColor $(if ($uncovered -gt 0) { 'Yellow' } else { 'Green' }) +if ($partial) { exit 3 } exit 0 diff --git a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/find-untested-sources/scripts/Find-UntestedSources.cs b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/find-untested-sources/scripts/Find-UntestedSources.cs index 9feb7680..e2f28b03 100644 --- a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/find-untested-sources/scripts/Find-UntestedSources.cs +++ b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/find-untested-sources/scripts/Find-UntestedSources.cs @@ -1,4 +1,5 @@ #:package Microsoft.CodeAnalysis.CSharp@4.14.0 +#:property PublishAot=false // Find-UntestedSources.cs // @@ -37,48 +38,63 @@ // } using System.Collections.Concurrent; +using System.Security.Cryptography; +using System.Text; using System.Text.Json; using Microsoft.CodeAnalysis; using Microsoft.CodeAnalysis.CSharp; using Microsoft.CodeAnalysis.CSharp.Syntax; +bool jsonV1 = args.Contains("json-v1", StringComparer.Ordinal); +int topN = int.MaxValue; +bool includeTested = false; if (args.Length < 1) { - Console.Error.WriteLine("usage: dotnet run Find-UntestedSources.cs -- [--top N]"); + Console.Error.WriteLine("usage: dotnet run Find-UntestedSources.cs -- [--top N] [--include-tested] [--output-format legacy|json-v1]"); return 1; } -string root = Path.GetFullPath(args[0]); -if (!Directory.Exists(root)) +for (int i = 1; i < args.Length; i++) { - Console.Error.WriteLine($"Not a directory: {root}"); - return 1; + if (args[i] == "--include-tested") { includeTested = true; continue; } + if (args[i] == "--output-format" && i + 1 < args.Length && args[i + 1] is "legacy" or "json-v1") + { + jsonV1 = args[++i] == "json-v1"; + continue; + } + if ((args[i] == "--top" || args[i] == "--limit-untested") && i + 1 < args.Length && + int.TryParse(args[i + 1], out int n) && n >= 0) + { + topN = n == 0 ? int.MaxValue : n; + i++; + continue; + } + return InputError($"Invalid argument: {args[i]}", jsonV1); } -int topN = int.MaxValue; -for (int i = 1; i < args.Length - 1; i++) +string root; +try { root = Path.GetFullPath(args[0]); } +catch (Exception ex) when (ex is ArgumentException or NotSupportedException or IOException) { - if (args[i] == "--top" && int.TryParse(args[i + 1], out var n)) - { - topN = n; - } + return InputError(ex.Message, jsonV1); } +if (!Directory.Exists(root)) { return InputError($"Not a directory: {root}", jsonV1); } var swTotal = System.Diagnostics.Stopwatch.StartNew(); // 1. Discover .cs files var allCs = new List(); -foreach (var f in EnumerateCsFiles(root)) +foreach (var f in EnumerateSourceFiles(root, "*.cs")) { allCs.Add(f); } Log($"Discovered {allCs.Count} .cs files"); // 2. Classify by walking up to nearest .csproj -var projectForDir = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); -var isTestProject = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); +var projectForDir = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); +var isTestProject = new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase); -string? ProjectFor(string filePath) +ProjectLookup ProjectFor(string filePath) { var dir = Path.GetDirectoryName(filePath)!; return projectForDir.GetOrAdd(dir, d => @@ -88,35 +104,51 @@ { string[] csprojs; try { csprojs = Directory.GetFiles(cur, "*.csproj"); } - catch { csprojs = Array.Empty(); } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Report("PROJECT_DISCOVERY_FAILED", cur, ex.Message); + return new ProjectLookup(null, false); + } if (csprojs.Length > 0) { - // Sort to keep selection deterministic when multiple .csproj - // files coexist in the same directory. - Array.Sort(csprojs, StringComparer.Ordinal); - return csprojs[0]; + if (csprojs.Length > 1) + { + Report("PROJECT_AMBIGUOUS", cur, "Multiple nearest project files; classification is unresolved."); + return new ProjectLookup(null, false); + } + return new ProjectLookup(csprojs[0], true); } cur = Path.GetDirectoryName(cur); } - return null; + return new ProjectLookup(null, true); }); } -bool IsTest(string filePath) +bool? IsTest(string filePath) { - var proj = ProjectFor(filePath); - if (proj is null) + var lookup = ProjectFor(filePath); + if (!lookup.IsResolved) + { + return null; + } + if (lookup.Path is null) { return false; } - return isTestProject.GetOrAdd(proj, ClassifyProjectAsTest); + return isTestProject.GetOrAdd(lookup.Path, ClassifyProjectAsTest); } var sourceFiles = new List(); var testFiles = new List(); foreach (var f in allCs) { - if (IsTest(f)) + var classification = IsTest(f); + if (classification is null) + { + Report("CLASSIFICATION_UNKNOWN", f, "Nearest project discovery or classification could not be completed."); + continue; + } + if (classification.Value) { testFiles.Add(f); } @@ -134,11 +166,8 @@ bool IsTest(string filePath) var swIndex = System.Diagnostics.Stopwatch.StartNew(); Parallel.ForEach(sourceFiles, file => { - string text; - try { text = File.ReadAllText(file); } - catch { return; } - var tree = CSharpSyntaxTree.ParseText(text); - var rootNode = tree.GetRoot(); + var rootNode = ReadSyntax(file); + if (rootNode is null) { return; } int count = 0; foreach (var node in rootNode.DescendantNodes()) { @@ -176,11 +205,8 @@ bool IsTest(string filePath) var swScan = System.Diagnostics.Stopwatch.StartNew(); Parallel.ForEach(testFiles, file => { - string text; - try { text = File.ReadAllText(file); } - catch { return; } - var tree = CSharpSyntaxTree.ParseText(text); - var rootNode = tree.GetRoot(); + var rootNode = ReadSyntax(file); + if (rootNode is null) { return; } var usings = new HashSet(StringComparer.Ordinal); foreach (var u in rootNode.DescendantNodes().OfType()) @@ -250,7 +276,7 @@ bool IsTest(string filePath) // 5. Build suggested test paths for untested files var untested = sourceToTests - .Where(kv => kv.Value.IsEmpty) + .Where(kv => kv.Value.IsEmpty && !AnalysisState.FailedFiles.ContainsKey(kv.Key)) .Select(kv => kv.Key) .OrderBy(s => s, StringComparer.Ordinal) .ToList(); @@ -260,8 +286,9 @@ bool IsTest(string filePath) string? SuggestTestPath(string srcFile) { - var proj = ProjectFor(srcFile); - if (proj is null) + var lookup = ProjectFor(srcFile); + var proj = lookup.Path; + if (!lookup.IsResolved || proj is null) { return null; } @@ -323,15 +350,80 @@ bool IsTest(string filePath) WriteIndented = true, TypeInfoResolver = new System.Text.Json.Serialization.Metadata.DefaultJsonTypeInfoResolver(), }; -Console.WriteLine(JsonSerializer.Serialize(result, opts)); +var diagnostics = AnalysisState.Diagnostics + .OrderBy(d => d.path, StringComparer.Ordinal).ThenBy(d => d.code, StringComparer.Ordinal) + .ThenBy(d => d.message, StringComparer.Ordinal).Distinct().ToArray(); +foreach (var diagnostic in diagnostics) { Console.Error.WriteLine($"{diagnostic.code}: {diagnostic.path}: {diagnostic.message}"); } +if (jsonV1) +{ + var declarations = declsByShort.Values.SelectMany(values => values) + .GroupBy(d => ToRel(root, d.File), StringComparer.Ordinal) + .ToDictionary(g => g.Key, g => g.Select(d => d.ShortName).Distinct(StringComparer.Ordinal).Order(StringComparer.Ordinal).ToArray(), StringComparer.Ordinal); + var pairedTests = pairedReport.Values.SelectMany(tests => tests).ToHashSet(StringComparer.Ordinal); + var orphanTests = testFiles.Where(file => !AnalysisState.FailedFiles.ContainsKey(file) && !pairedTests.Contains(ToRel(root, file))) + .Order(StringComparer.Ordinal).Select(file => new { path = ToRel(root, file), language = "csharp" }).ToArray(); + var unassessed = allCs.Where(AnalysisState.FailedFiles.ContainsKey).Order(StringComparer.Ordinal) + .Select(file => new { path = ToRel(root, file), language = "csharp" }).ToArray(); + var data = new Dictionary + { + ["repo_root"] = root, + ["summary"] = new + { + files = allCs.Count, source_files = sourceFiles.Count, test_files = testFiles.Count, + tested_source_files = pairedReport.Count, untested_source_files = untested.Count, + orphan_test_files = orphanTests.Length, unassessed_files = unassessed.Length, + languages = allCs.Count > 0 ? new[] { "csharp" } : Array.Empty(), + }, + ["untested_sources"] = untestedReport.Select(entry => new + { + path = entry.source, language = "csharp", declaration_count = entry.decl_count, + declarations = declarations.GetValueOrDefault(entry.source, Array.Empty()), + suggested_test_path = entry.suggested_test_path, + }).ToArray(), + ["orphan_tests"] = orphanTests, + ["unassessed_files"] = unassessed, + ["test_regions"] = Array.Empty(), + }; + if (includeTested) + { + data["tested_sources"] = pairedReport.Select(entry => new + { + path = entry.Key, language = "csharp", + declaration_count = declsByFile.GetValueOrDefault(Path.GetFullPath(Path.Combine(root, entry.Key))), + declarations = declarations.GetValueOrDefault(entry.Key, Array.Empty()), + covering_tests = entry.Value, + }).ToArray(); + } + var envelope = new + { + schema_version = "1", tool = "Find-UntestedSources", tool_version = "1.0", + status = diagnostics.Length > 0 ? "partial" : "complete", + scope = new + { + root, source_revision = (string?)null, evidence = "static-reference-pairing", + engine = "Roslyn", engine_version = typeof(CSharpSyntaxTree).Assembly.GetName().Version?.ToString(), + declarations = "C# type and delegate syntax declarations; not behavioral coverage", + languages = new[] { "csharp" }, exclude_symlinks = true, + untested_limit = topN == int.MaxValue ? 0 : topN, truncated = untestedReport.Count < untested.Count, + }, + inputs = AnalysisState.Inputs.OrderBy(entry => entry.Key, StringComparer.Ordinal) + .Select(entry => new { path = ToRel(root, entry.Key), sha256 = entry.Value }).ToArray(), + diagnostics, data, + }; + Console.WriteLine(JsonSerializer.Serialize(envelope, opts)); +} +else if (diagnostics.Length == 0) +{ + Console.WriteLine(JsonSerializer.Serialize(result, opts)); +} Log($"Total time: {swTotal.ElapsedMilliseconds} ms"); -return 0; +return diagnostics.Length > 0 ? 3 : 0; // ============================================================================ // Helpers // ============================================================================ -static IEnumerable EnumerateCsFiles(string root) +static IEnumerable EnumerateSourceFiles(string root, string pattern) { var stack = new Stack(); stack.Push(root); @@ -339,10 +431,15 @@ static IEnumerable EnumerateCsFiles(string root) { var dir = stack.Pop(); IEnumerable subdirs; - try { subdirs = Directory.EnumerateDirectories(dir); } - catch { continue; } + try { subdirs = Directory.GetDirectories(dir); } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Report("WALK_FAILED", dir, ex.Message); + continue; + } foreach (var sub in subdirs) { + if (IsLink(sub)) { continue; } var name = Path.GetFileName(sub); if (name.Length == 0) { @@ -361,11 +458,15 @@ static IEnumerable EnumerateCsFiles(string root) stack.Push(sub); } IEnumerable files; - try { files = Directory.EnumerateFiles(dir, "*.cs"); } - catch { continue; } + try { files = Directory.GetFiles(dir, pattern); } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Report("WALK_FAILED", dir, ex.Message); + continue; + } foreach (var f in files) { - if (IsSkippedFile(f)) + if (IsSkippedFile(f) || IsLink(f)) { continue; } @@ -394,20 +495,14 @@ static bool IsSkippedFile(string path) return false; } -static bool ClassifyProjectAsTest(string csproj) +static bool? ClassifyProjectAsTest(string csproj) { - var name = Path.GetFileNameWithoutExtension(csproj); - string[] testSuffixes = { ".Tests", ".Test", ".UnitTests", ".IntegrationTests", ".E2E", ".EndToEnd", ".Spec", ".Specs" }; - foreach (var suf in testSuffixes) - { - if (name.EndsWith(suf, StringComparison.OrdinalIgnoreCase)) - { - return true; - } - } try { - var content = File.ReadAllText(csproj); + var content = ReadSnapshot(csproj); + var name = Path.GetFileNameWithoutExtension(csproj); + string[] testSuffixes = { ".Tests", ".Test", ".UnitTests", ".IntegrationTests", ".E2E", ".EndToEnd", ".Spec", ".Specs" }; + if (testSuffixes.Any(suffix => name.EndsWith(suffix, StringComparison.OrdinalIgnoreCase))) { return true; } if (content.Contains("Microsoft.NET.Test.Sdk", StringComparison.OrdinalIgnoreCase)) { return true; } if (content.Contains("MSTest.Sdk", StringComparison.OrdinalIgnoreCase)) { return true; } if (content.Contains("Microsoft.Testing.Platform", StringComparison.OrdinalIgnoreCase)) { return true; } @@ -416,8 +511,10 @@ static bool ClassifyProjectAsTest(string csproj) if (content.Contains("NUnit", StringComparison.OrdinalIgnoreCase)) { return true; } if (content.Contains("TUnit", StringComparison.OrdinalIgnoreCase)) { return true; } } - catch + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or DecoderFallbackException) { + Report("PROJECT_READ_FAILED", csproj, ex.Message); + return null; } return false; } @@ -459,28 +556,24 @@ static bool IsUsingPrefix(HashSet usings, string ns) static Dictionary BuildProductionToTestProjectMap(string root) { var map = new Dictionary(StringComparer.OrdinalIgnoreCase); - List allCsproj; - try - { - allCsproj = Directory.EnumerateFiles(root, "*.csproj", SearchOption.AllDirectories).ToList(); - } - catch - { - return map; - } + List allCsproj = EnumerateSourceFiles(root, "*.csproj").ToList(); // Sort so the deterministic "first write wins" rule below selects the // same test project across runs/machines regardless of filesystem // enumeration order. allCsproj.Sort(StringComparer.Ordinal); foreach (var testProj in allCsproj) { - if (!ClassifyProjectAsTest(testProj)) + if (ClassifyProjectAsTest(testProj) != true) { continue; } string content; - try { content = File.ReadAllText(testProj); } - catch { continue; } + try { content = ReadSnapshot(testProj); } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or DecoderFallbackException) + { + Report("PROJECT_READ_FAILED", testProj, ex.Message); + continue; + } var testDir = Path.GetDirectoryName(testProj)!; var matches = System.Text.RegularExpressions.Regex.Matches(content, @" BuildProductionToTestProjectMap(string root) static string ToRel(string root, string path) { - try { return Path.GetRelativePath(root, path).Replace('\\', '/'); } - catch { return path; } + return Path.GetRelativePath(root, path).Replace('\\', '/'); } static void Log(string msg) => Console.Error.WriteLine($"[find-untested-sources] {msg}"); +static int InputError(string message, bool jsonV1) +{ + Console.Error.WriteLine($"INVALID_INPUT: {message}"); + if (jsonV1) + { + Console.WriteLine(JsonSerializer.Serialize(new + { + schema_version = "1", tool = "Find-UntestedSources", tool_version = "1.0", status = "error", + scope = new { }, inputs = Array.Empty(), data = new { }, + diagnostics = new[] { new { code = "INVALID_INPUT", severity = "error", message } }, + })); + } + return jsonV1 ? 2 : 1; +} + +static void Report(string code, string path, string message) +{ + AnalysisState.Diagnostics.Enqueue(new DiagnosticRecord(code, "warning", path, message)); + AnalysisState.FailedFiles[path] = 0; +} + +static bool IsLink(string path) +{ + try { return (File.GetAttributes(path) & FileAttributes.ReparsePoint) != 0; } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + Report("STAT_FAILED", path, ex.Message); + return true; + } +} + +static string ReadSnapshot(string path) +{ + byte[] bytes = File.ReadAllBytes(path); + string hash = Convert.ToHexString(SHA256.HashData(bytes)).ToLowerInvariant(); + if (AnalysisState.Inputs.TryGetValue(path, out string? previous) && previous != hash) + { + Report("INPUT_CHANGED", path, "Input changed during analysis."); + } + AnalysisState.Inputs[path] = hash; + using var stream = new MemoryStream(bytes, writable: false); + using var reader = new StreamReader(stream, new UTF8Encoding(false, true), detectEncodingFromByteOrderMarks: true); + return reader.ReadToEnd(); +} + +static SyntaxNode? ReadSyntax(string path) +{ + try + { + var tree = CSharpSyntaxTree.ParseText(ReadSnapshot(path)); + var errors = tree.GetDiagnostics().Where(d => d.Severity == DiagnosticSeverity.Error).Take(3).ToArray(); + if (errors.Length > 0) + { + Report("SYNTAX_ERROR", path, string.Join("; ", errors.Select(d => d.ToString()))); + return null; + } + return tree.GetRoot(); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or DecoderFallbackException) + { + Report("READ_FAILED", path, ex.Message); + return null; + } +} + internal readonly record struct Decl(string ShortName, string Namespace, string File); +internal readonly record struct ProjectLookup(string? Path, bool IsResolved); +internal sealed record DiagnosticRecord(string code, string severity, string path, string message); +internal static class AnalysisState +{ + internal static readonly ConcurrentQueue Diagnostics = new(); + internal static readonly ConcurrentDictionary Inputs = new(StringComparer.Ordinal); + internal static readonly ConcurrentDictionary FailedFiles = new(StringComparer.Ordinal); +} diff --git a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/find-untested-sources/scripts/find_untested_sources.py b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/find-untested-sources/scripts/find_untested_sources.py index 2ffd5e97..083446a2 100644 --- a/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/find-untested-sources/scripts/find_untested_sources.py +++ b/external-sources/upstreams/dotnet-skills/plugins/dotnet-test/skills/find-untested-sources/scripts/find_untested_sources.py @@ -1,6 +1,6 @@ """Polyglot source-to-test pairing analyzer using tree-sitter. -Given a repo root, identifies which source files are NOT covered by any test +Given a repo root, identifies which source files are NOT referenced by any test file via two complementary heuristics: 1. Identifier overlap: a test file declares (or references) the same name @@ -12,17 +12,20 @@ TypeScript, JavaScript, Go, Java, Rust, C#, Ruby, Kotlin, Swift, PowerShell, C++, ...). -Output: JSON to stdout matching the schema used by the C# `find-untested-sources` -skill, so the same prompt patterns can consume both tools. +Output: legacy engine-specific JSON, or a shared envelope with --output-format +json-v1. Pairing is heuristic static evidence, not behavioral coverage. Dependencies: - pip install tree-sitter-language-pack + pip install tree-sitter-language-pack==1.8.1 """ from __future__ import annotations import argparse +import hashlib +import importlib.metadata import json +import posixpath import re import sys from dataclasses import dataclass, field @@ -33,6 +36,7 @@ from tree_sitter_language_pack import ( ProcessConfig, detect_language_from_path, + get_parser, process, ) except ImportError as e: @@ -41,6 +45,12 @@ "Run: pip install tree-sitter-language-pack", file=sys.stderr, ) + if "json-v1" in sys.argv or "--output-format=json-v1" in sys.argv: + print(json.dumps({ + "schema_version": "1", "tool": "find_untested_sources", "tool_version": "1.0", + "status": "error", "scope": {}, "inputs": [], "data": {}, + "diagnostics": [{"code": "DEPENDENCY_MISSING", "severity": "error", "message": str(e)}], + })) raise SystemExit(2) from e @@ -89,7 +99,7 @@ "venv", "env", ".nuget", - "TestResults", + "testresults", "coverage", ".next", ".nuxt", @@ -211,6 +221,10 @@ class FileInfo: declarations: set[str] = field(default_factory=set) referenced_identifiers: set[str] = field(default_factory=set) imports: list[str] = field(default_factory=list) + is_source: bool = True + complete: bool = True + sha256: str | None = None + test_regions: list[dict] = field(default_factory=list) def __hash__(self) -> int: return id(self) @@ -223,27 +237,42 @@ def should_skip_filename(name: str) -> bool: return any(p.search(name) for p in SKIP_FILENAME_PATTERNS) -def walk_files(root: Path, lang_filter: set[str] | None = None) -> Iterable[Path]: +def diagnostic(diagnostics: list[dict], code: str, path: str, error: object) -> None: + diagnostics.append({"code": code, "severity": "warning", "path": path, "message": str(error)}) + + +def language_family(lang: str) -> str: + return "javascript" if lang in {"typescript", "tsx", "javascript"} else lang + + +def walk_files( + root: Path, lang_filter: set[str] | None = None, diagnostics: list[dict] | None = None +) -> Iterable[Path]: """Yield files under root, pruning common build/vendor directories.""" stack = [root] + if diagnostics is None: + diagnostics = [] while stack: current = stack.pop() try: - children = list(current.iterdir()) - except (PermissionError, OSError): + children = sorted(current.iterdir()) + except OSError as exc: + diagnostic(diagnostics, "WALK_FAILED", str(current), exc) continue for child in children: try: if child.is_symlink(): continue - except OSError: + is_dir, is_file = child.is_dir(), child.is_file() + except OSError as exc: + diagnostic(diagnostics, "STAT_FAILED", str(child), exc) continue - if child.is_dir(): + if is_dir: if child.name.lower() in PRUNE_DIRS: continue stack.append(child) continue - if not child.is_file(): + if not is_file: continue if should_skip_filename(child.name): continue @@ -257,55 +286,125 @@ def walk_files(root: Path, lang_filter: set[str] | None = None) -> Iterable[Path yield child -# Word-character regex used to harvest identifiers from a test file's source -# for the "identifier overlap" pairing strategy. Tree-sitter's `symbols` output -# only gives us declared names; references (`new Foo()`, `Foo.bar()`) won't -# appear, so we fall back to a simple word scan over the file body. -IDENTIFIER_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") +def syntax_nodes(root): + stack = [root] + while stack: + node = stack.pop() + if "comment" in node.kind(): + continue + yield node + stack.extend(node.named_child(i) for i in reversed(range(node.named_child_count()))) -def harvest_identifiers(text: str, lang: str) -> set[str]: - if lang == "powershell": - return set(re.findall(r"[A-Za-z_][A-Za-z0-9-]*", text)) - return set(IDENTIFIER_RE.findall(text)) +def node_text(node, source: bytes) -> str: + return source[node.start_byte():node.end_byte()].decode("utf-8") + + +def harvest_identifiers(text: str, lang: str, root=None, regions=None) -> set[str]: + root = root if root is not None else get_parser(lang).parse(text).root_node() + source = text.encode("utf-8") + kinds = { + "identifier", "type_identifier", "field_identifier", "property_identifier", + "constant", "simple_name", "simple_identifier", "command_name", + } + result = set() + for node in syntax_nodes(root): + if node.kind() not in kinds: + continue + if regions is not None and not any(start <= node.start_byte() < end for start, end in regions): + continue + value = node_text(node, source) + if re.fullmatch(r"[A-Za-z_][A-Za-z0-9_-]*", value): + result.add(value) + return result -def harvest_declarations(text: str, lang: str) -> set[str]: +def harvest_declarations(text: str, lang: str, root=None) -> set[str]: """Recover common declarations when the language-pack symbol view is sparse.""" + if lang not in {"powershell", "cpp"}: + return set() + root = root if root is not None else get_parser(lang).parse(text).root_node() + source = bytearray(text.encode("utf-8")) + stack = [root] + while stack: + node = stack.pop() + if "comment" in node.kind() or "string" in node.kind(): + source[node.start_byte():node.end_byte()] = b" " * (node.end_byte() - node.start_byte()) + continue + stack.extend(node.named_child(i) for i in range(node.named_child_count())) + code = source.decode("utf-8") if lang == "powershell": - return set(re.findall(r"(?im)^\s*function\s+([A-Za-z_][A-Za-z0-9-]*)", text)) + return set(re.findall(r"(?im)^\s*function\s+([A-Za-z_][A-Za-z0-9-]*)", code)) if lang == "cpp": return set( re.findall( r"\b(?:class|struct|enum(?:\s+class)?)\s+([A-Za-z_][A-Za-z0-9_]*)", - text, + code, ) ) return set() -def parse_file(path: Path, root: Path) -> FileInfo | None: +def rust_regions(root, source: bytes) -> tuple[list[tuple[int, int]], list[dict]]: + excluded = [] + tests = [] + for parent in syntax_nodes(root): + attributes = [] + for i in range(parent.named_child_count()): + node = parent.named_child(i) + kind = node.kind() + if kind == "attribute_item": + attributes.append(node_text(node, source)) + continue + if kind in {"mod_item", "function_item"}: + if any(re.search(r"#\[\s*cfg\s*\(\s*test\s*\)\s*\]", attr) for attr in attributes): + excluded.append((node.start_byte(), node.end_byte())) + if kind == "function_item" and any( + re.fullmatch(r"#\[\s*(?:\w+::)*test(?:\s*\([^]]*\))?\s*\]", attr) + for attr in attributes + ): + excluded.append((node.start_byte(), node.end_byte())) + name = node.child_by_field_name("name") + tests.append({ + "name": node_text(name, source), + "start_line": source[:node.start_byte()].count(b"\n") + 1, + "end_line": source[:node.end_byte()].count(b"\n") + 1, + "start_byte": node.start_byte(), "end_byte": node.end_byte(), + "kind": "inline-test", + }) + attributes = [] + return excluded, tests + + +def parse_file(path: Path, root: Path, diagnostics: list[dict] | None = None) -> FileInfo | None: rel_str = path.relative_to(root).as_posix() rel = PurePosixPath(rel_str) lang = detect_language_from_path(str(path)) if lang is None or lang not in SUPPORTED_LANGUAGES: return None - try: - text = path.read_text(encoding="utf-8", errors="replace") - except OSError: - return None - + if diagnostics is None: + diagnostics = [] is_test = is_test_path(rel, lang) - info = FileInfo(path=path, rel=rel, lang=lang, is_test=is_test) + info = FileInfo(path=path, rel=rel, lang=lang, is_test=is_test, is_source=not is_test) + try: + raw = path.read_bytes() + info.sha256 = hashlib.sha256(raw).hexdigest() + text = raw.decode("utf-8-sig") + except (OSError, UnicodeError) as exc: + info.complete = False + diagnostic(diagnostics, "READ_FAILED", rel_str, exc) + return info try: + syntax = get_parser(lang).parse(text).root_node() + if syntax.has_error(): + info.complete = False + diagnostic(diagnostics, "SYNTAX_ERROR", rel_str, "Parser reported missing or erroneous syntax.") cfg = ProcessConfig(language=lang, structure=True, imports=True, symbols=True) result = process(text, cfg) except Exception as exc: - # Emit a diagnostic so users can see when tree-sitter parsing silently - # degrades results (e.g. 0-declaration sources because the parser - # blew up on an unusual construct). Keep it short — one line per file. - print(f"WARN: tree-sitter parse failed for {rel_str} ({lang}): {exc}", file=sys.stderr) + info.complete = False + diagnostic(diagnostics, "PARSE_FAILED", rel_str, exc) return info # Top-level declarations: union of structure + symbols. The two views are @@ -331,7 +430,30 @@ def _kind_str(item: object) -> str: name = getattr(sym, "name", None) if name: info.declarations.add(name) - info.declarations.update(harvest_declarations(text, lang)) + info.declarations.update(harvest_declarations(text, lang, syntax)) + + reference_regions = None + if lang == "rust": + source = text.encode("utf-8") + excluded, info.test_regions = rust_regions(syntax, source) + if raw.startswith(b"\xef\xbb\xbf"): + for region in info.test_regions: + region["start_byte"] += 3 + region["end_byte"] += 3 + info.declarations = set() + for node in syntax_nodes(syntax): + if node.kind() not in {"function_item", "struct_item", "enum_item", "trait_item", "type_item", "const_item", "static_item"}: + continue + if any(start <= node.start_byte() < end for start, end in excluded): + continue + name = node.child_by_field_name("name") + if name is not None: + info.declarations.add(node_text(name, source)) + if info.test_regions: + info.is_test = True + info.is_source = not is_test and bool(info.declarations) + if not is_test: + reference_regions = excluded # Imports: keep the raw `source` field; we'll normalize per language. if getattr(result, "imports", None): @@ -339,11 +461,11 @@ def _kind_str(item: object) -> str: src = getattr(imp, "source", None) if src: info.imports.append(src) + if lang == "rust" and info.is_source and info.is_test: + info.imports.clear() - # For test files only, scan all identifier-like tokens — caller uses these - # to pair with source declarations by name. - if is_test: - info.referenced_identifiers = harvest_identifiers(text, lang) + if info.is_test: + info.referenced_identifiers = harvest_identifiers(text, lang, syntax, reference_regions) return info @@ -367,24 +489,19 @@ def _resolve_relative_js(test_rel: PurePosixPath, target: str) -> set[PurePosixP if target.startswith("/"): joined = PurePosixPath(target.lstrip("/")) else: - joined_str = (test_rel.parent / base).as_posix() - # Collapse any number of "/../" pairs, including chained ones - # like "a/b/../../c" → "c". The previous regex only collapsed a single - # occurrence so chained "../../foo" imports stayed partially - # un-normalized and failed to match indexed paths. - prev = None - while prev != joined_str: - prev = joined_str - joined_str = re.sub(r"(?:^|/)[^/]+/\.\./", "/", joined_str) - joined_str = joined_str.lstrip("/") - joined = PurePosixPath(joined_str) + joined = PurePosixPath(posixpath.normpath((test_rel.parent / base).as_posix())) + if ".." in joined.parts: + return set() # Try various extensions and /index suffix. + extensions = (".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs") + if joined.suffix in extensions: + return {joined} candidates = set() stems = [str(joined)] if str(joined).endswith("/index"): stems.append(str(joined)[: -len("/index")]) for stem in stems: - for ext in (".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs"): + for ext in extensions: candidates.add(PurePosixPath(stem + ext)) candidates.add(PurePosixPath(stem + "/index.ts")) candidates.add(PurePosixPath(stem + "/index.tsx")) @@ -484,7 +601,7 @@ def _extract_go_import_targets(import_text: str) -> set[str]: def _build_indexes(sources: list[FileInfo]) -> dict: """Build lookup indexes used to resolve test imports to source files.""" - by_rel: dict[str, FileInfo] = {s.rel.as_posix().lower(): s for s in sources} + by_rel: dict[str, FileInfo] = {s.rel.as_posix(): s for s in sources} by_decl: dict[str, list[FileInfo]] = {} for s in sources: for d in s.declarations: @@ -495,12 +612,12 @@ def _build_indexes(sources: list[FileInfo]) -> dict: # instead of O(#sources) per import target. by_filename: dict[str, list[FileInfo]] = {} for s in sources: - p = s.rel.as_posix().lower() + p = s.rel.as_posix() parts = p.split("/") for i in range(len(parts)): suffix = "/".join(parts[i:]) by_path_suffix.setdefault(suffix, []).append(s) - by_filename.setdefault(s.rel.name.lower(), []).append(s) + by_filename.setdefault(s.rel.name, []).append(s) return { "by_rel": by_rel, "by_decl": by_decl, @@ -516,7 +633,7 @@ def _resolve_test_imports(test: FileInfo, indexes: dict, lang: str) -> set[FileI by_path_suffix = indexes["by_path_suffix"] def add_candidate(path: PurePosixPath) -> None: - key = path.as_posix().lower() + key = path.as_posix() if key in by_rel: found.add(by_rel[key]) return @@ -543,7 +660,7 @@ def add_candidate(path: PurePosixPath) -> None: if not segs: continue last = segs[-1] - key = (last + ".go").lower() + key = last + ".go" for info in by_filename.get(key, ()): if not info.is_test: found.add(info) @@ -597,10 +714,10 @@ def _resolve_test_by_identifiers( if len(ident) < 4: continue sources = by_decl.get(ident) - if not sources: + if not sources or len(sources) != 1: continue for s in sources: - if s.lang == test.lang: + if language_family(s.lang) == language_family(test.lang): found.add(s) return found @@ -611,8 +728,8 @@ def build_pairings( ) -> tuple[dict[FileInfo, set[FileInfo]], list[FileInfo]]: """Return (source -> covering tests) and the list of orphan test files.""" by_lang_indexes: dict[str, dict] = {} - for lang in SUPPORTED_LANGUAGES: - lang_sources = [s for s in sources if s.lang == lang] + for lang in {language_family(lang) for lang in SUPPORTED_LANGUAGES}: + lang_sources = [s for s in sources if language_family(s.lang) == lang and s.complete] if lang_sources: by_lang_indexes[lang] = _build_indexes(lang_sources) @@ -620,7 +737,9 @@ def build_pairings( orphans: list[FileInfo] = [] for t in tests: - idx = by_lang_indexes.get(t.lang) + if not t.complete: + continue + idx = by_lang_indexes.get(language_family(t.lang)) if idx is None: orphans.append(t) continue @@ -653,7 +772,7 @@ def _test_filename(source: FileInfo) -> str: if lang == "java": return f"{stem}Test.java" if lang == "rust": - return f"{stem}_test.rs" + return "" if lang == "csharp": return f"{stem}Tests.cs" if lang == "ruby": @@ -674,11 +793,15 @@ def _suggest_test_path( sibling_test_dirs: dict[tuple[str, PurePosixPath], PurePosixPath], ) -> str: filename = _test_filename(source) + if source.lang == "rust": + # A sibling *_test.rs is not a Cargo target. Suggest the source file + # for an inline #[cfg(test)] module without creating or editing it. + return source.rel.as_posix() if not filename: return "" - language_family = "typescript" if source.lang in {"typescript", "tsx"} else source.lang - parent = sibling_test_dirs.get((language_family, source.rel.parent), source.rel.parent) + family = language_family(source.lang) + parent = sibling_test_dirs.get((family, source.rel.parent), source.rel.parent) return (parent / filename).as_posix() @@ -693,8 +816,8 @@ def build_output( tested: list[dict] = [] sibling_test_dirs: dict[tuple[str, PurePosixPath], PurePosixPath] = {} for paired_source, covering_tests in source_to_tests.items(): - language_family = "typescript" if paired_source.lang in {"typescript", "tsx"} else paired_source.lang - key = (language_family, paired_source.rel.parent) + family = language_family(paired_source.lang) + key = (family, paired_source.rel.parent) for test in covering_tests: candidate = test.rel.parent current = sibling_test_dirs.get(key) @@ -702,6 +825,8 @@ def build_output( sibling_test_dirs[key] = candidate for s in sources: + if not s.complete: + continue covering = sorted(source_to_tests.get(s, set()), key=lambda t: t.rel.as_posix()) entry = { "path": s.rel.as_posix(), @@ -714,23 +839,37 @@ def build_output( tested.append(entry) else: entry["suggested_test_path"] = _suggest_test_path(s, sibling_test_dirs) + if s.lang == "rust": + entry["suggestion_kind"] = "inline-test-module" untested.append(entry) return { "repo_root": str(repo_root), "summary": { + "files": len({s.rel for s in sources + tests}), "source_files": len(sources), "test_files": len(tests), "tested_source_files": len(tested), "untested_source_files": len(untested), "orphan_test_files": len(orphans), "languages": sorted({s.lang for s in sources} | {t.lang for t in tests}), + "unassessed_files": len({s.rel for s in sources + tests if not s.complete}), }, "untested_sources": sorted(untested, key=lambda e: (-e["declaration_count"], e["path"])), "tested_sources": sorted(tested, key=lambda e: e["path"]), "orphan_tests": [ {"path": t.rel.as_posix(), "language": t.lang} for t in sorted(orphans, key=lambda t: t.rel.as_posix()) ], + "test_regions": [ + {"path": t.rel.as_posix(), **region} + for t in sorted(tests, key=lambda t: t.rel.as_posix()) + for region in t.test_regions + ], + "unassessed_files": [ + {"path": info.rel.as_posix(), "language": info.lang} + for info in sorted(set(sources + tests), key=lambda info: info.rel.as_posix()) + if not info.complete + ], } @@ -751,6 +890,10 @@ def main() -> int: default=0, help="If > 0, truncate the untested_sources list to N entries.", ) + parser.add_argument( + "--output-format", choices=("legacy", "json-v1"), default="legacy", + help="Use json-v1 for versioned provenance and explicit partial results.", + ) parser.add_argument( "--include-tested", action="store_true", @@ -761,22 +904,33 @@ def main() -> int: root = args.root.resolve() if not root.is_dir(): print(f"ERROR: not a directory: {root}", file=sys.stderr) + if args.output_format == "json-v1": + print(json.dumps(error_report("INVALID_INPUT", f"Not a directory: {root}"))) + return 2 return 1 + if args.limit_untested < 0: + if args.output_format == "json-v1": + message = "--limit-untested must be nonnegative" + print(f"INVALID_INPUT: {message}", file=sys.stderr) + print(json.dumps(error_report("INVALID_INPUT", message))) + return 2 + parser.error("--limit-untested must be nonnegative") lang_filter = set(args.lang) if args.lang else None print(f"Scanning {root}...", file=sys.stderr) sources: list[FileInfo] = [] tests: list[FileInfo] = [] + diagnostics: list[dict] = [] parsed_count = 0 - for path in walk_files(root, lang_filter=lang_filter): - info = parse_file(path, root) + for path in walk_files(root, lang_filter=lang_filter, diagnostics=diagnostics): + info = parse_file(path, root, diagnostics) if info is None: continue parsed_count += 1 if info.is_test: tests.append(info) - else: + if info.is_source: sources.append(info) print(f"Parsed {parsed_count} files: {len(sources)} source, {len(tests)} test.", file=sys.stderr) @@ -788,10 +942,49 @@ def main() -> int: if not args.include_tested: output.pop("tested_sources", None) + for item in diagnostics: + print(f"{item['code']}: {item['path']}: {item['message']}", file=sys.stderr) + if args.output_format == "json-v1": + files = sorted(set(sources + tests), key=lambda item: item.rel.as_posix()) + output = { + "schema_version": "1", "tool": "find_untested_sources", "tool_version": "1.0", + "status": "partial" if diagnostics else "complete", + "scope": { + "root": str(root), "source_revision": None, "evidence": "static-reference-pairing", + "engine": "tree-sitter-language-pack", + "engine_version": importlib.metadata.version("tree-sitter-language-pack"), + "declarations": "syntax declarations; language-dependent, not a behavioral metric", + "languages": sorted(lang_filter or SUPPORTED_LANGUAGES), + "excluded_directories": sorted(PRUNE_DIRS), "exclude_symlinks": True, + "untested_limit": args.limit_untested, + "truncated": len(output["untested_sources"]) < output["summary"]["untested_source_files"], + }, + "inputs": [{"path": item.rel.as_posix(), "sha256": item.sha256} for item in files], + "diagnostics": diagnostics, "data": output, + } + elif diagnostics: + return 3 json.dump(output, sys.stdout, indent=2) sys.stdout.write("\n") - return 0 + return 3 if diagnostics else 0 + + +def error_report(code: str, message: str) -> dict: + return { + "schema_version": "1", "tool": "find_untested_sources", "tool_version": "1.0", + "status": "error", "scope": {}, "inputs": [], "data": {}, + "diagnostics": [{"code": code, "severity": "error", "message": message}], + } if __name__ == "__main__": - sys.exit(main()) + try: + sys.exit(main()) + except KeyboardInterrupt: + print("CANCELLED: analysis interrupted", file=sys.stderr) + sys.exit(130) + except Exception as exc: + print(f"TOOL_ERROR: {exc}", file=sys.stderr) + if "json-v1" in sys.argv or "--output-format=json-v1" in sys.argv: + print(json.dumps(error_report("TOOL_ERROR", str(exc)))) + sys.exit(2) diff --git a/external-sources/upstreams/roslynk/README.md b/external-sources/upstreams/roslynk/README.md index becee664..14360a4e 100644 --- a/external-sources/upstreams/roslynk/README.md +++ b/external-sources/upstreams/roslynk/README.md @@ -236,13 +236,13 @@ The skill directory is watched, so a fresh copy is picked up without restarting > - `find_definition`: Go to definition from a cursor position. Parameters: `solutionId`, `filePath`, `line`, `column` (1-based). > - `get_expression_info`: Compiler facts about the expression at a position: type and converted type, bound symbol and selected overload, nullability, constant value, implicit conversion, origin and doc summary; on a declared name, the declared symbol and its type. Parameters: `solutionId`, `filePath` (`.cs`/`.razor`/`.cshtml`), `line`, `column` (1-based). > - `get_symbol`: Identify a symbol and get its declaration. Parameters: `solutionId`, `symbolName` (fully-qualified). -> - `get_symbol_body`: Read a symbol's source text, body included. Parameters: `solutionId`, `symbolName` (fully-qualified), `includeLeadingTrivia`, `startLine` (page a long declaration; the response reports `truncated=Y`/`totalLines`/`nextStartLine`), `maxLines` (default 400), `part` (fetch one declaration of a partial type; oversized parts are listed with `omitted=Y` and line counts). +> - `get_symbol_body`: Read a symbol's source text, body included. Parameters: `solutionId`, `symbolName` (fully-qualified), `includeLeadingTrivia`, `startItem` (page a long declaration; the response reports `truncated=Y`/`count`/`nextStartItem`), `maxSize` (page size in characters; default the response budget), `part` (fetch one declaration of a partial type; oversized parts are listed with `omitted=Y` and line counts). > - `get_members`: List a type's members with their locations. Parameters: `solutionId`, `typeName`, `includeInherited`, `nameFilter`, `includeMethods`/`includeFields`/`includeProperties`/`includeEvents`/`includeNestedTypes`. > - `search_symbols`: Find symbols by partial name. Parameters: `solutionId`, `query`, `maxResults`. -> - `multi_query`: Run several of the read-only queries above in one call, against a single snapshot of the solution. Parameters: `solutionId`, `operations` (each names a tool and carries that tool's own parameters), optional `expectSnapshot`. The whole response stays within the server's response budget (80,000 characters by default); an over-budget slot is cut at a line boundary and marked `truncated=Y`, remaining slots return `error=Truncated` with `truncatedSlots=`. See the skill's multi_query section. For impact questions ("what uses X?", "what breaks if I change X?"), batch `find_references`, `get_callers`, `get_callees`, `find_implementations` and `get_type_hierarchy` in one call. +> - `multi_query`: Run several of the read-only queries above in one call, against a single snapshot of the solution. Parameters: `solutionId`, `operations` (each names a tool and carries that tool's own parameters), optional `expectSnapshot`. The whole response stays within the server's response budget (80,000 characters by default): every operation runs, whole results are delivered smallest-first until the budget is filled (in request order), and the rest are named once in the header as `retry=:` (size = the result's cost as a ratio of the budget, rounded up to 0.1). Sizes ≥ 1.0 must be re-sent alone (they page); sizes summing to ≤ 1.0 fit one continuation batch together. See the skill's multi_query section. For impact questions ("what uses X?", "what breaks if I change X?"), batch `find_references`, `get_callers`, `get_callees`, `find_implementations` and `get_type_hierarchy` in one call. > > **Relationships:** -> - `find_references`: Every usage of a symbol. Parameters: `solutionId`, `symbolName`, `maxResults`. +> - `find_references`: Every usage of a symbol. Parameters: `solutionId`, `symbolName`, `maxResults` (omit for every reference; an oversized result pages with `startItem`). > - `find_reads` / `find_writes`: Where a field, property or parameter is read or written, each location tagged `read`, `assign`, `compound`, `increment`, `ref`, `out` or `init`; `compound`, `increment` and `ref` appear in both. Parameters: `solutionId`, `symbolName` (a parameter is `Namespace.Type.Method:parameter`), `maxResults`. > - `get_callers`: Who calls a member (method, constructor, operator, property, indexer or event), overloads resolved, including calls the compiler inserts (`foreach`/`await` plumbing, `using` disposal, operators, conversions). Parameters: `solutionId`, `methodName`. > - `get_callees`: What a member calls - the inverse of `get_callers`; methods, constructors, property/event accessors and operators, overload-exact, with referenced-assembly callees grouped per assembly. Parameters: `solutionId`, `memberName`, `excludeExternal`. @@ -281,7 +281,7 @@ The skill directory is watched, so a fresh copy is picked up without restarting > `key=value` header block, a blank line, then a tab-indented outline; booleans are `Y`/`N`. Errors > are header-only: `error=Indexing` (retry), `Ambiguous`/`NotFound` (pick from the `candidate=` > lines), `Stale` (re-read and recompute), `Conflict` (re-run the discovery step). Watch for -> `truncated=Y` and raise `maxResults`. Leave defaulted parameters alone unless you need the +> `truncated=Y` and continue with `startItem` (or raise `maxResults`). Leave defaulted parameters alone unless you need the > non-default behaviour, and re-query rather than reusing an earlier response — the solution is > edited live. > diff --git a/external-sources/upstreams/roslynk/skills/roslynk/SKILL.md b/external-sources/upstreams/roslynk/skills/roslynk/SKILL.md index 8759a279..b5689822 100644 --- a/external-sources/upstreams/roslynk/skills/roslynk/SKILL.md +++ b/external-sources/upstreams/roslynk/skills/roslynk/SKILL.md @@ -32,7 +32,7 @@ Each tool's exact contract — parameters, output format, limits, error codes | Know what an expression means before editing it (type, overload chosen, nullability, constant, conversion) | `get_expression_info` | `var`, overloads, implicit conversions and nullable flow are invisible in the text | | Find implementations of an interface/abstract member | `find_implementations` | compiler's type graph | | See a type's members (and their local functions) / what a name refers to | `get_members` / `get_symbol` | compiler's view; correct across partial classes | -| Read a member's implementation (incl. a partial method's implementation part, and members only a generator declares) | `get_symbol_body` | returns the declaration verbatim, paged when long (`truncated=Y` → re-call with `startLine=`; a partial type's `omitted=Y` part is fetched with `part=`); a path marked `generated=Y` is virtual - read it with this tool, not from disk | +| Read a member's implementation (incl. a partial method's implementation part, and members only a generator declares) | `get_symbol_body` | returns the declaration verbatim, paged when long (`truncated=Y` → re-call with `startItem=`; a partial type's `omitted=Y` part is fetched with `part=`); a path marked `generated=Y` is virtual - read it with this tool, not from disk | | Explore base/derived types | `get_type_hierarchy` | includes referenced-assembly base types | | Find a symbol by partial name | `search_symbols` | compiler-declared symbols, generated code included | | Rename a symbol everywhere (incl. `.razor`/`.cshtml`) | `rename_symbol` | find-and-replace misses markup and same-named text; a symbol only a source generator declares is refused (`error=NotSupported`) - change the generator's input instead | @@ -50,7 +50,7 @@ All name arguments are fully-qualified (`Namespace.Type` or `Namespace.Type.Memb ## Combine tools, don't chain calls -When you need several facts before making a change, send them as **one `multi_query`** call instead of sequential calls: every operation runs against a single snapshot, so the facts can never disagree with each other. `multi_query` accepts only the read-only query tools, and each operation uses exactly that tool's own parameter names — a wrong key is `error=Invalid` naming the key, never silently ignored. The whole response stays within the server's response budget (about 80,000 characters by default; set `Roslynk:MaxResponseChars` to change it). Each operation gets what earlier slots left: `get_symbol_body` pages itself within its share (`truncated=Y`/`nextStartLine=` in its slot), and any other tool's oversized output is cut after its last whole line — its meta line gains `truncated=Y` and its body ends with a bracketed `[response budget exhausted: ...]` note naming the continuation. **Trust rule:** a slot whose *meta line* carries `truncated=Y` was cut by the batch budget, so its body is a prefix — do not derive tool-level paging (e.g. `nextStartLine`) from it; re-send that operation alone. Remaining slots are whole `error=Truncated` blocks with `truncatedSlots=`: re-send exactly those operations, passing the response's `snapshot=` value as `expectSnapshot`, so a continuation that would straddle a solution change is refused with `error=Stale` instead of mixing states. The envelope format is in the multi_query section of [references/tools.md](references/tools.md). +When you need several facts before making a change, send them as **one `multi_query`** call instead of sequential calls: every operation runs against a single snapshot, so the facts can never disagree with each other. `multi_query` accepts only the read-only query tools, and each operation uses exactly that tool's own parameter names — a wrong key is `error=Invalid` naming the key, never silently ignored. Every operation runs; results are then delivered whole, smallest first, until the response budget (about 80,000 characters) is filled, and operations whose result did not fit are named once in the header as `retry=:` (size = that result's cost as a ratio of the budget, rounded up to 0.1). Plan the continuation from the ratios: size ≥ 1.0 → re-send that operation alone (it pages with `nextStartItem=`); sizes summing to ≤ 1.0 → one continuation batch together; pass the response's `snapshot=` value as `expectSnapshot` so a continuation that would straddle a solution change is refused with `error=Stale` instead of mixing states. Delivered slots are always complete, so their own headers can be trusted. The envelope format is in the multi_query section of [references/tools.md](references/tools.md). **Impact analysis** — before renaming a symbol or changing a signature, and whenever someone asks "what uses X", "who calls X", "what breaks if I change X": batch `get_symbol` + `find_references` + `get_callers` + `get_callees` + `find_implementations` + `get_type_hierarchy` in one call, pointing each at the member or type you're changing (`get_callers` takes `methodName`, `get_callees` takes `memberName`, `get_type_hierarchy` takes `typeName`, the others `symbolName`). `get_callers` and `get_callees` are inverses — who calls this member, what does this member call — so alternating them walks a call chain one hop per query, including calls the compiler inserts (`foreach`/`await` plumbing, `using` disposal, operators and conversions); a caller inside an accessor is reported as its property, while `get_callees` names the accessor (`get_X`), so pass the property name back. Then follow each reference with `get_symbol_body` to read the calling convention before editing. Answer with Roslynk, never with grep over `*.cs`/`*.cshtml`/`*.razor` — text search both misses real usages and wrongly matches identically-named symbols. diff --git a/external-sources/upstreams/roslynk/skills/roslynk/references/tools.md b/external-sources/upstreams/roslynk/skills/roslynk/references/tools.md index 1acb54e0..687ab44a 100644 --- a/external-sources/upstreams/roslynk/skills/roslynk/references/tools.md +++ b/external-sources/upstreams/roslynk/skills/roslynk/references/tools.md @@ -63,7 +63,7 @@ container's parameter list, e.g. `N.Parent.Widget.Method1(int).localMethod(strin that form is what an `Ambiguous` candidate looks like. A bare name such as `inner` also resolves. Outlines report a local function's kind as `localfunction` and nest it under its containing member. -**Truncation.** Known-total tools emit `count=` and `truncated=Y` when capped; unknown-total scans (`find_dead_code`) emit only `truncated=Y`. Line-paged output (`get_symbol_body`) emits `truncated=Y`, `totalLines=` and `nextStartLine=` instead: re-call with `startLine=` to continue. Any result that would still exceed the server-wide response budget is cut after its last whole line and starts with `outputTruncated=Y` and `fullOutputChars=` — narrow the request (a filter, `maxResults` or a paging parameter) to see the rest. The budget defaults to about 80,000 characters and is set with `Roslynk:MaxResponseChars` (or the `Roslynk__MaxResponseChars` environment variable) in the daemon's host; the daemon reads it once at startup, so changing it requires a restart, and a shared daemon serves every client with one value (pick the smallest cap your clients need). +**Paging and truncation.** Tools whose result can be too large page by item: an oversized result carries `count=`, `truncated=Y` and `nextStartItem=` — call again with `startItem=` and the same other arguments for the next page. An item is one leaf record (a reference, caller, callee, implementor, diagnostic, member, candidate, search hit; for get_symbol_body one source line). Every page repeats the container lines open above its first item, so it reads standalone; the preamble counts against `maxSize` but not against `count=`. `maxSize` (characters; default the response budget, about 80,000 — set with `Roslynk:MaxResponseChars` or the `Roslynk__MaxResponseChars` environment variable, read once at daemon start) bounds a page; an item longer than a whole page is delivered as far as it fits and named `partialItem=` — the rest of that item is not retrievable. `startItem` past the end, or a `maxSize` too small to carry one item beside its container lines, is `error=Invalid`. `maxResults` bounds the result SET (a scan halts at the cap); paging walks that capped set and `count=` names its size — omit `maxResults` and page when you want everything. A result that fits whole carries no paging headers at all. The tools that cannot page (get_symbol, find_definition, get_expression_info, get_solution_status) cut an oversized result after its last whole line, starting with `outputTruncated=Y` and `fullOutputChars=`. **Defaults.** Don't pass a value for a defaulted parameter unless you need the non-default behavior. @@ -128,9 +128,9 @@ Output is header only, in this order (optional keys only when applicable): `solutionId`, `symbolName` (FQN, any symbol kind including members). Unambiguous source match → `#project`/`#path`/`#loc` headers + body containing the verbatim declaration text cut before the body (brace/`=>` excluded). Metadata symbol → `#source=metadata`, `#kind`, `#signature`, `#assembly`. Ambiguous → `error=Ambiguous` with one `candidate=` per match. Preferred over reading a file to identify a symbol. ### get_symbol_body -`solutionId`, `symbolName` (FQN, any declared symbol kind), `includeLeadingTrivia` (default false), `startLine` (default 1), `maxLines` (default 400), `part` (default 0). Returns the symbol's source text including its body, verbatim — original indentation, line endings and spacing, never reformatted. Headers `project`/`path`/`loc`, a blank line, then the text. `includeLeadingTrivia=true` extends the text (and the `loc`) back over the declaration's comments, XML docs and directives. A partial type, or a partial method/property/event, returns `parts=` and one `part=,project=...,path=...,loc=...,lines=` line per block; a partial member's parts are its defining declaration then its implementation, including one supplied by a source generator (`[GeneratedRegex]`, `[LoggerMessage]`, `[LibraryImport]`), whose part line ends `,generated=Y,generator=`. +`solutionId`, `symbolName` (FQN, any declared symbol kind), `includeLeadingTrivia` (default false), `startItem` (default 1), `maxSize` (characters; default the response budget), `part` (default 0). Returns the symbol's source text including its body, verbatim — original indentation, line endings and spacing, never reformatted. Headers `project`/`path`/`loc`, a blank line, then the text. `includeLeadingTrivia=true` extends the text (and the `loc`) back over the declaration's comments, XML docs and directives. A partial type, or a partial method/property/event, returns `parts=` and one `part=,project=...,path=...,loc=...,lines=` line per block; a partial member's parts are its defining declaration then its implementation, including one supplied by a source generator (`[GeneratedRegex]`, `[LoggerMessage]`, `[LibraryImport]`), whose part line ends `,generated=Y,generator=`. -Long text is paged, never cut silently: a call returns at most `maxLines` lines and stays within the response budget. Lines are counted from 1 at the first line of the returned text (its first comment line with `includeLeadingTrivia`), not in the file; for a Razor member that text is the generated C#, while `loc` stays the `#line`-mapped display span — the two coordinates are deliberately unrelated. A page that is not the whole text adds `truncated=Y`, `totalLines=` and `nextStartLine=`: re-call with `startLine=`, keeping `includeLeadingTrivia` and `part` unchanged; a `startLine` past the end is `error=Invalid` naming `totalLines`. There is no unlimited mode (the budget is the ceiling); a single line longer than a whole page's budget is delivered as far as it fits and named `partialLine=` — the rest of that line is not retrievable by paging. When a partial type's parts do not all fit, the result is the whole-or-omitted overview: `parts=`, `truncated=Y`, `omittedParts=`, each part that fits shown whole, the rest listed with their `lines=` count and `,omitted=Y` — fetch one with `part=`, which returns that part alone, paged as above with `parts=`/`part=` headers. `startLine` pages one part; combining it with several parts and no `part=` is `error=Invalid`. +Long text is paged, never cut silently: a call returns at most `maxSize` characters and stays within the response budget. Items are the text's lines, counted from 1 at the first line of the returned text (its first comment line with `includeLeadingTrivia`), not in the file; for a Razor member that text is the generated C#, while `loc` stays the `#line`-mapped display span — the two coordinates are deliberately unrelated. A page that is not the whole text adds `truncated=Y`, `count=` and `nextStartItem=`: re-call with `startItem=`, keeping `includeLeadingTrivia` and `part` unchanged; a `startItem` past the end is `error=Invalid` naming `count`. There is no unlimited mode (the budget is the ceiling). A single line longer than a whole page's budget is delivered as far as it fits and named `partialItem=` — the rest of that line is not retrievable by paging. When a partial type's parts do not all fit, the result is the whole-or-omitted overview: `parts=`, `truncated=Y`, `omittedParts=`, each part that fits shown whole, the rest listed with their `lines=` count and `,omitted=Y` — fetch one with `part=`, which returns that part alone, paged as above with `parts=`/`part=` headers. `startItem` pages one part; combining it with several parts and no `part=` is `error=Invalid`. Overloads (or any name matching several distinct symbols) — `error=Ambiguous`, resolved by retrying with a candidate's parameter-type list; a metadata/referenced-assembly symbol — `error=NotSupported` (use `get_symbol` for its signature), but a metadata spelling (``Ns.Box`1``, `N.Outer+Inner`) of a type the solution declares in source returns that source. Prefer this over reading or grepping the file to see what a member does. @@ -138,15 +138,15 @@ Overloads (or any name matching several distinct symbols) — `error=Ambiguous`, `solutionId`, `typeName` (FQN), `includeInherited` (default false), `nameFilter` (default null; trailing `*` = prefix match, otherwise case-insensitive substring), `includeMethods`/`includeFields`/`includeProperties`/`includeEvents`/`includeNestedTypes` (all default true). Lists all members including private, grouped by declaring file (`` bucket for referenced-assembly types). Member lines: `kind,name,loc[,paramType|paramType|...]` (param list only for methods with parameters). Returns declarations + spans, not bodies — use `get_symbol_body` for a member's body. Each member's local functions are listed beneath it in declaration order, one level deeper (`localfunction,,,`), a local function declared inside another one level further. ### search_symbols -`solutionId`, `query` (case-insensitive substring), `maxResults` (default 50). Source-declared symbols only (no metadata), local functions included. Declarations a source generator emits are included and marked `,generated=Y` — a generated-only type, member, nested type, namespace or local function is found even when nothing hand-written matches; a partial type with a generated part is still one leaf at its first location (the hand-written part), with the generated part visible through its own members. Matched members nest under their type, and a local function under its containing member (`localfunction` kind); a type's `loc` appears only if the type itself matched. +`solutionId`, `query` (case-insensitive substring), `maxResults` (omit for every match - the result then pages with startItem/maxSize). Source-declared symbols only (no metadata), local functions included. Declarations a source generator emits are included and marked `,generated=Y` — a generated-only type, member, nested type, namespace or local function is found even when nothing hand-written matches; a partial type with a generated part is still one leaf at its first location (the hand-written part), with the generated part visible through its own members. Matched members nest under their type, and a local function under its containing member (`localfunction` kind); a type's `loc` appears only if the type itself matched. ## Relationships ### find_references -`solutionId`, `symbolName` (FQN), `maxResults` (default 100). Compiler-accurate: skips comments, strings, unrelated same-named members. References grouped file→namespace→type→member; pipe-delimited `loc`s on one line for multiple hits. Deduped across `#if` projections. `count=`/`truncated=Y` when capped. Prefer this over text search for usages in `*.cs`, `*.cshtml` and `*.razor`; combine with get_callers/find_implementations via multi_query for impact analysis. +`solutionId`, `symbolName` (FQN), `maxResults` (omit for every reference - the result then pages with startItem/maxSize). Compiler-accurate: skips comments, strings, unrelated same-named members. References grouped file→namespace→type→member; pipe-delimited `loc`s on one line for multiple hits. Deduped across `#if` projections. `count=`/`truncated=Y` when capped. Prefer this over text search for usages in `*.cs`, `*.cshtml` and `*.razor`; combine with get_callers/find_implementations via multi_query for impact analysis. ### find_reads / find_writes -`solutionId`, `symbolName`, `maxResults` (default 100). Target a field or property by FQN (`N.T.Field`), or a parameter as its containing method, constructor, indexer or local function's FQN + `:` + parameter name (`N.T.Method:value`, `N.T.Method(int):value`; the member part follows the usual overload grammar). Local variables are not supported. Same body as find_references, but one location per leaf with the access kind appended: `,,,`. +`solutionId`, `symbolName`, `maxResults` (omit for every location - the result then pages with startItem/maxSize). Target a field or property by FQN (`N.T.Field`), or a parameter as its containing method, constructor, indexer or local function's FQN + `:` + parameter name (`N.T.Method:value`, `N.T.Method(int):value`; the member part follows the usual overload grammar). Local variables are not supported. Same body as find_references, but one location per leaf with the access kind appended: `,,,`. `accessKind`: `read`, `assign` (`=`, including deconstruction targets), `compound` (`+=`, `??=`, ...), `increment` (`++`/`--`), `ref` (`ref x` argument, `ref x` expression, `&x`), `out`, `init` (field/property declaration initialiser, object or `with` initialiser, attribute named argument, or an assignment to the constructor's own field/property inside its constructor; a lambda inside the constructor is `assign`). find_reads returns `read`, `compound`, `increment`, `ref`; find_writes returns everything except `read`. **`compound`, `increment` and `ref` appear in both tools with the same kind** — do not add the two counts. `nameof(...)`, doc crefs and named arguments (`M(value: 1)`) are omitted from both. @@ -211,7 +211,7 @@ MyApp - `projectName`: only that project's diagnostics - exactly the name the outline prints (the project file name with the `.csproj` extension omitted; other extensions such as `.vbproj` are kept, and a trailing `.csproj` you pass is stripped). Case-insensitive, and it matches every loaded target framework of a multi-targeted project. - `filePath`: only diagnostics located in that file - absolute or relative to the solution folder (the same spellings `find_definition` accepts), case-insensitive. A `.razor`/`.cshtml` path also matches the C# diagnostics `#line`-mapped into it and the `RZ*` diagnostics reported against it. - `ids`: only those diagnostic ids, e.g. `["CS0246","CS0103"]`, case-insensitive. Omit, or pass an empty list, for every id. -- `maxResults`: caps the entries listed in the body (id rows in summary mode). When something is dropped, `count=` and `truncated=Y` precede the body; both are absent otherwise, so the body is then the complete set. The per-severity header counts always stay complete. +- `maxResults`: caps the result set (the body lists id rows in summary mode). When the cap drops something, `count=` and `truncated=Y` precede the body; both are absent otherwise. The per-severity header counts always stay complete. Omit maxResults to page through everything with startItem/maxSize. - `summaryOnly`: `true` replaces the body with one line per id - `,,,,,` - ordered by count descending then id, carrying each id's count, one example location (its first diagnostic in body order) and that example's message. It covers every severity unless an include flag is set. `maxResults` caps the id rows, with `count=` counting rows. Every active filter is echoed as a `filter=:` header before the counts, and the counts describe the **filtered** set - `errors=3` under a `filter=` line means 3 errors in that filtered scope, not 3 in the solution. An empty body with a `filter=` header means the filter matched nothing, NOT that the solution is clean. A `projectName` or `filePath` that is not in the solution is `error=NotFound` with `candidate=` values (one of which the tool would accept); one that exists but has no diagnostics returns zero counts, not an error. An `ids` entry that matches nothing is zero counts too - ids are open-ended and never validated. @@ -301,7 +301,7 @@ Razor: the compiler never reports CS8019 in generated code, so each `.razor`/`.c ## Dead code ### find_dead_code -`solutionId`, `scope` (optional FQN prefix, e.g. `MyApp.Services` — use it on large solutions; the scan runs find-references per symbol), `includePublic` (default false — public/protected API excluded unless set), `maxResults` (default 50). +`solutionId`, `scope` (optional FQN prefix, e.g. `MyApp.Services` — use it on large solutions; the scan runs find-references per symbol), `includePublic` (default false — public/protected API excluded unless set), `maxResults` (bounds the expensive scan; omit it to scan everything - the result then pages with startItem/maxSize). Leaf lines: `memberKind,memberName,loc,confidence,reason` with confidence `High|Medium`. Automatically excludes interface implementations, virtual/override/abstract chains, test-attributed members (xUnit/NUnit/MSTest), generated code, and DI/reflection-attributed members (`[Export]`, `[ImportingConstructor]`). The `loc` is the full declaration span, ready to pass to `apply_patch` for removal — this tool never deletes anything itself. Treat results as candidates, not verdicts. @@ -311,7 +311,7 @@ Leaf lines: `memberKind,memberName,loc,confidence,reason` with confidence `High| ## Batching ### multi_query -`solutionId`, `operations` (1-25 items), optional `expectSnapshot`. +`solutionId`, `operations` (1+ items; the limit is the framing itself — a batch whose retry line cannot fit the budget is refused with `error=Invalid` before anything runs), optional `expectSnapshot`. Each operation is `{ "tool": , "arguments": { ... } }` where `tool` is one of the 15 read-only query tools (get_symbol, get_symbol_body, get_members, find_definition, find_implementations, find_references, find_reads, find_writes, get_callers, get_callees, search_symbols, get_type_hierarchy, find_dead_code, find_dead_conditionals, get_expression_info) and `arguments` uses exactly that tool's single-call parameter names - unknown or misspelled keys are rejected (`error=Invalid` naming the key), never ignored; omitted parameters take the tool's declared defaults. The schema's `tool` enum lists the legal names, so a write tool, get_diagnostics or get_solution_status is unrepresentable and fails the whole call at binding (`error=Invalid` naming the offending value and the permitted set). @@ -323,21 +323,22 @@ Everything runs against ONE snapshot of the solution, so results inside one resp operations= snapshot=<32-hex id of the snapshot every slot was computed against> boundary=<32-hex, fresh per request> -budget= +budget= +retry=1:3.0,7:0.2 (present only when operations were dropped) -- -slot= tool= +slot=2 tool=get_type_hierarchy - + -- -slot= tool= truncated=Y +slot=3 tool=get_callees - + ---- ``` -A failing operation does not abort the batch: its slot carries its normal `error=` block (NotFound/Ambiguous/Invalid/...) and the other slots still deliver. The response is a hard ceiling on the whole envelope: framing is reserved up front and each slot gets exactly what remains. A slot whose output would not fit is cut after its last whole line - its meta line gains `truncated=Y` and its body ends with a bracketed `[response budget exhausted: slot () was cut at a line boundary; of lines of its output shown. ... pass expectSnapshot= ...]` note - or, when not even one whole line fits, refused whole as `error=Truncated`. `get_symbol_body` pages itself within its share instead (its `truncated=`/`nextStartLine=` headers are its own, not the batch's). **Trust rule:** a slot whose META LINE carries `truncated=Y` was cut by the batch budget, so its body is a prefix - do not derive tool-level paging (e.g. `nextStartLine`) from it; re-send that operation alone. Only a complete slot's own body headers may drive tool-level paging. +Every operation runs (in parallel); each result is then delivered WHOLE, smallest first, until the budget is filled, and delivered slots are emitted in request order. A slot is complete or absent - never cut - so only a delivered slot's own body headers may drive tool-level paging (`nextStartItem=`). Operations whose result did not fit are listed once in the `retry=` header, in request order, as `:` where size is that result's cost (frame + body) as a ratio of `budget=`, rounded up to 0.1 (0.1 is the smallest value reported, never 0.0). Plan the continuation from the ratios: size ≥ 1.0 can never fit beside anything - re-send that operation alone (it pages); sizes summing to at most 1.0 fit one continuation batch together; `ceil(size)` is roughly a solo call's page count. Ratios smaller than 0.1 of the budget are indistinguishable, so about ten such operations per continuation batch is the safe number. Ratios describe one snapshot: pass the response's `snapshot=` as `expectSnapshot` on the continuation, or a re-run against a moved solution re-measures and re-decides. Re-send exactly the listed operations with their original arguments from your own copy of the request - slot frames carry the index and tool name only, never the arguments. A failing operation is delivered as its normal `error=` block and can itself be dropped like any oversized result. -**Truncation and continuation.** More than 25 operations, or a total output over the response budget, truncates: unexecuted operations still get their numbered slot carrying `error=Truncated`, and the header carries `truncatedSlots=` (counting every slot that did not deliver a complete result: refused or cut - a self-paged `get_symbol_body` slot does not count, its page is a complete answer). To continue, re-send exactly the operations whose slots were Truncated - the slot meta line gives the index and tool name, but NOT the arguments, so rebuild the continuation from your own copy of the request (the server does not echo them back). Pass the first response's `snapshot=` value as `expectSnapshot` when you do: if the solution changed between the calls (another client, or the user saving a file) the continuation is refused with `error=Stale` naming both ids plus a `snapshot=` header - re-run the whole batch instead of stitching two generations. Consistency is guaranteed within one response, never across two. `get_diagnostics` and `get_solution_status` are deliberately not multi-queryable - call them directly. +**Truncation and continuation.** Pass the first response's `snapshot=` value as `expectSnapshot` when you continue: if the solution changed between the calls (another client, or the user saving a file) the continuation is refused with `error=Stale` naming both ids plus a `snapshot=` header - re-run the whole batch instead of stitching two generations. Consistency is guaranteed within one response, never across two. `get_diagnostics` and `get_solution_status` are deliberately not multi-queryable - call them directly. diff --git a/external-sources/vendir.lock.yml b/external-sources/vendir.lock.yml index 3118b4f4..a6fc7447 100644 --- a/external-sources/vendir.lock.yml +++ b/external-sources/vendir.lock.yml @@ -2,27 +2,27 @@ apiVersion: vendir.k14s.io/v1alpha1 directories: - contents: - git: - commitTitle: 'Merge pull request #1284 from dotnet/abhitejjohn-skill-eval-follow-up...' - sha: 19bb55d785cd88ce47a1b43f9a21d0192c412dac + commitTitle: 'chore: refresh gh-aw pins and recompile workflows (#1303)...' + sha: b0bf1730479858941ae22bdf0b45e88cfa31bcc5 tags: - - skill-validator-nightly-24-g19bb55d7 + - skill-validator-nightly path: dotnet-skills - git: commitTitle: Add Cursor rules that reference the existing skill docs... sha: af2319bd01bb7cc881267a9ef42cafdaf5e9029d path: webgpu-claude-skill - git: - commitTitle: 'ci(factory-preview): resolve nested packages by nearest package.json - (#18316)' - sha: 9050e3819c51b0383cced278c278bc6f45c63468 + commitTitle: Fail the build when a prerender error is swallowed by a custom + fetch handler (#18355) + sha: 59b9cc00726495bb45f5c4b000dbbb7b5a19d662 tags: - - '@astrojs/markdoc@2.0.11-12-g9050e3819c' + - '@astrojs/vercel@11.0.14-11-g59b9cc0072' path: astro - git: - commitTitle: Remove YourKit credit - sha: 474cac03c49edfa5d0171c16746b9283621d2886 + commitTitle: 'multi_query improved paging (Fixes #96) (#98)' + sha: d59c7ba3d485da1202dc977f15d36a142d892db9 tags: - - 2.1.0-17-g474cac0 + - 2.1.0-18-gd59c7ba path: roslynk path: upstreams kind: LockConfig