Description
Parallel Azure SDK regeneration can fail nondeterministically when multiple library builds restore shared projects into the same MSBuild intermediate directory. This has occurred more than once and makes the typespec - http-client-csharp - publish regeneration pipeline flaky.
The most recent occurrence was internal build 6776805, triggered for #11818.
Observed failure
The pipeline dispatched 49 regeneration jobs with concurrency 12:
Dispatching 49 regeneration jobs (12 at a time)...
Azure.Communication.JobRouter then failed during restore, before generator behavior was exercised:
NuGet.targets(198,5): error : The file
'.../artifacts/obj/Azure.SdkAnalyzers/Azure.SdkAnalyzers.csproj.nuget.g.props'
already exists.
The run completed with:
- 48 libraries successfully regenerated
- 1 actual failure:
Azure.Communication.JobRouter
- 4 libraries skipped because an earlier regeneration batch failed
The same PR's normal Linux and Windows C# regeneration checks passed, further indicating that this was an orchestration race rather than a generator regression.
Likely cause
Invoke-SdkLibraryRegeneration starts independent library builds concurrently. Those builds can restore common projects such as Azure.SdkAnalyzers while sharing the repository-level artifacts/obj directory. Concurrent restore processes then race while creating the same generated NuGet files.
The error is consistent with two restore operations attempting to create:
artifacts/obj/Azure.SdkAnalyzers/Azure.SdkAnalyzers.csproj.nuget.g.props
Expected behavior
Parallel regeneration should isolate or coordinate shared restore/build outputs so independent libraries cannot write the same intermediate files concurrently. A transient restore race should also not abort all remaining regeneration batches without a retry.
Potential approaches include:
- perform a single restore of shared projects before dispatching parallel generation;
- use isolated MSBuild intermediate paths per regeneration worker;
- avoid restoring shared analyzer/tool projects independently in each worker;
- serialize the restore phase while keeping generation parallel; or
- retry failures that match known transient file-creation conflicts after concurrent work settles.
Acceptance criteria
- Repeated parallel full-regeneration runs do not fail with shared
artifacts/obj file conflicts.
- Parallel workers do not concurrently mutate the same MSBuild/NuGet intermediate files.
- Generated output remains identical to sequential regeneration.
- A focused test or pipeline stress validation exercises concurrent restoration of libraries sharing
Azure.SdkAnalyzers.
This is related to the race-free parallelism requirement in #11361, but is tracked separately as a concrete recurring pipeline bug.
- by copilot
Description
Parallel Azure SDK regeneration can fail nondeterministically when multiple library builds restore shared projects into the same MSBuild intermediate directory. This has occurred more than once and makes the
typespec - http-client-csharp - publishregeneration pipeline flaky.The most recent occurrence was internal build 6776805, triggered for #11818.
Observed failure
The pipeline dispatched 49 regeneration jobs with concurrency 12:
Azure.Communication.JobRouterthen failed during restore, before generator behavior was exercised:The run completed with:
Azure.Communication.JobRouterThe same PR's normal Linux and Windows C# regeneration checks passed, further indicating that this was an orchestration race rather than a generator regression.
Likely cause
Invoke-SdkLibraryRegenerationstarts independent library builds concurrently. Those builds can restore common projects such asAzure.SdkAnalyzerswhile sharing the repository-levelartifacts/objdirectory. Concurrent restore processes then race while creating the same generated NuGet files.The error is consistent with two restore operations attempting to create:
Expected behavior
Parallel regeneration should isolate or coordinate shared restore/build outputs so independent libraries cannot write the same intermediate files concurrently. A transient restore race should also not abort all remaining regeneration batches without a retry.
Potential approaches include:
Acceptance criteria
artifacts/objfile conflicts.Azure.SdkAnalyzers.This is related to the race-free parallelism requirement in #11361, but is tracked separately as a concrete recurring pipeline bug.
- by copilot