From abe83f51471f89691841a7ad8ab599072e2a0dae Mon Sep 17 00:00:00 2001 From: AlBurns-MSFT <6195567+AlBurns-MSFT@users.noreply.github.com> Date: Tue, 14 Jul 2026 14:53:34 -0400 Subject: [PATCH] Add TSG: Get-SolutionUpdate intermittent NullReferenceException (2505-2604) Customer-facing troubleshooting guide for an intermittent NullReferenceException from Get-SolutionUpdate / Get-SolutionUpdateEnvironment during solution updates on builds 2505-2604 (resolved in 2605+). The applicability gate uses only customer-observable signals (build range, exact error, intermittent behavior); guidance is retry -> contact support, with the permanent fix being an update to 2605+. Cross-links the existing GatewayTimeout TSG and indexes the new page in TSG/Update/README.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: AlBurns-MSFT <6195567+AlBurns-MSFT@users.noreply.github.com> --- ...ate-intermittent-NullReferenceException.md | 61 +++++++++++++++++++ TSG/Update/README.md | 1 + 2 files changed, 62 insertions(+) create mode 100644 TSG/Update/Get-SolutionUpdate-intermittent-NullReferenceException.md diff --git a/TSG/Update/Get-SolutionUpdate-intermittent-NullReferenceException.md b/TSG/Update/Get-SolutionUpdate-intermittent-NullReferenceException.md new file mode 100644 index 00000000..2e60ef97 --- /dev/null +++ b/TSG/Update/Get-SolutionUpdate-intermittent-NullReferenceException.md @@ -0,0 +1,61 @@ +# TSG | 2505-2604 | Get-SolutionUpdate fails intermittently with NullReferenceException + +On Azure Local builds **2505 through 2604**, `Get-SolutionUpdate` (or `Get-SolutionUpdateEnvironment`) can intermittently fail with a `NullReferenceException` during or shortly after a solution update. The failure is transient: retrying the same command usually succeeds. This is fixed in build **2605** and later. + +> **Applies to builds 2505-2604 only.** If your cluster is on **2605 or later**, this article does not apply. A persistent `NullReferenceException` on a fixed build is a different problem: collect diagnostics and contact Microsoft Support. + +# Symptoms + +`Get-SolutionUpdate` or `Get-SolutionUpdateEnvironment` returns: + +``` +Get-SolutionUpdateEnvironment : Object reference not set to an instance of an object. + + CategoryInfo : NotSpecified: (:) [Get-SolutionUpdateEnvironment], NullReferenceException + + FullyQualifiedErrorId : System.NullReferenceException,Microsoft.AzureStack.Lcm.PowerShell.GetSolutionUpdateEnvironmentCmdlet +``` + +Characteristics: + +- It appears **during or shortly after** a solution update (for example, while checking update status). +- It is **intermittent**: running the same cmdlet again a short time later often returns a normal result. +- The update itself keeps making progress. It is the status cmdlet that fails, not the update. + +# Issue Validation + +This article applies when **all** of the following are true: + +1. The cluster is on a build in the range **2505-2604** (23H2 `11.25xx`-`11.26xx`, or 24H2 `12.25xx`-`12.26xx`). Confirm the current version from the Azure portal (your Azure Local instance -> **Updates**), or from `Get-SolutionUpdate | ft Version, State` once the cmdlet returns. +2. The error above comes from `Get-SolutionUpdate` or `Get-SolutionUpdateEnvironment`. +3. It occurs during or around a solution update. +4. It is **intermittent**: a retry often succeeds. + +If the cmdlet instead **hangs or times out** (unresponsive for a long time, eventually a GatewayTimeout) rather than returning the error above, see [Get-SolutionUpdate GatewayTimeout](./Get-SolutionUpdate-GatewayTimeout.md). + +# Cause + +Under high concurrent request load (for example, frequent update-status polling combined with retrying or failed action plans), an internal timing issue in the Update Service can cause a request to the Orchestrator service to fail momentarily. The Update Service does not handle that failed response gracefully and surfaces a `NullReferenceException`. Because the condition is a transient timing issue, the same command normally succeeds when retried. The issue is resolved in build **2605** and later. + +# Mitigation Details + +## Step 1 - Retry (resolves most cases) + +Re-run the cmdlet. Because the failure is transient, it usually succeeds within a few attempts, and an in-progress update continues normally once the cmdlet returns. + +```powershell +Get-SolutionUpdate | ft Version, State, UpdateStateProperties +``` + +## Step 2 - If it does not clear + +If `Get-SolutionUpdate` keeps failing after several retries over a few minutes, or the update appears **stalled** (its state is not advancing), stop retrying. This may be a different issue (for example, an unresponsive Update Service or an unhealthy Orchestrator service), not the transient condition described here. + +- If the cmdlet is **hanging or timing out** rather than throwing the error above, follow [Get-SolutionUpdate GatewayTimeout](./Get-SolutionUpdate-GatewayTimeout.md). +- Otherwise, collect diagnostics and contact Microsoft Support. Run `Send-DiagnosticData` from any Azure Local machine to upload logs before opening the case; for the current steps, see [Collect diagnostic logs](https://learn.microsoft.com/azure/azure-local/manage/collect-logs). + +```powershell +Send-DiagnosticData +``` + +## Permanent resolution - update to 2605 or later + +The timing issue is fixed in build **2605**. Once 2605 or later is offered on your update train, moving to it removes the condition entirely. You do **not** need to start an update solely to work around this issue: if a retry already unblocked you, the fix will arrive through the normal update path. diff --git a/TSG/Update/README.md b/TSG/Update/README.md index f9f3f04e..8660deb5 100644 --- a/TSG/Update/README.md +++ b/TSG/Update/README.md @@ -8,6 +8,7 @@ * [Mitigation for 'Cannot remove item C:\UpdateDistribution' with access denied](./mitigation-for-cannot-remove-c-update-distribution-with-access-denied.md) * [Add/Repair Node fails at "GenerateSSLCertificatesForAddScaleUnitNodes" with "The system cannot find the file specified."](./AddNode-RepairNode-Fail-GenerateSSLCertificatesForAddScaleUnitNodes.md) * [Get-SolutionUpdate GatewayTimeout](./Get-SolutionUpdate-GatewayTimeout.md) +* [Get-SolutionUpdate fails intermittently with NullReferenceException (2505-2604)](./Get-SolutionUpdate-intermittent-NullReferenceException.md) * [UpdateOsBuildNumberInEce interface fails during Solution update because of inconsistent OS versions between nodes](./UpdateOsBuildNumberInEce-MismatchedBuilds.md) * [Update during .NET Updates Installation Cannot validate argument on parameter ClusterName](./Update-during-.NET-Updates-Installation-Cannot-validate-argument-on-parameter-ClusterName.md) * [Update during .NET Updates Installation Out of Band .Net Updates Installed on the cluster](./Update-during-.NET-Updates-Installation-Out-of-Band-.Net-Updates-Installed-on-the-cluster.md)