OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability#3918
Open
marcschier wants to merge 108 commits into
Open
OPC UA Part 4 6.6 Redundancy (server + client) with opt-in distributed high-availability#3918marcschier wants to merge 108 commits into
marcschier wants to merge 108 commits into
Conversation
…ng with hardened shared store Adds an opt-in provider model (Opc.Ua.Server.Distributed) to replicate address-space topology/values and session state across server replicas for active/passive and active/active HA, exposed via documented OPC UA redundancy (ServiceLevel/RedundantServerArray). The single-instance in-memory path stays zero-overhead (default NullRecordProtector). Building blocks: ISharedKeyValueStore (+in-memory), INodeStateStore, address-space synchronizer, leader election (static + shared-store lease CAS), service-level providers, distributed value cache, shared session store. Wired via IServerStartupTask hosting seam + fluent DI (UseDistributedAddressSpace / AddServerServiceLevel / redundancy options). CustomNodeManager2 opts in via ILocalAddressSpaceSource. Security hardening (plan 30, SDL + IEC 62443): IRecordProtector + AesCbcHmacRecordProtector (AES-256-CBC Encrypt-then-MAC, verify-before-decrypt, fail-closed) on every shared record; KeyRingRecordProtector (staged key rotation); SharedSingleUseNonceRegistry (cross-replica single-use server nonce via store CAS); key zeroization. Docs/HighAvailabilitySecurity.md captures the STRIDE threat model and operator guidance. Docs: HighAvailability.md, HighAvailabilitySecurity.md, Docs/README link, HighAvailabilityServer sample. Plans 28/29/30. 85 Distributed unit tests (net10.0 + net48).
…reconnect (S5) Implements the opt-in mirrored fast-reconnect from plans 30/31. After a failover a client reconnects to a standby by re-running ActivateSession; the AuthenticationToken is only a lookup key and the standby still performs the full client-certificate signature validation against a mirrored, single-use serverNonce (token-only hijack and nonce replay are both closed). - SharedSessionEntry: extended with the full reconstruction state (serverNonce, clientNonce, client cert blob, SecurityPolicy/Mode, endpoint, timeout, client description); encrypted at rest via the record protector. - SessionManager: additive, backward-compatible RestoreSessionAsync + SupportsSessionRestore seam in the ActivateSession miss-path (default returns null => unchanged behaviour; m_sessions stays private). - DistributedSessionManager: mirrors encrypted session state on create/activate, removes on close; on restore enforces REQ-UA-7 (same SecurityPolicy/Mode), consumes the serverNonce single-use across the replica set (replay defence), reconstructs the session, and logs provenance with a one-way token digest. - ISessionManagerFactory seam on StandardServer (wired from DI by the hosted service, supplying a server-certificate provider) + DistributedSessionManagerFactory + UseDistributedSessions(...) fluent API. Safe default EnableFastReconnect=false (re-auth on failover). - Docs (HighAvailability.md, HighAvailabilitySecurity.md) updated. Tests: 97 Distributed (net10+net48); no regression (61 Session + 57 client SessionTests integration pass). Remaining: two-server network e2e (S6).
…roring (S6) DistributedSessionMirrorIntegrationTests stands up a fully-started server whose ISessionManagerFactory builds a DistributedSessionManager, and verifies end-to-end that a session created/activated through the real service handlers is mirrored encrypted to the shared store (a wrong-key reader fails closed) and removed on close. This closes the factory -> StandardServer.CreateSessionManager -> mirror runtime-wiring gap. The full secured two-server token-reuse reconnect happy-path remains a documented follow-up (the stack client does re-auth-on-failover; the direct-service helper only drives unsecured sessions). The restore decision logic (REQ-UA-7 + single-use nonce/replay) is unit-tested and the base ActivateSession signature path is integration-tested. 98 Distributed tests pass (net10 + net48).
…t (F6/F9) Workstream A from plans/32: - F6: SharedKeyValueSessionStore keys entries by the SHA-256 digest of the AuthenticationToken (SharedKeyValueSessionStore.KeyFor) instead of the raw token, so a backend's key enumeration/monitoring/dumps never expose the token. Test asserts the keyspace contains no raw token. - F9: a successful cross-replica restore emits a distinct AuditSessionEventState (Session/RestoredFromSharedStore) via the new IAuditEventServer.ReportAuditSessionRestoredEvent, in addition to the standard AuditActivateSession, with a one-way token digest for provenance. - F7: analyzed — the decrypted serverNonce becomes the session's retained working Nonce (no extra plaintext copy in the manager); Nonce.Data zeroization on dispose is a pre-existing server-wide Core concern tracked separately. Docs updated. 99 Distributed tests pass (net10 + net48).
…ng in sample (FG) Workstream FG from plans/32: - New Docs/KubernetesDeployment.md: worked replicaset deployment (StatefulSet + headless Service, leader election, readiness tied to ServiceLevel, KEK + shared ApplicationInstanceCertificate provisioning via Secrets, security checklist). Linked from Docs/README.md and HighAvailability.md. - HighAvailabilityServer sample: wires UseDistributedSessions (opt-in HA_FAST_RECONNECT) and an optional AesCbcHmacRecordProtector from a base64 HA_RECORD_KEY (encrypted shared store), demonstrating the production-hygiene pattern. - Transparent redundancy remains a documented deployment pattern (single virtual endpoint + shared session store + subscription transfer; no new transport).
…se failover) design
…A-13, B) On failover to a redundant server, when EnableTokenReuseFailover is set, the client re-activates the existing session by reusing the current AuthenticationToken (signing over the new channel + last serverNonce) instead of CreateSession, falling back to re-authentication if the standby rejects the token. - Session.cs: extracted ReactivateExistingSessionAsync (the token-reuse activation core, shared with UpdateSessionAsync); RecreateInPlaceCoreAsync tries it first (adopting the failover server's cert) before the existing clear + fresh-CreateSession fallback; new EnableTokenReuseFailover property (copied across recreate clones). OpenAsync is untouched. - ManagedSession: EnableTokenReuseFailover option threaded through CreateAsync + ctor and applied to the inner session; ManagedSessionOptions.EnableTokenReuseFailover; ManagedSessionBuilder.WithTokenReuseFailover(). Default off (re-auth on failover). No regression: 57 client SessionTests + 135 reconnect/failover integration tests pass (net10). Updated the ctor-reflection test for the new parameter.
DistributedSessionFailoverIntegrationTests: two secured servers share one store via DistributedSessionManager; a ManagedSession client with WithTokenReuseFailover fails over from the active to the standby. The standby restores the mirrored session and re-activates it with the reused AuthenticationToken, so the client's SessionId is preserved (a fresh re-authentication would change it). Passes on net10 + net48. Docs (HighAvailability.md, HighAvailabilitySecurity.md) updated: client WithTokenReuseFailover() usage + the e2e validation; plans/32 marks B and C done.
marcschier
commented
Jun 26, 2026
Split all distributed/HA implementation into a new OPCFoundation.NetStandard.Opc.Ua.Server.Distributed project that references Opc.Ua.Server; core now keeps only the seams. - Move ILocalAddressSpace/ILocalAddressSpaceSource to the Opc.Ua.Server namespace (NodeManager/); extract a shared internal PredefinedNodesAddressSpace reused by both CustomNodeManager2 and AsyncCustomNodeManager (both now implement ILocalAddressSpaceSource). Apply path is async (no sync-over-async). - Remove the node-state-store-registry hook from ServerInternalData; the distributed startup task owns its own registry. - Add CryptoUtils.ZeroMemory/FixedTimeEquals polyfills and reuse them in the record protector, EncryptedSecret, and KeyCredentialBridgeAuthenticator. - Fix CA2213 in AddressSpaceSynchronizer (dispose enumerator in DisposeAsync). - Rebase HighAvailabilityServer sample onto AsyncCustomNodeManager; expand the sample README with shared-store wiring and an active/active setup. - Strip internal tracking tags (REQ-UA/Finding/IEC/SDL) from code and XML docs. - Merge HighAvailabilitySecurity.md into HighAvailability.md and fix links. - Wire the new project into UA.slnx, the test project, and the sample.
Add a separate net8.0+ package providing true active/active (multi-writer) replication for the distributed server, built on the Crdt + Crdt.Transport NuGet packages. Opt-in; the base distributed library and its active/passive path are unchanged. - New Libraries/Opc.Ua.Server.Distributed.Crdt project (net8.0;net9.0;net10.0, no-op shell on legacy CI legs via RestrictForLegacyTfm); references the base distributed project + Crdt + Crdt.Transport. - Address space A/A: CrdtAddressSpaceSynchronizer models topology and values as last-writer-wins maps and gossips state over Crdt.Transport (in-memory / TCP / UDP, optional TLS); every replica writes, received state is merged and applied. A topology merge preserves the locally-known value so it never regresses a concurrently-updated value (values are versioned by their own entries). - Sessions A/A: CrdtSharedKeyValueStore replicates encrypted session entries by gossip and reuses DistributedSessionManager; the single-use server nonce stays on a strongly-consistent ISingleUseNonceRegistry (CRDTs cannot enforce exactly-once), and the CRDT store rejects compare-and-swap. - Fluent opt-in: UseCrdtAddressSpace(...) / UseCrdtSessions(...) with a shared CrdtGossipOptions base (ReplicaId, UseTcpGossip/UseUdpGossip/AddPeer, limits). - Tests: new Opc.Ua.Server.Distributed.Crdt.Tests (net8/net10) — convergence, multi-writer, concurrent-value LWW, serializer round-trip, CAS/Watch boundary; plus a CRDT exercise in the AOT test project. - Docs: HighAvailability.md 'Active/active with CRDTs' section incl. the single-use-nonce security boundary; Crdt/Crdt.Transport added to CPM. Crdt/Crdt.Transport 1.0.0 (MIT, NativeAOT-ready) restore from nuget.org.
marcschier
commented
Jun 26, 2026
marcschier
commented
Jun 26, 2026
marcschier
commented
Jun 26, 2026
…mple, clarify HA docs - Rename the user-facing fluent surface from Crdt* to Replicated*: UseReplicatedAddressSpace / UseReplicatedSessions, ReplicatedAddressSpaceOptions / ReplicatedSessionOptions / ReplicatedGossipOptions, ReplicatedServerBuilderExtensions (implementation classes and the Crdt package/namespace keep their names). - Integrate the CRDT package into the HighAvailabilityServer sample with an HA_MODE switch (ap = active/passive leader-write, aa = active/active CRDT gossip via HA_GOSSIP_PORT / HA_GOSSIP_PEERS); document running two AA replicas in the sample README. - HighAvailability.md: reference the implemented CRDT library (not 'deferred'), document that a write to a non-leader replica is discarded in active/passive, and rewrite the value-participation note to describe how participation works. Addresses review comments on Docs/HighAvailability.md and the sample README.
…Ua.Server.Distributed.Tests Organize the source files of both Opc.Ua.Server.Distributed and Opc.Ua.Server.Distributed.Crdt into logical subfolders (AddressSpace, KeyValueStore, Redundancy, Values, Sessions, Security for the base library; AddressSpace, Sessions for the CRDT adapter), with the top-level DI entry points kept at the project root. Namespaces are unchanged. Mirror the organization in the test projects: - Extract the base distributed tests from Opc.Ua.Server.Tests/Distributed into a dedicated Opc.Ua.Server.Distributed.Tests project (mirroring the src project), with the same subfolders; add it to UA.slnx and InternalsVisibleTo. Test namespaces are unchanged. - Organize Opc.Ua.Server.Distributed.Crdt.Tests into AddressSpace/Sessions subfolders mirroring the CRDT src. Addresses review comments asking to mirror the test projects to the src side and to organize the files logically without changing namespaces.
marcschier
changed the title
Add unit tests for the previously-uncovered CRDT adapter surface flagged by codecov (patch coverage 67.65%): - ReplicatedGossipOptions/ReplicatedSessionOptions: defaults, UseTcp/UseUdp transport factory wiring, AddPeer, CreateReaderOptions/CreateTransport, null-arg guards. - CrdtAddressSpaceStartupTask: attaches a synchronizer to opted-in (ILocalAddressSpaceSource) node managers and skips the rest; null-arg guards. - CrdtSessionManagerFactory + UseReplicatedSessions/UseReplicatedAddressSpace registration; null-arg guards. - ByteStringCrdtSerializer JSON round-trip (null/empty/data); CrdtSharedKeyValueStore.ScanAsync prefix filtering. CRDT package line coverage 67% -> 93.7%; 25/25 tests pass on net8.0 and net10.0.
Collaborator
Author
|
Addressed the codecov patch-coverage feedback in 4f4d7ed. The 80% patch gap was concentrated in the new
CRDT package line coverage rises from ~67% to 93.7%; 25/25 tests pass on net8.0 and net10.0. Codecov will re-evaluate the patch on this commit. |
…t + non-transparent Server model: per-mode Server.ServerRedundancy (RedundancySupport, RedundantServerArray, non-transparent ServerUriArray, transparent CurrentServerId); sub-range ServiceLevel providers (Table 105) + load balancing; RequestServerStateChange/Maintenance/EstimatedReturnTime (6.6.5, admin-gated); NTRS capability; FindServers returns the RedundantServerSet (IRedundantServerSetProvider). New shared Opc.Ua.ServiceLevels/ServiceLevelSubrange in core. Client: RedundantManagedClient realizing all Table 107 modes (Cold/Warm/Hot(a)/Hot(b)/HotAndMirrored); RedundancySupport wording; ServiceLevel sub-range failover rules; FindServers ServerUri->endpoint resolution (no security downgrade); client redundancy via TransferSubscriptions; non-transparent network redundancy. State mirroring (opt-in provider seams; single-instance unchanged): session takeover, subscription-definition mirror, async notification sequence/Republish mirror, best-effort continuation-point envelope, deterministic EventId provider, RegisterNodes replica-consistent. Extensions: Opc.Ua.Server.Distributed.Crdt (active/active) and Opc.Ua.Server.Distributed.Kubernetes (Lease election, peer discovery, ServiceLevel->readiness). Samples (HighAvailabilityServer, RedundantClient), docs mapped to 6.6, conformance tests. Security: AES-CBC+HMAC record protection, fail-closed RecordProtectionGuard for external stores (base + CRDT session paths), single-use server-nonce replay protection, K8s TLS hostname validation. Validated on net10 and net48.
… review findings CRDT active/active extension (Opc.Ua.Server.Distributed.Crdt): - Bump Crdt/Crdt.Transport 1.0.0 -> 1.0.2 (now ship netstandard2.0/2.1 assets) - Widen TFMs from net8.0+ to $(LibTargetFrameworks) (net472/net48/netstandard2.1/net8/9/10); remove RestrictForLegacyTfm; gate IsAotCompatible/binding-gen to net8+ - Fail closed for unauthenticated networked gossip (TCP without mutual TLS, UDP); add AllowUnauthenticatedGossip opt-out for dev/test; gossip-port NetworkPolicy guidance Server review findings: - AddServerRedundancy warns when non-transparent mode has no IServiceLevelProvider - Rename AddManualFailover -> AddRequestServerStateChange ([Obsolete] shim retained) - Consolidate ServerRedundancyOptions peer inputs (RedundantPeers canonical) - DeterministicEventId: exclude per-replica ReceiveTime, compute after distinguishing fields are populated (replica-stable per 6.6.2.2) - Subscription retransmission mirror: delta path, namespace/server tables once per subscription, bounded-parallel drain Client review findings: - Maintenance(0) fails over to a healthy peer (Table 105); EstimatedReturnTime backoff - Warm backups use MonitoringMode.Disabled until failover (Table 107) - Hot(b) dedup uses exact value identity (no hash-collision drops); value-struct key - Subscription-template ownership: client disposes retained templates - De-duplicate WithNetworkRedundancy registration Validated: 0 warnings; tests green on net10 + net48 (CRDT 30, Distributed 178, Client redundancy 75, InformationModel redundancy 7).
…eview feedback Rename (full identity: folders, csproj, AssemblyName, PackageId, RootNamespace, C# namespaces, references, UA.slnx, InternalsVisibleTo): - Opc.Ua.Server.Distributed -> Opc.Ua.Server.Redundancy - Opc.Ua.Server.Distributed.Crdt -> Opc.Ua.Server.Redundancy.Crdt - Opc.Ua.Server.Distributed.Kubernetes -> Opc.Ua.Server.Redundancy.K8s (identity only; the word "Kubernetes" and k8s client types are preserved in prose/code) - Tests mirror the rename (.Tests/.Crdt.Tests/.K8s.Tests/.Integration.Tests) - Application HighAvailabilityServer -> RedundantServer - Documentation updated to the new names (present tense, merged-to-master state) Review feedback (verified previously-resolved PR comments are satisfied; fixed gaps): - async node-manager base, A/A sample README, REQ-UA tags, CA2213, central CryptoUtils polyfills, ILocalAddressSpace namespace - all confirmed in current code Roadmap findings implemented: - OPCFoundation#28 ServerUriArray-only peers are now failover candidates (connect + read live level) - #8 FetchRedundancyInfo follow-up reads merged into one ReadValuesAsync - OPCFoundation#29 ContinuationPoint mirroring documented as envelope-only (partial-SHALL boundary) - OPCFoundation#30 Server.ServerRedundancy typed to NonTransparent/Transparent subtype by mode - OPCFoundation#20 Add* (standard) vs Use* (extension) convention documented + ServiceLevel cross-ref - OPCFoundation#24 transparent-mode shared application-key blast-radius/rotation guidance expanded CI green (fixes pre-existing all-TFM build break, unrelated to the rename): - Opc.Ua.Sessions.Tests redundancy test doubles: implement IServerRedundancyHandler. ShouldFailover and use RedundancySupport (RedundancyMode was removed) - Integration tests: add RestrictForLegacyTfm + CustomTestTarget fallback so legacy TFM CI passes no-op instead of failing with empty TargetFrameworks - Integration Maintenance assertion updated to spec-aligned behavior (Maintenance with an available peer warrants failover, Part 4 Table 105) Validated: full UA.slnx builds on net10 + net48; Redundancy 178, Crdt 30, K8s 27, Integration 3, Client redundancy 76, Sessions failover 4 - all pass.
The private ManagedSession constructor gained enableTokenReuseFailover (bool) and networkRedundancy (NetworkRedundancyOptions?) parameters during the network-redundancy work, but two reflection-based test helpers still used the old signature, so GetConstructor returned null and failed the fixture SetUp (52 cascading failures = the test-ubuntu-latest-Client / Fast PR test CI failures). - ManagedSessionComplianceTests.CreateManagedSessionWithInner: add the 4th bool + NetworkRedundancyOptions to the ctor type list and invoke args - ManagedSessionTests: add NetworkRedundancyOptions to the ctor type list and invoke args Validated: full Opc.Ua.Client.Tests now 1530 passed / 0 failed on net10.
The legacy-TFM no-op shell strips all references, so on netstandard2.0/2.1 there is no
assembly providing System.Object and the empty compile failed with CS8021 ("No value
for RuntimeMetadataVersion"). net4x builds got System.Object from the implicit targeting
pack so they worked. Supply an explicit RuntimeMetadataVersion for the no-op shell so the
empty assembly compiles on every legacy TFM.
This was the netstandard2.0 leg of the build-*-all-tfm failure; it affected every
RestrictForLegacyTfm project (Core.Diagnostics, Network.Fuzz*, Redundancy.K8s*,
Redundancy.Integration.Tests, RedundantServer, RedundantClient, Mcp, Minimal* samples).
Validated: full UA.slnx builds on net10, net48, and netstandard2.0; the no-op shell
compiles cleanly on net472/net48/netstandard2.0/netstandard2.1.
marcschier
changed the title
… from codecov The new redundancy libraries are ~80% unit-covered, but codecov/patch was dragged down by genuinely-untestable Kubernetes IO (real HTTP to the API server, the readiness HTTP listener, and cluster-bound startup tasks). - Add KubernetesServerBuilderExtensionsTests: a DI-registration test that exercises UseKubernetes / UseKubernetesLeaderElection / UseKubernetesPeerDiscovery / UseKubernetesReadiness (the previously 0%-covered builder wiring) plus null-builder guards, resolving the registered services out-of-cluster with no IO. - codecov.yml: ignore the four integration-only K8s IO files (KubernetesHttpApiClient, KubernetesReadinessServer, KubernetesReadinessStartupTask, KubernetesPeerDiscoveryStartupTask) - mirrors the existing Applications/** exemption; the Kubernetes logic (models, factory, lease election, peer parsing, readiness mapping, builder wiring) stays measured. Measured redundancy-library line coverage (codecov view) is now ~88%, above the 80% patch target. K8s tests: 29 passed.
…Lock ambiguity) The net10-only RedundantServer sample used RestrictForLegacyTfm, which only no-ops legacy TFMs (net4x/ns2.x). Under the Linux all-TFM CI leg (CustomTestTarget=net8.0) the app still built net10 while referencing the net8-built Opc.Ua.Types, so its System.Threading.Lock polyfill collided with net10's BCL Lock (CS0433) in HaSampleNodeManager. Follow CustomTestTarget like the libraries so the app's framework references match the TFM being built (net8 app + net8 types => only the polyfill Lock). Validated: RedundantServer builds on net8.0 and net10.0.
Core.Diagnostics.Tests builds net8/net9/net10 and had an unconditional build-ordering ProjectReference to the net10-only McpServer (so its assembly exists for the reflective McpServer tests). On the Linux all-TFM CI legs (CustomTestTarget=net8.0/net9.0) the solution build forced McpServer to the leg TFM, which it cannot target (its own deps build net8/net9), failing the UA.slnx build. - Reference McpServer only when building net10 (CustomTestTarget '' or net10.0). - Make both reflective LoadMcpAssembly helpers Assert.Ignore (skip) when the net10 Opc.Ua.Mcp assembly is absent, instead of asserting it exists - so the net8/net9 test legs skip the MCP reflective tests cleanly while net10 still runs them. Pre-existing infra issue (unrelated to the redundancy feature) surfaced once the RedundantServer net8 Lock fix let the Linux all-TFM build progress past net8. Validated: full UA.slnx builds on net8.0/net9.0/net10.0; Core.Diagnostics McpServer tests pass on net10 (13) and skip/pass without failure on net8.
marcschier
commented
Jun 28, 2026
marcschier
commented
Jul 2, 2026
…efs to issues, diagram + client-HA simplification, multiprocess client Thread 1 (RedundantClient/Program.cs): reframe the in-process/in-memory --replicas replica set as local-testing-only; add a scalable multiprocess demo (independent managed clients via the scale compose 'clients' profile, --scale client=N) and document the coordinated leader-elected variant (AddRedundantClientSession + Raft client store). Thread 2 (RedundantServer): move docker-compose.loaddirection/scale/transparent.yml into loaddirection/, scale/, transparent/ folders each named docker-compose.yml (nginx.transparent.conf -> transparent/nginx.conf); fix build contexts (../.. -> ../../..) and the nginx volume path; update all doc references; all three render via docker compose config. Thread 3 (HighAvailability.md): replace the two plans/28-distributed-ha-remaining.md references with the tracking issues OPCFoundation#3938/OPCFoundation#3939 and delete the plan file. Thread 4 (HighAvailability.md): add two clients to the active/active distributed-extensions diagram. Thread 5 (HighAvailability.md): reorder the client-HA section so the AddRedundantClientSession simplification leads, with the low-level seams moved under an 'Under the hood' heading.
…nt facade; accept BadSessionIdInvalid in durable close race
marcschier
commented
Jul 2, 2026
# Conflicts: # Libraries/Opc.Ua.Client/Session/ManagedSession.cs
…r-owned handle) for HA session cert provider; resolve ManagedSession conflict keeping both ReactivateMirroredSessionAsync and RequiresSessionRecreate
…ders (LoadDirection/Scale/Transparent); HA doc wording (goes further than, CRDT/Raft/WAL first-use full names + Raft thesis link + etcd/Kafka note, bold directed, opt-in/nuget note, nonce-registry rewording)
… facade, Server state/sessions/load-direction, Kubernetes) from 66.5% toward >80%
…tMessageQueue/SessionContinuationPoints (raises PR diff coverage further above 80%)
marcschier
added a commit
that referenced
this pull request
Jul 3, 2026
Addresses PR #3949 review feedback: - Merge Docs/PubSubKafka.md into Docs/PubSub.md as the last transport under "Transports" (Apache Kafka now follows the DTLS transport-status note), condensing the code snippet and pointing at the reference sample for full publisher/subscriber configurations. Delete PubSubKafka.md and repoint the Docs/README.md and NugetREADME.md links to PubSub.md#apache-kafka. - Document PubSub high availability + redundancy in PubSub.md: renamed the HA section, added an "Active/standby activation" subsection covering IPubSubActivationCoordinator, IPubSubLeaseStore, PubSubComponentRole, fencing tokens, AlwaysActiveCoordinator/LeaseActivationCoordinator, and PubSubRedundancyMode. - Align the redundancy docs to #3918's core abstractions (Opc.Ua.Redundancy.ISharedKeyValueStore CompareAndSwapAsync + ILeaderElection, ServiceLevel) and note the intended convergence of the PubSub lease store onto the shared compare-and-swap primitive; cross-reference Docs/HighAvailability.md. - Update the Native AOT section for the Kafka per-TFM client split.
…de PKI store (server); load client config from base dir + V2 subscription monitoring with publishing enabled (client)
…ullRecordProtector for the isolated demo (production uses mutual TLS + HA_RECORD_KEY)
…or Raft/strong shared store when HA_RECORD_KEY unset (recommended active-passive.env demo now forms a cluster; verified leader election + failover)
…ored item + PublishingEnabled (covers RedundantClient monitoring fix)
… aa-convergence bug + sample Counter is process-local on failover) and inline caveats
…(registry-backed, injectable; direct-construct fallback retained)
…e is shared across replicas and continues on failover (seed-on-promotion + write-through)
… fail closed without key); demo env files set HA_INSECURE with secure-run guidance
…DI registration + not-started fail-fast, warm-standby pre-connect
…n and secure-by-default HA_INSECURE; strong-Raft Counter continuity now works (README + HighAvailability.md)
…dation#3939) Add an async queue-restore path so a networked ISubscriptionStore can re-hydrate per-monitored-item data/event queues without blocking the synchronous MonitoredItem creation path, plus a continuous shared-store mirror so queued-but-unpublished values survive an HA failover. - ISubscriptionStore: add Restore{DataChange,Event}MonitoredItemQueueAsync (sync hooks kept as fallback) - IStoredMonitoredItem/StoredMonitoredItem: transient Restored* queue properties - MasterNodeManager: pre-hydrate queues before MonitoredItem construction - MonitoredItem.RestoreQueue: prefer pre-hydrated queue, fall back to sync - Make core queue mutators virtual for mirroring subclasses - StandardServer + hosted service: DI seams for ISubscriptionStore and IMonitoredItemQueueFactory - Opc.Ua.Redundancy.Server: SharedKeyValueMonitoredItemQueueFactory + mirroring queues; wire into UseDistributedSubscriptionMirroring and SharedKeyValueSubscriptionStore async restore - Tests (core, redundancy, AOT) and docs (HighAvailability, migration guide)
…red-item-queues Server: restore monitored-item data/event queues on failover (OPCFoundation#3939)
…ring test (extension now needs it for the failover monitored-item-queue factory merged from OPCFoundation#3939)
…rusted/issuer/rejected + CRLs) SharedKeyValueCertificateStore + provider persist public certs and CRLs in the shared HA key/value store so a RedundantServerSet shares one trust list; every record is IRecordProtector-protected (fail-closed integrity). CertificateManagerOptions.StoreProviders makes it injectable (built-in Directory/X509 providers retained). Read-through gives live cross-replica trust decisions. 13 unit/integration tests (CRUD, cross-replica share, tampered fail-closed, rejected trim, CRL revocation, provider).
…link from README and HighAvailability.md
marcschier
commented
Jul 5, 2026
Collaborator
Author
There was a problem hiding this comment.
Remove and add all to ha doc, link from cert manager doc.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch implements OPC 10000-4 §6.6 Redundancy end to end — server and client, transparent and non-transparent, across all failover modes (
None,Cold,Warm,Hot,HotAndMirrored,Transparent) — and layers an opt-in distributed high-availability (HA) capability on top so replicas can share address-space, session, and subscription state and expose redundancy to clients through the documented OPC UA mechanisms (for example, running a server replica set across nodes in Kubernetes).The single-instance, in-memory path stays zero-overhead: every distributed feature is opt-in through dependency injection, with a direct-construction fallback, and replaced APIs are kept and marked
[Obsolete].New packages
OPCFoundation.NetStandard.Opc.Ua.Server.RedundancyOPCFoundation.NetStandard.Opc.Ua.Server.Redundancy.CrdtOPCFoundation.NetStandard.Opc.Ua.Server.Redundancy.K8sAlso adds the worked samples
Applications/RedundantServerandApplications/RedundantClient, and the guidesDocs/HighAvailability.mdandDocs/HighAvailabilityKubernetes.md.Server side (§6.6)
AddServerRedundancy(...)populates the liveServer.ServerRedundancynodes (RedundancySupportplusRedundantServerArray/ServerUriArray/CurrentServerId) and drivesServer.ServiceLevelfrom anIServiceLevelProvider(leader high, standby low).Server.ServerRedundancyis typed to the NonTransparent or Transparent subtype according to the configured mode.AddRequestServerStateChange(...)implements the §6.6.5 manual-failover method, withEstimatedReturnTimeand the Maintenance / NoDataServiceLevelsub-ranges.FindServers(ConfiguredRedundantServerSetProvider) and advertise theNTRSdiscovery capability for GDS/NTRS registration.EventIdsynchronization for Transparent and HotAndMirrored sets (§6.6.2.2), excluding per-replica fields so de-duplicating clients do not double-process events.Client side (§6.6)
DefaultServerRedundancyHandlerreadsServer.ServerRedundancyandServer.ServiceLeveland fails over to the highest-ServiceLevel peer. It honors Maintenance (Part 4 Table 105 — disconnect to an available peer), keeps Warm backups disabled until failover (Table 107), and treats peers known only throughServerUriArrayas failover candidates.RedundantManagedClientimplements the Cold, Warm, Hot (a), Hot (b), and HotAndMirrored client patterns: one active session, optional lightweight ServiceLevel status-check sessions to backups, and — for Hot (b) — merging of concurrent reporting streams with exact value-identity de-duplication. HotAndMirrored failover re-activates the mirrored session with the existingAuthenticationTokeninstead of recreating it.Distributed state (opt-in; an extension beyond §6.6)
ISharedKeyValueStore(in-memory default) andINodeStateStore, with anAddressSpaceSynchronizerthat bridges aCustomNodeManager2's predefined nodes to the shared store, so node and reference additions/removals and value changes propagate to other replicas.IDistributedValueCachelets read/write callbacks cache the last value with a freshness bound; monitored items use the normal read pipeline and therefore participate only when the read path does.DistributedSessionManager(wired through a new additiveISessionManagerFactoryseam onStandardServer) mirrors encrypted session state across replicas. On a failover reconnect the standby restores the session and performs the fullActivateSessionclient-certificate signature validation against a single-useserverNonceconsumed via compare-and-swap across the replica set, enforcing the same SecurityPolicy and SecurityMode. TheAuthenticationTokenis a lookup key only, never an authenticator: entries are keyed by the SHA-256 digest of the token, and a cross-replica restore emits a distinct audit event.IRecordProtector; secret-bearing mirrors require a registered protector or an explicit opt-out.Active/active (CRDT)
Opc.Ua.Server.Redundancy.Crdtadds leaderless multi-writer address-space replication with CRDTs and gossip (UseReplicatedAddressSpace) plus CRDT-backed session metadata (UseReplicatedSessions). Networked gossip is fail-closed without authenticated transport (mutual TLS for TCP; an explicit opt-out for isolated dev/test). CRDT state is eventually consistent, so exactly-once decisions (such as the single-use nonce registry) stay on a strongly consistent store. The package is available on all stack target frameworks.Kubernetes
Opc.Ua.Server.Redundancy.K8sprovides Kubernetes Lease leader election, EndpointSlice-based peer discovery, and ServiceLevel-driven readiness for running an OPC UA server replica set.Docs/HighAvailabilityKubernetes.mdcovers StatefulSet/Deployment and Service manifests, RBAC, probes, time synchronization, secrets, gossip-port NetworkPolicy, and GDS/NTRS registration.Compatibility and validation
ISessionManagerFactoryandIServerStartupTask); replaced APIs marked[Obsolete]for migration.net472,net48,netstandard2.0,netstandard2.1,net8.0,net9.0,net10.0); NativeAOT-compatible where the runtime supports it.See
Docs/HighAvailability.mdfor the full OPC 10000-4 §6.6 mapping, theAdd*(standard nodes) versusUse*(extension) builder convention, and the security/rotation guidance.