Skip to content

Socket ownership and recovery

The WebSocket connection ID is its Dapr actor ID. A logical connection retains that ID across activation changes, but each activation creates its own physical socket. Socket objects cannot move between processes.

The activation's receive/reconnect supervisor has one cancellation lifetime. Deactivation cancels it, aborts the socket, and waits for the supervisor (including any already-admitted publish) before removing the local object. Recovery intent is not deleted by deactivation; user Disconnect deletes it. Retry attempts reuse the lifetime token, so cancellation cannot be undone by opening a fresh socket.

Heartbeats refresh local connected records. They also invoke non-local live auto-reconnect intents by actor ID through Dapr placement. They do not open sockets directly on every replica or revive intentionally disconnected records. The first heartbeat runs at startup. Persisted intents must still exist: CLEAR_ALL_ON_START=TRUE deliberately deletes them and disables restart recovery.

Upstream session restoration

Transport reconnection is not application-session restoration. For a server that streams immediately from its URL, the new receive loop can resume traffic. For protocols requiring authentication or subscription messages, connect with a handshake: on every open, including reconnects and handoffs, WSM asks the connection's handler for fresh messages and sends them before the connection counts as connected. WSM does not replay arbitrary SendRaw frames: they may have non-idempotent effects, and a login may be valid only once. Without a handshake, callers must restore the session themselves after the connected lifecycle event (identified by connection_id); lifecycle delivery is best-effort, so such a client must also handle missed notifications and confirm upstream readiness.

Verification requirements

A process restart alone cannot reproduce an orphan on a surviving old owner. An ownership test must deactivate actor X while the old process stays alive, wait for its socket supervisor to stop, then activate X on another process. Verify no old-owner publishes or reconnects, a single live new socket, and no remaining socket after Disconnect(X). Keep record-clearing disabled and test transport recovery separately from protocol re-subscription.

The installed Dapr.Actors.Next 1.18.10 runtime is exercised by ActorHandoffTests: two live service providers, generated actor dispatch, real upstream WebSockets, and recovery records round-tripped through the production JSON context. Runtime A deactivates X and completes cleanup before runtime B activates X; Disconnect(X) then stops the recovered supervisor. This proves the orderly SDK lifecycle path, not automatic sidecar placement, simultaneous activations, or fencing during network partitions.

NativeAOT verification

Connection records and indexes use ConnectionPersistenceJsonContext, which preserves the validating record converter. IL2026 and IL3050 are build errors. An osx-arm64 native publish completes. Dependency-level IL2104/IL3053 remain in gRPC JSON transcoding and server reflection; successful publication does not prove those endpoints work under trimming. Linux native CI and endpoint smoke tests are still required before release.

Expanded native diagnostics (TrimmerSingleWarn=false) revealed IL2026/IL3050 calls in Microsoft's JSON transcoding runtime metadata and generic construction. REST is now being retired in the separate feat/retire-rest change; the custom REST adapter has been removed from this lifecycle PR. Reflection v1 and v1alpha are registered with explicit generated protobuf descriptors, rather than reflection-based discovery of service types. No diagnostics are suppressed. The replacement publishes natively on osx-arm64 with no IL warnings; health and service/symbol reflection smoke tests passed before the adapter was removed. Linux native CI still needs validation. This PR must be reconciled with the new 0.6.0 REST-retirement baseline before publishing its next version.

Callback channel loss

SidecarPlacementTests now exercises real local placement/daprd 1.18.3 and the installed .NET SDK 1.18.10 with two live application hosts, real upstream WebSockets, and serialized recovery records. Joining sidecar B moves actors: old A supervisors finish, B publishes under the same IDs, and A counters stop increasing. Killing only sidecar A (keeping app A alive) closes its remaining sockets; B recovers their intents and Disconnect clears all local registrations. The opt-in test passed twice in approximately 21 seconds per run.

Run with RUN_SIDECAR_PLACEMENT_TESTS=true dotnet test --filter FullyQualifiedName~SidecarPlacementTests; local Docker, placement and daprd are prerequisites. The fixture uses a disposable Redis actor store and cleans up its own resources. It does not touch demo. Recovery persistence is an in-memory repository round-tripped with production JSON, not the deployed API. This test uses Dapr 1.18.3, not demo's 1.18.1.

An unresponsive-sidecar suspension probe did not complete reliably and is not included in the passing test. Therefore undetected network-partition fencing remains unverified. A rigorous no-stale-publications guarantee would require the publication receiver to enforce activation/fencing tokens atomically; local cancellation and actor ID equality alone do not provide that guarantee.

Fenced publication candidate

Publication uses heartbeat fencing. Every activation claims the API's publication fence (ChangePublicationFence) before opening its socket, then renews it every PUBLICATION_FENCE_RENEW_SECONDS (default 1) with a lease of PUBLICATION_FENCE_LEASE_SECONDS (default 10). Frames go out through the regular PublishEvent, but only while the activation's lease is current:

  • When a successor claims the fence, the old holder's next renewal is rejected (ABORTED) and it closes its socket. A stale owner therefore publishes for at most one renewal interval after the claim; frames of both owners can interleave within that window.
  • A holder that cannot renew stops publishing once its lease lapses.

No physical upstream socket exclusivity is claimed.

Claiming the token and persisting the recovery record are separate operations; the lease makes that window recoverable. A claim that cannot be persisted is revoked at once; otherwise a successor that only knows the persisted token takes over when the holder's lease expires. Deactivation and disconnect stop publishing and revoke the fence without changing its token, so they never write the recovery record (a stale activation must not overwrite its successor's).

SidecarPlacementTests runs the handoff with real daprd/placement; with FENCED_API_DLL pointing at a built Virtufin.Api.dll it also runs the real API with its own sidecar, Redis pub/sub and a fence state store (Redis, or PostgreSQL with FENCED_E2E_STATE=postgresql), asserting the old owner stops publishing after the handoff and every published frame reaches pub/sub.

The actor callback transport closes local socket lifetimes when its read stream ends or faults, before returning that failure to the SDK reconnect loop. A cached activation checks for missing local recovery intent before its next method invocation. This fails closed on observed callback-channel loss, but is not a distributed fencing token: an undetected partition can persist until the transport reports failure. Cross-process placement/partition tests remain required; the local gate and channel-loss test alone do not prove fencing.