Repository navigation
e2e: test async-suspended local checkpoints for restorable and disabled SandboxConfig versions - #3556
Open
Shrutiya Mohan (shrutiyam-glitch) wants to merge 5 commits into
Conversation
Shrutiya Mohan (shrutiyam-glitch)
changed the title
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
e2e-paused-runtime
branch
2 times, most recently
from
November 2, 2026 18:29
81d72d4 to
1a832b0
Compare
Shrutiya Mohan (shrutiyam-glitch)
marked this pull request as ready for review
November 2, 2026 18:36
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
e2e-paused-runtime
branch
from
November 4, 2026 00:29
ff7f728 to
e2d1e9f
Compare
A test of a paused actor needs the actor to hold a node-local MEMORY checkpoint instead of a durable snapshot. suspend takes the durability mode and returns the snapshot it checked, so a test can compare generations across a detach. Signed-off-by: Shrutiya <shrutiyam@google.com>
…d pod deletion A paused actor's MEMORY checkpoint is pinned to its node. With the old version still Enabled, it restores MEMORY in place through restorable_runtimes. Once that version is disabled, ResumeActor detaches the checkpoint to durable storage and falls back to ROOTFS on the new default. Deleting the worker pod leaves the checkpoint intact, and the actor restores MEMORY on the pool's replacement worker. Signed-off-by: Shrutiya <shrutiyam@google.com>
After an asynchronous suspend, atelet reports the background upload at some later point, and that report holds the actor's lease. A suspend that arrives meanwhile fails with Aborted, so go through RetryAborted as the other tests that pause actors do. Signed-off-by: Shrutiya <shrutiyam@google.com>
atelet detaches a paused actor's checkpoint on its own after --snapshot-detach-delay, 30s in e2e. The pause came before the SandboxConfig edit and the wait for the worker to re-register, so that timer could run out first and step 4 passed without ResumeActor detaching anything. Disable v1 while the actor runs, then pause and resume back to back. Signed-off-by: Shrutiya <shrutiyam@google.com>
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
e2e-paused-runtime
branch
from
November 11, 2026 21:57
e2d1e9f to
02edb93
Compare
Comment on lines
+259
to
+279
| // 5. Pause on v2, delete the worker pod, and resume on the replacement. | ||
| f.suspend(t, ateapi.DurabilityMode_DURABILITY_MODE_ASYNCHRONOUS, v2Runtime) | ||
| worker, err := f.clients.SubstrateAPI.GetWorker(f.ctx, &ateapi.GetWorkerRequest{Worker: f.workerRef}) | ||
| if err != nil { | ||
| t.Fatalf("failed to get Worker %s before pod deletion: %v", f.workerRef.GetName(), err) | ||
| } | ||
| if err := f.clients.K8s.CoreV1().Pods(f.namespace).Delete(f.ctx, worker.GetWorkerPod(), metav1.DeleteOptions{}); err != nil { | ||
| t.Fatalf("failed to delete worker pod %s/%s: %v", f.namespace, worker.GetWorkerPod(), err) | ||
| } | ||
| replacement := f.waitForReplacementWorker(t, v2Runtime) | ||
| afterPodDelete, err := f.clients.SubstrateAPI.GetActor(f.ctx, &ateapi.GetActorRequest{Actor: f.actorRef}) | ||
| if err != nil { | ||
| t.Fatalf("failed to get paused Actor after worker pod deletion: %v", err) | ||
| } | ||
| if got := afterPodDelete.GetStatus().GetState(); got != ateapi.ActorState_ACTOR_STATE_SUSPENDED { | ||
| t.Fatalf("paused Actor state after worker pod deletion = %v, want SUSPENDED", got) | ||
| } | ||
| status = f.resume(t, ateapi.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, v2Runtime) | ||
| if got := status.GetWorkerAssignment().GetWorker().GetName(); got != replacement.GetName() { | ||
| t.Fatalf("workerAssignment.worker after replacement resume = %q, want %q", got, replacement.GetName()) | ||
| } |
Collaborator
There was a problem hiding this comment.
I think the step 5 test is a good edge case to cover, but let's move it to its own test case since it doesn't touch SandboxConfig
🤖 blocking 🔴 – Step 5 doesn't touch SandboxConfig, and as written it still passes even if pod deletion wipes or detaches the local checkpoint:
- Waiting for the old pod to terminate and a replacement pod to start takes longer than
--snapshot-upload-delay=2s(and under load can exceed--snapshot-detach-delay=30s), soatelethas already uploaded a durableMEMORYcopy beforef.resumeruns. - Step 5 never checks
hasLocalSnapshotorAssignedNode, and whenlocalDiris missingateletsilently restoresMEMORYfrom that object-storage copy. - On a multi-node cluster,
PoolReplicas: 1with noNodeSelectorcan also place the replacement pod on another node and force a detach.
Let's drop step 5 from this PR (keeping a final f.suspend on v2Runtime at the end of step 4) and cover paused-actor pod deletion in a separate PR/test: use PoolReplicas: 2 pinned to one node via PoolTemplate.NodeSelector["kubernetes.io/hostname"] so both workers are ACTIVE before pausing, assert hasLocalSnapshot and AssignedNode stay intact across resume, and check ate_actor_restore_duration (ate_snapshot_kind="local").
Collaborator
Author
There was a problem hiding this comment.
Will cover pod deletion with two pinned workers and the local-restore metric in a separate PR..
Shrutiya Mohan (shrutiyam-glitch)
changed the title
…pend The step passed whether or not the local checkpoint survived: the wait for a replacement worker outlasts the upload delay, so atelet restores from the durable copy either way. It gets its own test later. The SandboxConfig changes come before each asynchronous suspend, not while the actor is suspended, so the doc says so. The fallback assertions move to requireDetached, which TestSandboxConfigEdit runs on its ROOTFS rows too, so sync and async suspends verify the same resume. Signed-off-by: Shrutiya <shrutiyam@google.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3008. Depends on #3147
Adds
TestPausedActorSandboxRuntimeininternal/e2e/suites/demoto cover a paused actor (DURABILITY_MODE_ASYNCHRONOUS), whoseMEMORYcheckpoint is pinned to its node, acrossSandboxConfigversion transitions and worker pod deletion:v1, call twice (memory=2, file=2, rootfs=2), and suspend with av1-stampedMEMORYsnapshotv1SandboxConfigdefaultVersiontov2withv1stillEnabled, restoreMEMORYonv1(3, 3, 3), pause, and restoreMEMORYin place (4, 4, 4)v1viarestorable_runtimes;ResumeActorrestoresMEMORYonv1without detaching the local checkpointv1inSandboxConfigwhile the actor runs on itv1; the running actor keeps runningv1and resume immediatelyResumeActordetaches thev1-stamped local checkpoint to durable storage at the same generation, then falls back toROOTFSonv2(memory=1, file=5, rootfs=5)v2, delete the worker pod, wait for the pool's replacement worker, and resumeACTOR_STATE_SUSPENDEDacross worker pod deletion and restoresMEMORYon the replacement worker (memory=2, file=6, rootfs=6)Generated with the assistance of AI (Gemini) to design and draft the e2e test and review the diff.