Repository navigation
e2e: test WorkerPool configRef changes and bad SandboxConfig asset SHA256 - #3566
Open
Shrutiya Mohan (shrutiyam-glitch) wants to merge 6 commits into
Open
Shrutiya Mohan (shrutiyam-glitch) wants to merge 6 commits into
Shrutiya Mohan (shrutiyam-glitch) wants to merge 6 commits into
Conversation
Shrutiya Mohan (shrutiyam-glitch)
marked this pull request as ready for review
November 2, 2026 20:01
Deleting the SandboxConfig a pool uses leaves it Terminating under the workerpool-protection finalizer, and a configRef to a missing config or one of the other class is refused; through all three the pool keeps its config and the worker its runtime. A switch to another config of the same class releases and deletes the old one and re-registers the worker while an actor keeps running on the old runtime. That actor's next snapshot is stamped from its assignment, not the worker's new default, and its next resume drops to ROOTFS on the new default. A second pool on the new config is up before the switch. With both pools free, the resume lands on the old pool's worker, the only one advertising the snapshot's runtime. Signed-off-by: Shrutiya <shrutiyam@google.com>
atelet records a SandboxRuntimeSyncFailed Event on the worker pod and leaves the version out: a broken restorable version is omitted, and a broken default leaves the worker on what it last registered, so a suspended actor still restores MEMORY. Once the SHA256 is fixed the worker converges to the new default with the old version restorable. Signed-off-by: Shrutiya <shrutiyam@google.com>
The scheduler drops its runtime requirement only when no worker meets it. poolB cannot restore v1, so it is never a candidate for the first pass. Signed-off-by: Shrutiya <shrutiyam@google.com>
Signed-off-by: Shrutiya <shrutiyam@google.com>
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
e2e-configref-and-assets
branch
from
November 11, 2026 21:57
328ffe1 to
a1283ec
Compare
Check that configA is still Terminating right before the switch to configB: the refused changes move spec off it, so only the switch may release it. Have waitForPoolCondition require the Resolved condition at the pool's current generation with no pendingConfigRef, so a stale status or a leaked claim cannot satisfy it. poolB runs on poolA's node, so the node pin from the suspend cannot exclude it and both workers report the same host CPU. The wait for poolA's re-registration compares against poolB's runtime. Signed-off-by: Shrutiya <shrutiyam@google.com>
atelet stops at the first asset that fails to stage, and with every SHA256 wrong it re-downloads whichever asset it visits first on each retry, the kata image on microVM. The assets shared with v1 now stay cache hits. Signed-off-by: Shrutiya <shrutiyam@google.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3008.
Adds two end-to-end tests in
internal/e2e/suites/demo:TestWorkerPoolSwitchSandboxConfig(sandboxconfig_test.go)Covers
atecontrollerhandling of aWorkerPool'sconfigRef, multi-pool scheduler placement across twoWorkerPools matching the sameActorTemplateselector, and switchingconfigRefwhile an actor is running:poolA(configAatv1,memory=2, file=2, rootfs=2), suspend with av1-stampedMEMORYsnapshot, and startpoolB(configBatv2) with the same selector labelspoolBstarts warming up in parallelconfigAwhilepoolAuses it, then pointpoolA'sconfigRefat a missingSandboxConfigand a wrong-classSandboxConfigconfigAstaysTerminatingunderate.dev/workerpool-protection;poolAreportsResolved=False(SandboxConfigDeleting,SandboxConfigNotFound,SandboxClassMismatch) while keepingconfigAinstatus.sandboxClasses[0].configRefand its worker onv1poolA(v1) andpoolB(v2)ACTIVEand free, resume the actor, then switchpoolA'sconfigReftoconfigBwhile the actor runspoolA's worker (MEMORYrestore:3, 3, 3); switchingpoolAtoconfigBreleases and deletesconfigA, re-registerspoolA's worker onv2, and leaves the running actor onv1(4, 4, 4)v1fromWorkerAssignment.sandbox_runtime(not the worker's newv2default), and the next resume drops toROOTFSonv2(memory=1, file=5, rootfs=5)TestBadVersionSHA(sandboxassets_test.go)Covers
atelethandling of aSandboxConfigversion whose assetSHA256fails verification:v1(memory=2, file=2, rootfs=2) and suspend with av1-stampedMEMORYsnapshotv1v2with a wrongSHA256whiledefaultVersionstaysv1ateletrecords aWarningSandboxRuntimeSyncFailedEvent (incomplete sandbox runtimes) on the worker pod and omitsv2fromrestorable_runtimesdefaultVersionto the brokenv2and resume the actorateletrecordsSandboxRuntimeSyncFailedforv2, keeps the worker registered onv1, and the actor restoresMEMORYonv1(3, 3, 3)v2'sSHA256v2asdefault_runtimewithv1inrestorable_runtimesAI-assisted, I reviewed and tested it.