Repository navigation
feat(runtime): add platform-owned native build and Compose profiles - #148
Conversation
[mirror] #202: fix(coordinator): retain in-flight ownership and isolate metadata waits Created-by: openjiuwen-sync Commit-by: 月可;hky1999 Merged-by: robbluo Description: Paired: [GitHub #202](#202) ↔ [GitCode !156](https://gitcode.com/openJiuwen/agent-dx/merge_requests/156) <!-- bot1-mirror-meta --> github_pr: 202 head_repo: hky1999/agent-dx head_ref: hky1999/snapshot-reference-contention-20261104 head_sha: abd6cea base_repo: openJiuwen-ai/agent-dx base_ref: master github_repo: openJiuwen-ai/agent-dx base_branch: master base_sha: ec4d210 github_user: hky1999 <!-- /bot1-mirror-meta --> GitHub PR mirror for CodeArts CI. <!-- bot1-source-body --> Snapshot/Volume publication prepared metadata before entering its commit queue and waited for Redis under the scheduler mutex. Concurrent captures could therefore lose reference updates, exhaust CAS retries and delay heartbeats. This change retains in-flight ownership, moves ordinary metadata waits off that mutex, and reuses the committed snapshot inventory instead of transferring and decoding all historical tombstones on every publication/list/GC pass. ## Changes - Read the complete snapshot/Volume catalog and compared header after queue admission. Preserve the original epoch, source-record, node-session, immutable-identity, receipt and CAS checks and retry budget. - Serialize initial artifact/Volume publication preparation in consistent lock order. Keep runtime work outside the metadata queue. Preparation failures before EVAL do not dirty publication state; ambiguous submitted writes require authoritative repair. - Keep snapshot publication/get/list/delete, node inspection and collector I/O off the global scheduler mutex. Readiness checks do not wait for a cache writer. - Retain valid same-process dispatched reservations during catalog repair; persisted records win and replacement/retired sessions are excluded. Deadlines/resources and fresh-process recovery semantics remain unchanged. - Filter committed snapshot inventory before cloning. Ownership publication borrows records to test the same storage/location predicate; Deleting still owns its artifact. An unavailable inventory cannot mean “unowned.” Standalone storage sessions without an initialized view retain their durable reads. - Fence the cached view by Redis transport generation. Reconnection/header success does not revalidate cached ownership; in-flight writers cannot restore an earlier-generation view, and a recovery crossing transport replacement fails closed. - Include master `ec4d2108`, which already contains #181, #195 and #208. ## Validation Current PR head: `abd6ceadfc4260c07c90e722b0537e39a526b2c9`. [Current-head CodeCheck](#202 (comment)) passed all five configured checks. UT/ST/package build are N/A in that report; executable evidence is separate. Clean combined source `7a0614b43078c96ff9e80bcacace129374e49503` integrates this PR and [!153](https://gitcode.com/openJiuwen/agent-dx/merge_requests/153) `3432d411`. Coordinator, platform/shared crates and the snapshot contract are byte-identical to this PR head; ADXlet is byte-identical to [!153](https://gitcode.com/openJiuwen/agent-dx/merge_requests/153). Executed gates: - **224 Coordinator tests**, including isolated real Redis contention, reservation/session fencing, write ambiguity, snapshot/Volume protection and allocation budgets. - **457 ADXlet tests**; one existing dedicated empty-Sandboxd integration remains ignored. - **88 official real Redis/mTLS control-RPC tests**; 25 library cases overlap the Coordinator group. - Formatting, full workspace/all-target/all-feature Clippy with warnings denied, documentation and diff checks pass, with unchanged source hashes. - RED reproduced redundant HVALS reads and stale ownership after real Redis restart. The hot-path, reconnect and in-flight-writer fencing regressions pass. - Fresh complete maintained platform/static-musl Execd/EROFS archives were built and verified from the same clean source. ## Current deployed fault acceptance: passed The single HK1019 deployment uses immutable framework `sha256:fb78c1fea551230feeac0c9d3da3ec846a154c6fddddaddd96b7da65108112ad` on all 13 workers and Coordinator/API/Redis. Actual running executable hashes were checked. Framework binary-build source is AKernel `478aef4d`; chart source is separately `092f2f27`. Managed tools `sha256:fbed1c21…` are independently pinned. Redis PVC and the sole public API/Gateway entry were preserved. This is combined-candidate evidence, not a bare-PR build or formal release approval. Using public native SDK/HTTP APIs: - **16 live environments**, each with an exclusive directory Volume, 64MiB high-entropy touched heap, 32MiB private file, open FD offset and established framed TCP stream. - Two synchronized 16-way batches created **32 reusable snapshots**, with a 1.5s Redis write pause and 16 concurrent unrelated Volume creations in the first batch. All 32 calls returned 200. Replaying one exact request ID/body returned the same immutable snapshot. - All 16 referenced-Volume deletions were rejected. Recreating a Volume produced a new UID; the stale UID was rejected with **409 CONFLICT, outcome=not_started, retry=never**, without replacement occupancy or references. - Replaced the existing Redis Pod using the same PVC, restarted the existing Coordinator, then replaced one worker Pod. Sources remained usable after Redis/Coordinator recovery. After worker session loss, all **16 original-process restores** proved PID/start time, nonce, heap hash, FD offset, framed TCP, private-file hash and Volume UID/data. - Cleanup reached **0 test environments and 0 Volumes**. All 32 snapshot tombstones were Deleted with empty references, and all 32 physical checkpoint directories were absent. Temporary domain/data/configuration were removed; the exact **118 baseline snapshot IDs** remained and all 13 workers were healthy. The first current-package fault attempt stopped before injection because an operator projection exported unrelated historical tombstones and exceeded the exec-output frame. That failure was retained; the fresh passing attempt uses bounded explicit HGET projections. Product code and fault assertions did not change. Scope: same-node native rootfs/runtime snapshots and real directory Volumes. Redis replacement was requested during the second capture batch; exact overlap of an individual EVAL with disconnect was not established. Committed baseline had WAITAOF; AOF everysec stayed unchanged. This does not qualify forced power-loss/ambiguous-EVAL durability, storage-side split-brain fencing, managed image-collection daemon/host restart, cross-node portability or artifact GC. Forged stale-session RPC negatives remain component/RPC evidence; the cluster negative used the public stale-UID API. ## Current 1024 concurrent C/R qualification: passed The same immutable deployment completed two globally synchronized rounds over **1024 resident subjects**: **2048 captures, 2048 restore API successes and 2048 complete original-process proofs**. All eight clients exited 0. Cleanup debt is 0, all 2048 physical snapshot directories were absent, and the exact 118 baseline snapshots remained. | Public SDK boundary | p50 | p95 | Maximum | | --- | --- | --- | --- | | Checkpoint | 33.321s | 36.553s | 38.307s | | Restore API | 2.284s | 5.231s | 7.075s | | Complete payload proof | 15.114s | 22.629s | 24.879s | Complete proof includes bounded client verification after all restore APIs return. It is separate from runtime restore latency. Each subject has 64MiB random touched heap and a 32MiB private upper file; requests are 10mCPU/64MiB/512MiB, with unchanged 2CPU/8GiB/20GiB hard caps. Captures are whole-sandbox runtime calls, without guest Python or application signals. Fourteen node samples and 16 control-plane samples had no collection errors or new observed OOM kills. Sampled peaks: Coordinator 504.31MiB, Redis 124.58MiB, busiest worker sandbox-parent 21.266GiB. Service Pod UIDs did not change during the cohort. Original Python-free 0774 net-stats and original kv-live-surgery also passed mixed Harbor Oracle plus independent verifiers (reward 1 each), two captures and original-process restore per case. Compose main/loadgen identities and a one-launch count were retained. No real model inference or Claude usage-accounting claim is made for this Oracle run. ## Failure history and remaining gates Earlier 1024 requalification on framework f86b7d3b failed after 2047 captures and 1024 restore proofs: a 300s snapshot budget was truncated by the generic 15s catalog-publication RPC timeout. The remaining-budget repair is independently in [!153](https://gitcode.com/openJiuwen/agent-dx/merge_requests/153) `3432d411`, with real-gRPC RED/GREEN tests. The current matched-package two-round pass above qualifies that integrated fix; it does not erase the old failure. Older capacity/fault evidence retains its original artifact identity. An independent cross-version EROFS identity failure was corrected by optional runtime-artifact pinning in [AKernel #148](inclusionAI/AKernel#148). A snapshot captured under actual old services then restored under the new services with the original process proof. This preserves one qualified runtime/Execd tuple, not arbitrary runtime migration or all historical snapshot compatibility. Original-Dockerfile/context startup remains a separate pending gate: the latest 128-workflow cohort reached 127/128 before native export stalled. HK BuildKit has not been changed. The independently reproduced converter cancellation fix in [Nydus Draft #817](containerd/nydus-snapshotter#817) is not deployed and is not claimed to explain the entire stall. The 1024 C/R pass establishes a tested waterline, not an absolute maximum or a Dockerfile-startup result. <!-- /bot1-source-body --> See merge request: openJiuwen/agent-dx!156
|
PR consolidation and current HK1019 deployment audit (2026-11-05):
Remaining integration review should concentrate on this PR. Independently, akernel-dev/gvisor#12 remains open: pinned release-20261012.0-akernel.3 source 63de4896444eb140221c1284fa621394666224af still uses the blocking Loader state-query lock and does not contain that fix: That runtime fix is not a duplicate of Sandboxd #98. The source/deployment gap is also recorded on akernel-dev/gvisor#12. Converter PR containerd/nydus-snapshotter#817 is closed without merge and is not deployed. This audit changes PR organization only; deployment and source worktrees are unchanged, and the main thread's pressure test is not interrupted. |
3388033 to
3804a2d
Compare
|
Rebased onto latest develop b43d548. New head: 3804a2d. Retained upstream Agent DX e611b9e4 source + matching platform/Execd URL/checksum pairs and published adx-environment==0.1.0rc4. Kept Sandboxd #101 and runtime bundle pins unchanged. Original 44 commits were replayed; a provenance-only follow-up aligns qualification-sources.json. The old 3388033 history remains reachable on the fork branch hky1999/managed-runtime-profile-before-rebase-20261106 for existing Harbor pins. Updated this PR with an explicit force-with-lease; no other worktree or delivery lock was reset. Local validation: 524 SDK tests, 35 benchmark-driver tests, Ruff/mypy, 48 runtime-adapter regressions, 19 runtime-contract tests, 77 build-helper tests, 84 standalone-helper tests, deployment syntax, Helm lint, Go 1.25.5 helper vet/tests/build, and separate clean wheel/sdist installations passed. Two unprivileged chart filesystem-fixture errors reproduce on untouched develop; all 20 relevant filesystem-fixture tests pass as root. Full new-image/cluster qualification was not run, and the PR remains Draft. No production/BuildKit deployment was changed. New-head CI is being checked separately. |
Provide image discovery, validated snapshot registration and a Docker runtime image. Preserve upstream attribution and include bridge regression tests so nested containers can reuse prepared read-only image layers. Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Delegate Dockerfile and Compose builds to official Buildx remote and publish immutable Nydus artifacts. Add authorized image preparation, collection ownership, resource metadata and native publication recovery. Include runtime regressions and operator documentation, and pin merged Sandboxd sources with matching official Agent DX platform/Execd artifacts. Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Add image, project, service and process handles with managed profile admission, network forwarding and restore attachment without replay. Preserve command timeout identity and partial results, and coalesce authenticated resource discovery. Include SDK regressions, integration probes, examples and API guidance. Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Add opt-in runtime rootfs pinning with explicit Execd verification and node configuration. Preserve writable shared-memory defaults and standalone restore settings, with chart/helper regressions and operator instructions. Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Run runtime adapter and bridge checks, validate packaged SDK helpers, and record the managed runtime maintenance and qualification contracts. Keep native plan validation distinct from live runtime acceptance. Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
12c3afd to
1657ead
Compare
rcyw
left a comment
rcyw
left a comment
There was a problem hiding this comment.
Approve for the documented opt-in managed-runtime preview and this SDK release at 1657ead0f483701c353ef785d08c099e6ccc51ba. No blocking regression found within the reviewed scope.
The design direction is reasonable: keep native Dockerfile/Compose interpretation in official tools, let the platform own build/distribution/profile authorization and collection leases, and keep SDK restore attachment free of fresh startup or command replay. Storage admission and the same-node snapshot/runtime compatibility boundary are explicit. Existing paths retain their defaults except the intentional standard private /dev/shm configuration and additive error/config handling.
Validation:
- Independently fetched the history cleanup: old
12c3afdand current1657eadhave the identical full Git treead8bc15e46913c9015c724875d3ba4b1d8bb9a7f, including gitlinks, rules, code and tests. Existing local results therefore apply to the current code without repeating unchanged suites. - Local isolated checkout: 524 SDK unit tests + 35 benchmark-driver tests, Ruff/mypy, 57 runtime-adapter unit tests, 19 runtime-contract tests, 14 rootfs-pin/shared-memory/Helm tests, and image-bridge Go tests/vet/build passed.
- Independently verified official Compose 2.39.2 / Buildx 0.29.1 release checksums and ran all 5 native offline plan cases. Native config/build --print/bake --print are real; publication metadata is simulated. These did not perform solves, Registry operations or deployment.
- Rehashed the actual cached platform/Execd archives against the PR Dockerfile pins and freshly fetched official OBS manifest: source
2f61d44d, amd64 target, all 45 platform and 4 component manifest files, and matching platform/component Execd + rootfs bytes passed. - Current-head CI 42196584062 is completed/success, including Python 3.10–3.14, distributions/quality, managed adapters/Go helper, deployment/release contracts, Standalone and runsc E2E.
Pin audit: Sandboxd d23687d equals upstream main. Agent DX 2f61d44d is two commits behind upstream master 58826bf9; master is that repository's default branch. Keeping the verified source/platform/Execd tuple for this release is reasonable, with the newer recovery changes deferred as agreed. A later update must preserve source/artifact consistency.
One reproduced non-blocking follow-up is tracked in #156: the published native rc4 dependency loses structured command-observation error metadata at the AKernel boundary in an exceptional HTTP-error fixture. It still raises, preserves HTTP status/request identity and the original cause, and did not replay a command. The existing upstream repair should be integrated through a formal dependency release and tested through clean packaged SDK installations.
This approval retains the documented preview scope. Actual managed-profile BuildKit/Registry execution and checkpoint/restore with the rebuilt pinned tool image have not been established by these local tests or the ordinary standalone CI. The separate cluster/1024-final-task qualification is not claimed here.
|
Follow-up tracking after approval: #156 records the reproduced non-blocking published-SDK command-observation metadata gap. It does not block this SDK preview release. With For this release, keeping the verified Agent DX The managed profile remains an opt-in preview. Before broad runtime enablement, record real BuildKit/Registry execution and same-node checkpoint/restore with the exact rebuilt, pinned tool image and the released SDK. Offline plan tests and ordinary standalone CI do not establish that qualification; full 1024-final-task acceptance remains separate. |
Harbor needs to run original Dockerfile and Compose tasks without client-owned image plans, Registry credentials or node paths. This opt-in platform runtime delegates native builds to official Compose/Buildx remote, publishes immutable Nydus references and prepares shared ROFS lower layers through Sandboxd/Dragonfly. Harbor retains task/environment semantics and checkpoint timing; the SDK transfers inputs and exposes artifact, project, service and process handles.
Behavior
/describepath. They do not mount task roots on disposable preparation helpers. Compose registers its required ROFS lower through the existing image manager.Dependencies and packaging
The branch includes upstream
developthrough 724238c. Current head: 1657ead. History is grouped into five functional commits; its final Git tree is identical to12c3afd26cf9a68c8ee23a56025f03f3f3bc55ba.inclusionAI/sandboxd:main, d23687d62ef0afd643b4028aedbcbc0e8bd346dc, including merged test(e2e): read Redis lifecycle limits from execution spec #101/Release akernel-sdk 0.2.0rc1 from develop for reproducible cluster validation #102 and checkpoint-lease preflight.e4390134ed9428936a241622dfd01a4420b0ddc494b3dab40bf0b9712238cf33; Execd archive SHA256:240bfcc19326faab9ea5ffa7587c9e6e5ca48241a86de74778b9a6c26403bbee.Operators build and pin the runtime tool image separately from the all-in-one node bundle. The Compose adapter fixes in this head require that rebuilt tool image; updating only a client SDK or node bundle does not replace a previously pinned tool image.
Validation
build --print/bake --printparsing is real; publication metadata is simulated. No builds, Registry requests or cluster changes are involved.sha256:b84d2a995f74af2a84435b4a8f8686320ea6b04c28ce1b83014799d8164b8aeb. The build used an existing owned idle client Pod and changed no deployments; this artifact is retained for later runtime acceptance.sha256:ef349f37be991178c777383de9fcb4c5c0517bc123b95cb23d57f1779d5b78f9with managed toolssha256:b84d2a995f74af2a84435b4a8f8686320ea6b04c28ce1b83014799d8164b8aeb. Both were built through official Buildx remote on the existing HK service. The tested source12c3afd2and official develop mergef81d0337have exactly the same Git tree:ad8bc15e46913c9015c724875d3ba4b1d8bb9a7f.adx-environment==0.1.0rc4wheel was compared with the qualified upstream2f61d44dSDK: 20 of 21 package files match, but itscommands.pylacks merged command-observation consistency handling. The Harbor delivery therefore retains the explicit upstream Git override until an indexed release includes those changes; it is not a private patch.Scope and release requirements
This is an optional preview profile. Managed snapshots require retained same-node collections, a compatible mount view, current tenant authorization, sufficient storage and a matched runtime tuple. Registry artifact GC, cross-node collection reconstruction, changed provider identity and per-tenant Registry credential issuance are outside this contract. A rootfs pin preserves filesystem identity; it does not provide arbitrary cross-version runsc checkpoint compatibility. Retain compatible runtime artifacts or explicitly drain affected snapshot generations before changing them.