From c6887552fd59cac435e4dc9d00836b489e9c801b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9C=88=E5=8F=AF?= Date: Fri, 6 Nov 2026 17:37:52 +0800 Subject: [PATCH 1/5] feat(runtime): add shared ROFS image bridge for nested Docker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Provide image discovery, validated snapshot registration and a Docker runtime image. Preserve upstream attribution and include bridge regression tests so nested containers can reuse prepared read-only image layers. Signed-off-by: 月可 --- builder/dind/Dockerfile | 12 + builder/image-bridge/NOTICE | 5 + builder/image-bridge/README.md | 29 +++ builder/image-bridge/discover.go | 272 ++++++++++++++++++++ builder/image-bridge/discover_test.go | 172 +++++++++++++ builder/image-bridge/go.mod | 78 ++++++ builder/image-bridge/go.sum | 266 +++++++++++++++++++ builder/image-bridge/main.go | 353 ++++++++++++++++++++++++++ builder/image-bridge/main_test.go | 74 ++++++ 9 files changed, 1261 insertions(+) create mode 100644 builder/dind/Dockerfile create mode 100644 builder/image-bridge/NOTICE create mode 100644 builder/image-bridge/README.md create mode 100644 builder/image-bridge/discover.go create mode 100644 builder/image-bridge/discover_test.go create mode 100644 builder/image-bridge/go.mod create mode 100644 builder/image-bridge/go.sum create mode 100644 builder/image-bridge/main.go create mode 100644 builder/image-bridge/main_test.go diff --git a/builder/dind/Dockerfile b/builder/dind/Dockerfile new file mode 100644 index 00000000..b9496334 --- /dev/null +++ b/builder/dind/Dockerfile @@ -0,0 +1,12 @@ +# Build from the repository root: docker build -f builder/dind/Dockerfile . +ARG GO_BUILD_IMAGE=golang:1.25.5-bookworm +FROM ${GO_BUILD_IMAGE} AS bridge +WORKDIR /src +COPY builder/image-bridge/go.mod builder/image-bridge/go.sum ./ +RUN go mod download +COPY builder/image-bridge/ ./ +RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -o /out/akernel-image-bridge . + +FROM docker:28.3.3-dind +RUN apk add --no-cache bash python3 coreutils util-linux shadow tar xz +COPY --from=bridge /out/akernel-image-bridge /opt/akernel-image-bridge diff --git a/builder/image-bridge/NOTICE b/builder/image-bridge/NOTICE new file mode 100644 index 00000000..d1c89dff --- /dev/null +++ b/builder/image-bridge/NOTICE @@ -0,0 +1,5 @@ +Managed Docker image bridge source attribution + +The snapshotter derives from tianyuzhou95's agent-worker implementation at +commit 9b3434d36086d920b6bb7f3322da642381a254b8. Retain this attribution when +redistributing the derived helper. Repository license terms apply. diff --git a/builder/image-bridge/README.md b/builder/image-bridge/README.md new file mode 100644 index 00000000..d2fe15f6 --- /dev/null +++ b/builder/image-bridge/README.md @@ -0,0 +1,29 @@ +# Managed Docker image bridge + +This helper exposes sandboxd-provided immutable ROFS image mounts as committed containerd snapshots, with private upper/work directories owned by the sandbox. It does not introduce another registry layer cache. Metadata-only image registration and source Dockerfile/Compose discovery use containerd, compose-go and BuildKit parsers. + +Source attribution is retained in [NOTICE](NOTICE). + +The operator packages this helper in the [managed runtime](../managed-docker/README.md). Platform-managed clients do not need a local helper, Go toolchain, Registry credentials or node paths. The following discovery/startup details apply to client-managed mode. + +Build a Linux/amd64 helper from a host with the Go version required by [go.mod](go.mod): + +```sh +cd builder/image-bridge +CGO_ENABLED=0 go test ./... +CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -o /path/to/akernel-image-bridge . +``` + +In client-managed mode, the Python SDK's `prepare_compose_images()` invokes `--mode discover`, resolves tags and verifies OCI manifest/config bytes. Credentials stay on the client; the runtime image manager must independently be authorized to fetch the pinned images. Only metadata is uploaded. Layer bytes are mounted by the existing sandboxd OCI/ROFS provider, which owns node-wide caching and distribution. + +`DockerRuntime.sandbox_options()` supplies the ROFS mounts and `fileBackedTmpfsMounts` to fresh creation and snapshot restore. `DockerRuntime.start()` uploads the helper when configured, starts its private containerd and bridge, registers original image aliases, then starts dockerd. Call start only once on a fresh sandbox; never on a restored sandbox. Failed startup is cleaned up by the owner deleting the newly created sandbox, not by touching shared node state. + +Automatic Compose uses these mounted roots for image-only services as well as build bases. After registration, the SDK verifies aliases against their pinned target digests and checks the committed ROFS snapshots. It does not use native Compose pull for those inputs: Docker Compose can pull digest references even with `--policy missing`, which would incorrectly enter the ordinary layer extraction path. Automatic `up` also disables repeat pulls. An absent registration fails instead of bypassing the mount plan or silently copying image layers into private storage. Different image manifests/configurations can legitimately share one chain ID; the validated configuration and chain bind their ROFS roots before verification allows that sharing. + +Client-managed discovery requires **discovery schema 2**. Check with `akernel-image-bridge --mode capabilities` before uploading project data. The probe sends no project data. Discovery rejects Dockerfile/build-context paths or `.env` links escaping the frozen project and does not inherit unspecified client environment variables. External `include` / `extends.file`, remote contexts and platform emulation are outside client-managed discovery; pass explicit supported Compose files instead. Use matching SDK and helper capability contracts. + +Use `DockerRuntime.default_image`, the public Docker Hub `docker.io/library/docker:28.3.3-dind` image. Fresh startup installs missing Alpine support packages, so no custom runtime image build or push is required. An explicitly supplied outer image must provide Docker 28.3.3, containerd/ctr, Python 3, findmnt/mount, Bash, iproute tools, iptables-legacy and iptables-legacy-restore. `builder/dind/Dockerfile` optionally packages these and the guest helper; use repository root as build context. The SDK can upload the matching helper from the client. No helper binary is downloaded automatically. + +The default Linux/amd64 runsc profile preserves default/custom IPv4 Compose bridge networks, DNS and TCP/UDP userland port mappings. The operator must enable `extra_args = ["--net-raw=true", "--allow-packet-socket-write=true"]` under `[plugin.runtime.runsc]` for Docker 28. The SDK owns in-sandbox legacy forwarding/SNAT and per-service tunnel entries; Docker owns bridges/IPAM/DNS. After restore call `ComposeProject.reconnect_network()` to rebind the new outer TAP without restarting Docker. Task Dockerfiles and Compose files remain unchanged. Host networking is an explicit compatibility option. Remote contexts, runtime-created unknown image references and platform emulation remain outside this profile. Client-managed native builds run inside the sandbox; the platform-managed profile delegates builds to remote BuildKit. This is not a complete shared Docker content store: exporting/saving/pushing an image may still fetch layers. + +The file-backed upper requires sandboxd managed tmpfs support and runsc checkpoint support for those mounts. It consumes the sandbox storage limit. Reflink must be configured by the deployment, on a compatible filesystem and same mount; the SDK does not change deployment durability policy or replace runtime bundles. Memory mode is an explicit alternative and counts file contents toward memory snapshots. diff --git a/builder/image-bridge/discover.go b/builder/image-bridge/discover.go new file mode 100644 index 00000000..bf569be4 --- /dev/null +++ b/builder/image-bridge/discover.go @@ -0,0 +1,272 @@ +package main + +// Discovery uses the Compose and Dockerfile implementations, not regular +// expressions over user files. The original project is still built by Docker. +import ( + "context" + "encoding/json" + "fmt" + "io" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + + "github.com/compose-spec/compose-go/v2/cli" + "github.com/compose-spec/compose-go/v2/types" + "github.com/distribution/reference" + "github.com/moby/buildkit/frontend/dockerfile/instructions" + "github.com/moby/buildkit/frontend/dockerfile/parser" + "github.com/moby/buildkit/frontend/dockerfile/shell" + "gopkg.in/yaml.v3" +) + +func projectPath(root, path string) (string, error) { + if !filepath.IsAbs(path) { + path = filepath.Join(root, path) + } + resolved, err := filepath.EvalSymlinks(path) + if err != nil { + return "", fmt.Errorf("cannot resolve managed project input: %w", err) + } + relative, err := filepath.Rel(root, resolved) + if err != nil || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) { + return "", fmt.Errorf("managed Compose inputs must stay within the project directory") + } + return resolved, nil +} + +func validateComposeSources(root string, files []string) error { + // Only explicitly supplied top-level files may live outside the frozen + // context (SDK overlays do). Never let user YAML load other client files. + for _, file := range files { + data, err := os.ReadFile(file) + if err != nil { + return err + } + var document map[string]any + if err = yaml.Unmarshal(data, &document); err != nil { + return err + } + if document["include"] != nil { + return fmt.Errorf("managed Compose discovery does not support include; supply explicit project files") + } + services, _ := document["services"].(map[string]any) + for _, value := range services { + service, _ := value.(map[string]any) + extends, _ := service["extends"].(map[string]any) + if extends["file"] != nil { + return fmt.Errorf("managed Compose discovery does not support extends.file; supply explicit project files") + } + } + } + if _, err := os.Lstat(filepath.Join(root, ".env")); err == nil { + _, err = projectPath(root, ".env") + return err + } else if !os.IsNotExist(err) { + return err + } + return nil +} + +func dockerfileImages(source string, build *types.BuildConfig) ([]string, error) { + parsed, err := parser.Parse(strings.NewReader(source)) + if err != nil { + return nil, err + } + stages, args, err := instructions.Parse(parsed.AST, nil) + if err != nil { + return nil, err + } + values := map[string]string{ + "BUILDPLATFORM": "linux/amd64", "BUILDOS": "linux", "BUILDARCH": "amd64", "BUILDVARIANT": "", + "TARGETPLATFORM": "linux/amd64", "TARGETOS": "linux", "TARGETARCH": "amd64", "TARGETVARIANT": "", + } + lex := shell.NewLex(parsed.EscapeToken) + expand := func(word string) (string, error) { + env := make([]string, 0, len(values)) + for k, v := range values { + env = append(env, k+"="+v) + } + value, _, err := lex.ProcessWord(word, shell.EnvsFromSlice(env)) + return value, err + } + for _, command := range args { + for _, arg := range command.Args { + if override, ok := build.Args[arg.Key]; ok && override != nil { + values[arg.Key] = *override + continue + } else if ok { + if value, present := os.LookupEnv(arg.Key); present { + values[arg.Key] = value + continue + } + } + if arg.Value != nil { + value, err := expand(*arg.Value) + if err != nil { + return nil, err + } + values[arg.Key] = value + } + } + } + var images []string + knownStages := map[string]bool{} + for _, stage := range stages { + knownStages[strings.ToLower(stage.Name)] = true + } + add := func(name string, allowStage bool) error { + if strings.EqualFold(name, "scratch") || (allowStage && knownStages[strings.ToLower(name)]) { + return nil + } + if allowStage { + if _, err := strconv.Atoi(name); err == nil { + return nil + } + } + if context, ok := build.AdditionalContexts[name]; ok { + if !strings.HasPrefix(context, "docker-image://") { + return fmt.Errorf("automatic image preparation needs a docker-image context for %s", name) + } + name = strings.TrimPrefix(context, "docker-image://") + } + ref, err := reference.ParseNormalizedNamed(name) + if err != nil { + return fmt.Errorf("invalid external image %q: %w", name, err) + } + images = append(images, reference.TagNameOnly(ref).String()) + return nil + } + previousStages := map[string]bool{} + for _, stage := range stages { + if stage.Platform != "" { + platform, err := expand(stage.Platform) + if err != nil || platform != "linux/amd64" { + return nil, fmt.Errorf("automatic image preparation supports linux/amd64, got %q", platform) + } + } + name, err := expand(stage.BaseName) + if err != nil { + return nil, err + } + if !previousStages[strings.ToLower(name)] { + if err = add(name, false); err != nil { + return nil, err + } + } + previousStages[strings.ToLower(stage.Name)] = true + for _, command := range stage.Commands { + if copy, ok := command.(*instructions.CopyCommand); ok && copy.From != "" { + if err = add(copy.From, true); err != nil { + return nil, err + } + } + if run, ok := command.(*instructions.RunCommand); ok { + for _, mount := range instructions.GetMounts(run) { + if mount.From != "" { + if err = add(mount.From, true); err != nil { + return nil, err + } + } + } + } + } + } + return images, nil +} + +func discover(input io.Reader, output io.Writer) error { + var request struct { + Directory string `json:"directory"` + Files []string `json:"files"` + } + if err := json.NewDecoder(input).Decode(&request); err != nil { + return err + } + root, err := filepath.Abs(request.Directory) + if err != nil { + return err + } + root, err = filepath.EvalSymlinks(root) + if err != nil { + return err + } + if err = validateComposeSources(root, request.Files); err != nil { + return err + } + options, err := cli.NewProjectOptions(request.Files, + cli.WithWorkingDirectory(request.Directory), cli.WithName("harbor"), + cli.WithOsEnv, cli.WithEnvFiles(), cli.WithDotEnv, + cli.WithoutEnvironmentResolution, + ) + if err != nil { + return err + } + project, err := options.LoadProject(context.Background()) + if err != nil { + return err + } + seen := map[string]bool{} + for _, service := range project.Services { + if service.Platform != "" && service.Platform != "linux/amd64" { + return fmt.Errorf("service %s requires unsupported platform", service.Name) + } + if service.Build == nil { + ref, err := reference.ParseNormalizedNamed(service.Image) + if err != nil { + return err + } + seen[reference.TagNameOnly(ref).String()] = true + continue + } + build := service.Build + contextPath, err := projectPath(root, build.Context) + if err != nil { + return fmt.Errorf("service %s build context: %w", service.Name, err) + } + build.Context = contextPath + build.Args = build.Args.Resolve(func(key string) (string, bool) { + value, ok := project.Environment[key] + return value, ok + }) + for _, platform := range build.Platforms { + if platform != "linux/amd64" { + return fmt.Errorf("service %s requires unsupported build platform", service.Name) + } + } + source := build.DockerfileInline + if source == "" { + name := build.Dockerfile + if name == "" { + name = "Dockerfile" + } + if !filepath.IsAbs(name) { + name = filepath.Join(build.Context, name) + } + name, err = projectPath(root, name) + if err != nil { + return fmt.Errorf("service %s Dockerfile: %w", service.Name, err) + } + data, err := os.ReadFile(name) + if err != nil { + return fmt.Errorf("read Dockerfile for service %s: %w", service.Name, err) + } + source = string(data) + } + images, err := dockerfileImages(source, build) + if err != nil { + return fmt.Errorf("discover bases for service %s: %w", service.Name, err) + } + for _, image := range images { + seen[image] = true + } + } + images := make([]string, 0, len(seen)) + for image := range seen { + images = append(images, image) + } + sort.Strings(images) + return json.NewEncoder(output).Encode(map[string]any{"references": images}) +} diff --git a/builder/image-bridge/discover_test.go b/builder/image-bridge/discover_test.go new file mode 100644 index 00000000..6dc9e223 --- /dev/null +++ b/builder/image-bridge/discover_test.go @@ -0,0 +1,172 @@ +package main + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/compose-spec/compose-go/v2/types" +) + +func TestDockerfileImageDiscovery(t *testing.T) { + version := "bookworm" + build := &types.BuildConfig{Args: types.MappingWithEquals{"VERSION": &version}} + images, err := dockerfileImages(`ARG VERSION=alpine +ARG BASE=debian:${VERSION}-slim +FROM --platform=$TARGETPLATFORM ${BASE} AS build +RUN < 4 && fields[4] == seed.Target { + mounted = true + break + } + } + if !mounted { + if err = bindReadOnly(seed.Source, seed.Target); err != nil { + return err + } + } + } + } else if !os.IsNotExist(err) { + return err + } + if err = os.MkdirAll(filepath.Dir(socket), 0700); err != nil { + return err + } + listener, err := net.Listen("unix", socket) + if err != nil { + return err + } + server := grpc.NewServer() + snapshotsapi.RegisterSnapshotsServer(server, snapshotservice.FromSnapshotter(b)) + log.Printf("image bridge snapshotter ready: %s", socket) + return server.Serve(listener) +} + +func register(plan Plan, planDir, address string) error { + client, err := containerd.New(address) + if err != nil { + return err + } + defer client.Close() + ctx := namespaces.WithNamespace(context.Background(), "moby") + ctx, done, err := client.WithLease(ctx) + if err != nil { + return err + } + defer done(ctx) + cs := client.ContentStore() + for _, image := range plan.Images { + for _, blob := range image.Blobs { + if filepath.Base(blob.File) != blob.File { + return errors.New("invalid metadata file name") + } + data, err := os.ReadFile(filepath.Join(planDir, blob.File)) + if err != nil { + return err + } + labels := map[string]string{} + var object struct { + Config *ocispec.Descriptor `json:"config"` + Manifests []ocispec.Descriptor `json:"manifests"` + } + if err = json.Unmarshal(data, &object); err != nil { + return err + } + if object.Config != nil { + labels["containerd.io/gc.ref.content.config"] = object.Config.Digest.String() + } + for n, d := range object.Manifests { + labels[fmt.Sprintf("containerd.io/gc.ref.content.m.%d", n)] = d.Digest.String() + } + if blob.Descriptor.Digest == image.Config.Digest { + labels["containerd.io/gc.ref.snapshot.akernel"] = image.ChainID + } + if err = content.WriteBlob(ctx, cs, blob.Descriptor.Digest.String(), bytes.NewReader(data), blob.Descriptor, content.WithLabels(labels)); err != nil { + return err + } + } + sn := client.SnapshotService("akernel") + if _, err = sn.Stat(ctx, image.ChainID); errdefs.IsNotFound(err) { + key := "seed-" + strings.TrimPrefix(image.ChainID, "sha256:") + if _, err = sn.Prepare(ctx, key, "", snapshots.WithLabels(map[string]string{rootLabel: image.Root})); err != nil { + return err + } + if err = sn.Commit(ctx, image.ChainID, key, snapshots.WithLabels(map[string]string{rootLabel: image.Root})); err != nil { + return err + } + } else if err != nil { + return err + } + existing, err := client.ImageService().Get(ctx, image.Reference) + if errdefs.IsNotFound(err) { + _, err = client.ImageService().Create(ctx, images.Image{Name: image.Reference, Target: image.Target}) + } else if err == nil && existing.Target.Digest != image.Target.Digest { + return errors.New("image reference changed") + } + if err != nil { + return err + } + log.Printf("registered mounted image without layer transfer: %s chain=%s", image.Reference, image.ChainID) + } + return nil +} + +func main() { + mode := flag.String("mode", "serve", "serve, register or discover") + planPath := flag.String("plan", "", "mounted image plan JSON") + root := flag.String("root", "/var/lib/docker/akernel-snapshotter", "snapshot metadata and writable layers") + socket := flag.String("socket", "/run/akernel-image-bridge/snapshotter.sock", "snapshotter socket") + address := flag.String("containerd", "/run/akernel-image-bridge/containerd.sock", "containerd socket") + flag.Parse() + if *mode == "capabilities" { + returnErr := json.NewEncoder(os.Stdout).Encode(map[string]any{"discover_schema": 2}) + if returnErr != nil { + log.Fatal(returnErr) + } + return + } + if *mode == "discover" { + if err := discover(os.Stdin, os.Stdout); err != nil { + log.Fatal(err) + } + return + } + plan, err := readPlan(*planPath) + if err != nil { + log.Fatal(err) + } + if *mode == "serve" { + err = serve(plan, *root, *socket, *planPath) + } else if *mode == "register" { + err = register(plan, filepath.Dir(*planPath), *address) + } else { + err = errors.New("unknown mode") + } + if err != nil { + log.Fatal(err) + } +} diff --git a/builder/image-bridge/main_test.go b/builder/image-bridge/main_test.go new file mode 100644 index 00000000..e7d8f983 --- /dev/null +++ b/builder/image-bridge/main_test.go @@ -0,0 +1,74 @@ +package main + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/containerd/containerd/errdefs" + "github.com/containerd/containerd/snapshots" + "github.com/opencontainers/go-digest" + ocispec "github.com/opencontainers/image-spec/specs-go/v1" +) + +func TestUnregisteredLayersCannotFallBackToDownload(t *testing.T) { + // A nil underlying snapshotter proves rejection precedes ordinary Prepare. + b := &bridge{allowed: map[string]bool{}} + _, err := b.Prepare(context.Background(), "layer", "", snapshots.WithLabels(map[string]string{"containerd.io/snapshot.ref": "sha256:abc"})) + if !errdefs.IsFailedPrecondition(err) { + t.Fatalf("Prepare()=%v, want failed precondition", err) + } +} + +func TestRuntimePlanAuthorizesNewImmutableRoot(t *testing.T) { + path := filepath.Join(t.TempDir(), "plan.json") + write := func(plan Plan) { + t.Helper() + data, err := json.Marshal(plan) + if err != nil { + t.Fatal(err) + } + if err = os.WriteFile(path, data, 0600); err != nil { + t.Fatal(err) + } + } + write(Plan{Automatic: true}) + b := &bridge{allowed: map[string]bool{}, planPath: path} + d := digest.FromString("built after sandbox startup") + root := "/opt/akernel-images/" + d.Encoded() + if allowed, err := b.rootAllowed(root); err != nil || allowed { + t.Fatalf("unexpected initial permission: %t %v", allowed, err) + } + write(Plan{Automatic: true, Images: []Image{{Reference: "registry.test/poc/main@" + d.String(), Root: root, ChainID: d.String(), Target: ocispec.Descriptor{Digest: d}}}}) + if allowed, err := b.rootAllowed(root); err != nil || !allowed { + t.Fatalf("new runtime root was not authorized: %t %v", allowed, err) + } + if allowed, err := b.rootAllowed("/etc"); err != nil || allowed { + t.Fatalf("foreign root authorized: %t %v", allowed, err) + } +} + +func TestInvalidRuntimePlanCannotAuthorizeRoot(t *testing.T) { + path := filepath.Join(t.TempDir(), "plan.json") + d := digest.FromString("artifact") + data := `{"images":[{"reference":"registry.test/poc/main@` + d.String() + `","root":"/etc","chain_id":"` + d.String() + `","target":{"digest":"` + d.String() + `"}}]}` + if err := os.WriteFile(path, []byte(data), 0600); err != nil { + t.Fatal(err) + } + b := &bridge{allowed: map[string]bool{}, planPath: path} + _, err := b.Prepare(context.Background(), "root", "", snapshots.WithLabels(map[string]string{rootLabel: "/etc"})) + if !errdefs.IsInvalidArgument(err) || !strings.Contains(err.Error(), "invalid runtime image plan") { + t.Fatalf("invalid plan: %v", err) + } +} + +func TestUnregisteredRootCannotBeMounted(t *testing.T) { + b := &bridge{allowed: map[string]bool{}} + _, err := b.Prepare(context.Background(), "layer", "", snapshots.WithLabels(map[string]string{rootLabel: "/etc"})) + if !errdefs.IsInvalidArgument(err) { + t.Fatalf("Prepare()=%v, want invalid argument", err) + } +} From 53571165e80b50f455c1ed06245ea668fc6667f0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9C=88=E5=8F=AF?= Date: Fri, 6 Nov 2026 17:37:52 +0800 Subject: [PATCH 2/5] feat(runtime): add platform-managed native build and Compose profiles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Delegate Dockerfile and Compose builds to official Buildx remote and publish immutable Nydus artifacts. Add authorized image preparation, collection ownership, resource metadata and native publication recovery. Include runtime regressions and operator documentation, and pin merged Sandboxd sources with matching official Agent DX platform/Execd artifacts. Signed-off-by: 月可 --- builder/managed-docker/Dockerfile | 12 + builder/managed-docker/README.md | 137 ++++ builder/managed-docker/compose.py | 712 ++++++++++++++++++ .../control-capacity.example.yaml | 12 + builder/managed-docker/image_provider.py | 379 ++++++++++ builder/managed-docker/profile.example.toml | 19 + builder/managed-docker/profile_metadata.py | 38 + builder/managed-docker/provider_service.py | 179 +++++ .../managed-docker/qualification-sources.json | 63 ++ builder/managed-docker/test_compose.py | 228 ++++++ builder/managed-docker/test_compose_native.py | 149 ++++ .../managed-docker/test_compose_outputs.py | 192 +++++ builder/managed-docker/test_export_retry.py | 192 +++++ builder/managed-docker/test_image_provider.py | 176 +++++ .../managed-docker/test_profile_metadata.py | 17 + .../managed-docker/test_provider_service.py | 135 ++++ builder/managed-docker/test_publication.py | 397 ++++++++++ builder/node.Dockerfile | 5 +- builder/runtime.Dockerfile | 4 +- .../akernel-managed-images.service | 15 + src/agent-dx | 2 +- src/sandboxd | 2 +- 22 files changed, 3059 insertions(+), 6 deletions(-) create mode 100644 builder/managed-docker/Dockerfile create mode 100644 builder/managed-docker/README.md create mode 100644 builder/managed-docker/compose.py create mode 100644 builder/managed-docker/control-capacity.example.yaml create mode 100644 builder/managed-docker/image_provider.py create mode 100644 builder/managed-docker/profile.example.toml create mode 100644 builder/managed-docker/profile_metadata.py create mode 100644 builder/managed-docker/provider_service.py create mode 100644 builder/managed-docker/qualification-sources.json create mode 100644 builder/managed-docker/test_compose.py create mode 100644 builder/managed-docker/test_compose_native.py create mode 100644 builder/managed-docker/test_compose_outputs.py create mode 100644 builder/managed-docker/test_export_retry.py create mode 100644 builder/managed-docker/test_image_provider.py create mode 100644 builder/managed-docker/test_profile_metadata.py create mode 100644 builder/managed-docker/test_provider_service.py create mode 100644 builder/managed-docker/test_publication.py create mode 100644 builder/systemd_services/akernel-managed-images.service diff --git a/builder/managed-docker/Dockerfile b/builder/managed-docker/Dockerfile new file mode 100644 index 00000000..188a8a4f --- /dev/null +++ b/builder/managed-docker/Dockerfile @@ -0,0 +1,12 @@ +# This image contains tools/configuration only. Session credentials are injected +# separately by the runtime and must never be baked into image layers. +FROM docker:28.3.3-dind +RUN apk add --no-cache python3 bash coreutils util-linux shadow tar xz ca-certificates +COPY --chmod=0755 akernel-image-bridge /opt/akernel-image-bridge +COPY image_bridge_guest.py /usr/lib/akernel/image_bridge_guest.py +COPY docker_network_guest.py /usr/lib/akernel/docker_network_guest.py +COPY compose.py /usr/lib/akernel/compose.py +RUN test -s /usr/lib/akernel/compose.py && \ + test -s /usr/lib/akernel/image_bridge_guest.py && \ + test -s /usr/lib/akernel/docker_network_guest.py && \ + python3 -m py_compile /usr/lib/akernel/*.py diff --git a/builder/managed-docker/README.md b/builder/managed-docker/README.md new file mode 100644 index 00000000..7c745651 --- /dev/null +++ b/builder/managed-docker/README.md @@ -0,0 +1,137 @@ +# Managed Docker runtime + +This opt-in runtime profile delegates Dockerfile/Compose semantics to Docker's official Buildx remote driver and Compose/Bake frontend. It connects to an operator-provided BuildKit service without changing that service's deployment. `compose.py` changes publication targets and generates a separate image-reference override; it does not rewrite submitted task files or interpret Dockerfile instructions. + +The runtime tool image contains Docker, Compose, Buildx and the AKernel snapshotter/registration helper. Session endpoints and credentials are injected at launch, outside the image. Buildx pushes native Nydus OCI artifacts directly to an internal CNCF Distribution Registry and returns digest references. No OCI archive passes through the Harbor client and no task artifact is published to ACR. + +Build targets are matched to services through native Compose service labels, +preserving case and punctuation. Image-only projects need no native build +targets; their images still pass through the native format-conversion exporter. +Publication names are unique within each invocation, and `outputs.json` retains +the service-to-target/reference mapping used to read Buildx results. Image +references are opaque handles; consumers must not derive them by lowercasing a +service name. A missing or ambiguous build target is an error, even when the +service also declares an image. + +`image_provider.py` runs on the runtime node. Its session capability accepts only immutable references in an operator-authorized Registry/repository prefix. It validates OCI metadata, calls sandboxd's private image-management Unix socket, and exports existing ROFS through a read-only collection. Dragonfly and the node's existing image manager own content distribution/caching; this adapter does not maintain a second layer cache. `akernel-image-bridge` imports checked metadata and committed roots into inner containerd; unknown extraction roots still fail closed. Private service uppers remain inside the outer sandbox's checkpoint boundary. + +The Registry requires private routes from build workers for push and from runtime nodes for pull. Harbor clients do not require direct Registry access or Kubernetes credentials. Registry credentials are separate from AKernel API keys. Use TLS with a trust chain available to build workers, runtime nodes and the configured distribution path. Private HTTP requires an explicit isolated-network policy and compatible Dragonfly configuration. Standard scoped Bearer authentication can retain access control without applying a password hash to each manifest/blob request. `registry.insecure` is an explicit operator option, not a default. + +BuildKit owns cache validity. Worker-local caches reuse steps on the same worker. Cache persistence and request affinity depend on the BuildKit deployment and are not guaranteed by this adapter. Cross-worker registry cache import/export is a separate native Buildx policy; runtime Nydus artifacts must not be represented as BuildKit cache records. ROFS and Dragonfly cache final content independently from build-step caches. + +The runtime retains streamed Buildx diagnostics and allows at most three attempts +of the same native Bake command when Buildx definitively exits1 with the terminal +`failed to get compression blob "nydus": not found` export error, including its +`merge nydus layers: get compression blob` variant. Any native +`no-cache` target limits the operation to one attempt. Build failures, access +errors, process/transport exceptions, and signal exits propagate directly. This +is bounded recovery for that specific exporter/cache failure; it does not +repair the BuildKit implementation or establish an exactly-once contract for +external side effects in Dockerfile instructions. Final metadata and runtime +image registration are used only after a successful Bake result. + +`ComposeProject.bind_runtime(sandbox, descriptor)` binds a newly prepared runtime and permits initial peer-restricted tunnel creation. `ComposeProject.attach()` remains the restore binding: it must not start daemons, build, run Compose up or create replacement tunnels. `network_mode: service:...` is resolved using Docker's native namespace-owner inspection, preserving the task's network configuration. + +Single-container `image-build` returns verified artifact/process metadata through the authorized `/describe` operation without mounting or registering the task rootfs on its temporary build helper's node. The final single-container sandbox mounts that image through the ordinary runtime path. Compose `/prepare` still registers the ROFS lower required by inner containerd. Both operations retain digest, platform and repository-prefix validation and use the same session capability. + +Versioned managed collections use Sandboxd's node-local live/checkpoint leases. The broker holds the collection's `flock` during each operation and sends its explicit owner to the private image-management API. The last live/in-flight/retained-checkpoint reference releases exported roots and management ownership while preserving ordinary rootfs consumers; retained managed checkpoints intentionally keep dependencies alive and count toward the node's ROFS budget. Missing/copied checkpoint markers fail closed. Legacy collections remain pinned for an operator drain because their history cannot be inferred safely. Deploy the broker and Sandboxd together: the new management API rejects anonymous ownership, and old Sandboxd rejects the new owner field. See Sandboxd's checkpoint/restore contract for local-filesystem and recovery requirements. Cross-node reconstruction and Registry artifact GC remain separate work. + +Run adapter regressions with `python3 -m unittest discover -s builder/managed-docker -p 'test_*.py'`. SDK changes additionally require `make sdk-check`; helper changes require `go test ./...` in `builder/image-bridge`. Use an owned namespace/data directory and delete sandbox/snapshot records before unmounting collections or deleting Registry storage. + +To also validate plans with official Compose and Buildx, supply executable paths +from the selected runtime toolchain: + +```sh +AKERNEL_TEST_DOCKER_COMPOSE=/path/to/docker-compose \ +AKERNEL_TEST_BUILDX=/path/to/docker-buildx \ +python3 -m unittest discover -s builder/managed-docker -p 'test_*.py' +``` + +These additional tests use an empty client configuration and an unavailable +Docker socket. They exercise native parsing and `bake --print`, without a +daemon, builds, Registry traffic or cluster changes. Real publication and +checkpoint/restore require separate runtime integration tests. + +## Platform-owned profile + +Native builds publish to a fresh invocation tag and return immutable digests. +If Buildx exits with the specific post-publication `unknown blob ... in history` +error, the runtime inspects every planned output with the official Docker CLI +and verifies its raw manifest digest, size and platform. It adopts the published +outputs only after all checks pass, records the original error in a publication +receipt, and does not replay the build. A separate exact internal-deadline case +requires completed native manifest-push acknowledgements for every target and +matching Registry digests; explicit attestations are ineligible for this recovery. +Buildx exit 100 alone does not establish publication. Metadata-file provenance +defaults to Docker's supported `disabled` mode; an explicit operator setting is +preserved. This skips the optional client history lookup, without claiming to +repair server-side history or storage failures. Missing or partial publication, ordinary +build errors, authentication failures and unknown command outcomes still fail. + +Managed builds export the original native Docker/Buildx targets directly. They do not append an application interpreter, synthetic harness layer or PATH entries. Whole-sandbox checkpoint/restore is performed externally by Sandboxd/runsc and requires no Python or process-tree stop/continue in task images. Runtime control tools may use Python in the platform's own tools image; this is separate from application image requirements. + +`registry_auth_file` accepts Sandboxd's flat credential map or a standard Docker `auths` configuration for the platform-owned build client. Configure the image manager with a separate pull-only credential; do not give its account push permissions. With standard Registry token authentication, Buildx, the image manager and Nydus exchange credentials for short-lived, repository-scoped Bearer tokens. The private Registry and issuer require authenticated access; private networking is not an authorization policy. + +The runtime broker delegates bounded, digest-addressed manifest/config reads to Sandboxd's private `/registry_metadata` Unix interface. Sandboxd reuses `go-containerregistry`'s standard challenge, scoped token reuse and refresh implementation. Neither the SDK nor the Python broker parses credentials or implements a second Bearer protocol. Metadata size/digest validation and collection repository-prefix checks remain mandatory; blob data continues through the existing Dragonfly/ROFS path. The Unix interface is platform-only and is never mounted in a sandbox. + +See [the operator-only Sandboxd configuration](profile.example.toml). + +Size the control plane separately from task sandboxes. The +[core-chart capacity overlay](control-capacity.example.yaml) provides an example +Coordinator budget for environment catalogs and retained snapshot tombstones. It changes no task resource limits and is not a concurrency +guarantee. Apply it as an operator Helm values file, nested under `core` when +using the umbrella chart. Track Coordinator memory, restarts and request latency; +catalog filtering fixes do not eliminate the need for a suitable control budget. + +Harbor's maintained managed path sends `managed_runtime_profile="harbor-v1"`. +`DockerRuntime(profile=...)` forwards only that capability in `extra_config`. +Sandboxd authorizes the trusted ADX tenant label against the operator allowlist, +chooses the pinned tool image and creates a distinct collection for each fresh +environment, including independent verifiers. It injects readonly configuration, +credentials and collection mounts; client-selected reserved mounts are rejected. + +Package `provider_service.py` and `image_provider.py` on the runtime node. The +all-in-one image includes the opt-in `akernel-managed-images.service`; operators +supply `/etc/akernel/managed-runtime.env` containing `MANAGED_RUNTIME_ROOT`, +`MANAGED_IMAGE_SOCKET`, `MANAGED_IMAGE_PORT`, configure matching Sandboxd +`[plugin.managed_runtime]` / `[plugin.managed_runtime.profiles.harbor-v1]`, then +run `systemctl enable --now akernel-managed-images.service`. No unit/profile is enabled by default. Keep the private broker +and Registry on reachable internal routes. Never expose the image Unix socket. + +The Sandboxd profile owns `image` (digest pinned), `registry`, `repository_prefix`, +`registry_auth_file`, `registry_insecure` (explicit private-HTTP configuration only), +`buildkit_endpoint`, `provider_url`, `bootstrap_root`, `allowed_tenants`, `writable_mb`, and `minimum_root_mb` (defaults to 4096 MiB). The total writable-storage floor is the image upper plus this root-layer reservation. Before starting each worker, run `python3 builder/managed-docker/profile_metadata.py /path/to/sandboxd.toml` and merge the resulting labels into its ADXlet service `config.labels`. Labels contain only profile names and minimum MiB; regenerate them with every storage-geometry change. The SDK queries this authenticated resource metadata using its own connection before allocating a managed sandbox, reserves the default floor, and adds a hard placement condition for compatible workers. Explicit requests or limits below the floor fail before creation. Sandboxd checks the actual total quota again before any collection filesystem changes. Ordinary image sandboxes do not perform this preflight. + +Managed Compose inputs are frozen and uploaded unchanged. The runtime's official `docker compose config` resolves services after startup, including native `!reset` and `!override` tags; the SDK does not parse or merge managed YAML. Call `start_runtime()` before obtaining service handles. + +The snapshot compatibility fingerprint excludes `allowed_tenants`. Restore still checks the tenant against the current allowlist first: expanding or reordering authorization preserves compatibility; revocation denies restore. Tool, bootstrap, storage and other captured runtime configuration changes remain incompatible. An incompatible fingerprint/schema change requires draining the affected snapshot generation before a profile upgrade. Node-local lease recovery does not imply cross-node portability or a qualified arbitrary runtime upgrade. +The operator, rather than the Harbor user, builds and publishes the tool image +from this directory and supplies the matching image-bridge binary in its context. + +Checkpoint output includes `managed-runtime.json` with an opaque collection ID, +tenant and profile fingerprint, without credentials or node paths. Restore checks +this server-created metadata and reuses the retained collection. Missing or changed +profile/collection fails closed. It must not rebuild, restart Docker or substitute +an empty collection. Versioned checkpoints additionally retain their local collection +lease marker; collection leases and checkpoint output must share a filesystem. +Restore is same-node only. Per-tenant Registry credential issuance, cross-node collection reconstruction +and automatic Registry artifact GC are not provided by this profile. + +## Packaging and compatibility + +Build the all-in-one image with the repository's maintained build helper. The +Sandboxd gitlink and runtime manifests select the runtime implementation; the +Agent DX source reference and platform/Execd URL/checksum pairs must identify the +same published source. Verify complete package manifests, architecture and +matching platform/component Execd bytes before packaging. Downloaded optional +AFS/envd payloads do not enable those services. + +For control-only upgrades, retain the deployment's qualified guest-rootfs pin +and complete compatible runsc bundle while snapshots depend on them. Changing +the guest runtime, provider identity or captured profile requires an explicit +compatibility and retention plan. See +[runtime-rootfs pinning](../../deploy/docs/runtime-rootfs-pinning.md). + +Record tested sources, image digests, workload results and cleanup in release +qualification evidence. Keep run-specific measurements, PR status and rollout +history outside this operator guide. diff --git a/builder/managed-docker/compose.py b/builder/managed-docker/compose.py new file mode 100644 index 00000000..2414784c --- /dev/null +++ b/builder/managed-docker/compose.py @@ -0,0 +1,712 @@ +"""Runtime-side Compose/Buildx adapter. Docker owns Dockerfile/Compose semantics. + +The client uploads a context and invokes this tool; image metadata and ROFS +registration remain on the runtime side. This preview uses an operator-issued +session capability and pins its complete collection until session cleanup. +""" + +from __future__ import annotations + +import argparse +import base64 +import hashlib +import json +import os +import re +import subprocess +import sys +import time +import uuid +from pathlib import Path +from urllib.request import ProxyHandler, Request, build_opener + + +def execute(args: list[str], *, capture: bool = False) -> str: + result = subprocess.run( + args, check=True, text=True, stdout=subprocess.PIPE if capture else None + ) + return result.stdout if capture else "" + + +def execute_bake(args: list[str], plan: dict) -> None: + """Retry a completed Nydus variant export failure using the same native plan. + + Buildx progress is streamed while retaining only a bounded diagnostic tail. + Native no-cache targets have a single attempt. Transport/process exceptions + propagate with their original identity rather than becoming another solve. + """ + if args[:3] != ["docker", "buildx", "bake"]: + raise ValueError("Expected the native Buildx Bake command") + attempts = 1 if any(t.get("no-cache") for t in plan["target"].values()) else 3 + for attempt in range(attempts): + tail = b"" + with subprocess.Popen(args, stderr=subprocess.PIPE) as process: + assert process.stderr is not None + while chunk := process.stderr.read(4096): + sys.stderr.buffer.write(chunk) + sys.stderr.buffer.flush() + tail = (tail + chunk)[-65536:] + code = process.wait() + if code == 0: + return + error = subprocess.CalledProcessError(code, args, stderr=tail) + terminal = [ + line + for line in tail.decode(errors="replace").splitlines() + if line.startswith("ERROR:") + ] + known_export_failure = ( + code == 1 + and terminal + and any( + message in terminal[-1] + for message in ( + 'failed to get compression blob "nydus": not found', + 'merge nydus layers: get compression blob "nydus": not found', + ) + ) + ) + if not known_export_failure or attempt + 1 == attempts: + raise error + print( + f"AKernel: native Nydus export exited {code}; retry {attempt + 2}/{attempts} " + "with the same Bake plan and native cache (first failure retained above).", + file=sys.stderr, + flush=True, + ) + time.sleep(1) + + +def bake_metadata(args: list[str], plan: dict, path: Path, profile: dict) -> dict: + """Reconcile narrowly identified result errors against published outputs. + + A native build is never replayed here. Each invocation publishes unpredictable + tags; Registry inspection must verify every active target before adopting its + immutable digest. Ordinary solve, authorization and process errors propagate. + """ + os.environ.setdefault("BUILDX_METADATA_PROVENANCE", "disabled") + path.unlink(missing_ok=True) + receipt_path = path.with_name(path.name + ".publication.json") + receipt_path.unlink(missing_ok=True) + try: + execute_bake(args, plan) + except subprocess.CalledProcessError as error: + tail = error.stderr or b"" + if isinstance(tail, bytes): + tail = tail.decode(errors="replace") + terminal = [line for line in tail.splitlines() if line.startswith("ERROR:")] + history_failure = bool(terminal) and re.fullmatch( + r"ERROR: .*unknown blob sha256:[0-9a-f]{64} in history", terminal[-1] + ) + publication_deadline = bool(terminal) and terminal[-1] == ( + "ERROR: failed to solve: Internal: context deadline exceeded" + ) + if error.returncode != 100 or not (history_failure or publication_deadline): + raise + try: + outputs = {} + tags = set() + prefix = profile["registry"] + "/" + profile["prefix"].rstrip("/") + "/" + names = plan["group"]["default"]["targets"] + if not names: + raise ValueError("No native outputs to reconcile") + for name in names: + tagged = plan["target"][name]["tags"] + if len(tagged) != 1 or not tagged[0].startswith(prefix): + raise ValueError("Output is not in this operation's repository") + tag = tagged[0].rsplit(":", 1)[-1] + if not re.fullmatch(r"build-[0-9a-f]{32}", tag): + raise ValueError("Output lacks a fresh publication tag") + tags.add(tag) + outputs[name] = tagged[0] + if len(tags) != 1: + raise ValueError("Outputs belong to different build invocations") + acknowledged = {} + if publication_deadline: + for name, tagged in outputs.items(): + if plan["target"][name].get("attest"): + raise ValueError( + "Explicit attestations require native completion" + ) + digests = set( + re.findall( + r"(?m)^#\d+ pushing manifest for " + + re.escape(tagged) + + r"@(sha256:[0-9a-f]{64})[^\r\n]* done\r?$", + tail, + ) + ) + if len(digests) != 1: + raise ValueError( + "No complete native manifest push acknowledgement" + ) + acknowledged[name] = digests.pop() + recovered = {} + for name, tagged in outputs.items(): + inspection = json.loads( + execute( + [ + "docker", + "manifest", + "inspect", + "--verbose", + *( + ["--insecure"] + if profile.get("registry_insecure", False) + else [] + ), + tagged, + ], + capture=True, + ) + ) + if not isinstance(inspection, dict): + raise TypeError("Expected one native linux/amd64 image manifest") + descriptor = inspection["Descriptor"] + raw = base64.b64decode(inspection["Raw"], validate=True) + manifest = json.loads(raw) + digest = "sha256:" + hashlib.sha256(raw).hexdigest() + if name in acknowledged and acknowledged[name] != digest: + raise ValueError( + "Registry digest differs from the native push acknowledgement" + ) + if ( + descriptor.get("digest") != digest + or descriptor.get("size") != len(raw) + or descriptor.get("mediaType") + != "application/vnd.oci.image.manifest.v1+json" + or descriptor.get("platform", {}).get("os") != "linux" + or descriptor.get("platform", {}).get("architecture") != "amd64" + ): + raise ValueError( + "Registry manifest digest, size or platform mismatch" + ) + if ( + not isinstance(manifest, dict) + or manifest.get("schemaVersion") != 2 + or manifest.get("mediaType") != descriptor["mediaType"] + or not isinstance(manifest.get("config"), dict) + or not isinstance(manifest.get("layers"), list) + ): + raise ValueError( + "Registry did not return a native OCI image manifest" + ) + recovered[name] = { + "containerimage.digest": digest, + "akernel.publication-reconciled": True, + } + receipt_path.write_text( + json.dumps( + { + "native_exit_code": error.returncode, + "native_error": terminal[-1], + **({"history_error": terminal[-1]} if history_failure else {}), + "failure_kind": "history-result" + if history_failure + else "acknowledged-publication-deadline", + "outputs": { + name: {"tagged": outputs[name], **data} + for name, data in recovered.items() + }, + "verification": "Fresh invocation tags; official Docker Registry inspection; raw manifest SHA256, size and platform verified", + "native_build_replayed": False, + }, + indent=2, + ) + ) + path.write_text(json.dumps(recovered, indent=2)) + except Exception as verification: + try: + path.unlink(missing_ok=True) + except OSError: + pass + error.add_note( + "Published output verification failed: " + type(verification).__name__ + ) + raise error from verification + print( + "AKernel: native image publication verified after a native result error; " + "using committed immutable artifacts without replaying the build.", + file=sys.stderr, + flush=True, + ) + return recovered + return json.loads(path.read_text()) + + +def compose_command(descriptor: dict) -> list[str]: + command = [ + "docker", + "compose", + "-p", + descriptor["project_name"], + "--project-directory", + descriptor["directory"], + ] + for filename in descriptor["files"]: + command.extend(["-f", filename]) + return command + + +def describe(descriptor: dict) -> list[str]: + """Resolve service identities natively, without exposing interpolated secrets.""" + os.environ.update(descriptor.get("environment", {})) + config = json.loads( + execute( + [*compose_command(descriptor), "config", "--format", "json"], capture=True + ) + ) + services = config.get("services", {}) + if not services: + raise ValueError("Compose project requires at least one active service") + return list(services) + + +def build_plan( + descriptor: dict, + config: dict, + bake: dict, + registry: str, + prefix: str, + no_cache: bool = False, + registry_insecure: bool = False, + publication_tag: str = "built", +) -> tuple[dict, dict[str, str], dict[str, str]]: + """Adapt native Bake targets' output only; do not interpret Dockerfiles.""" + if not re.fullmatch(r"[a-z0-9][a-z0-9/_-]*", prefix) or ".." in prefix: + raise ValueError("Invalid session repository prefix") + if not re.fullmatch(r"built|build-[0-9a-f]{32}", publication_tag): + raise ValueError("Invalid native publication tag") + targets = bake.setdefault("target", {}) + native_services = {} + for target_name, target in targets.items(): + service_name = target.get("labels", {}).get("com.docker.compose.service") + if service_name is not None: + if service_name in native_services: + raise ValueError( + "Ambiguous native Bake targets for service: " + service_name + ) + native_services[service_name] = target_name + references = {} + service_targets = {} + for index, (name, service) in enumerate(config["services"].items()): + if not re.fullmatch(r"[a-zA-Z0-9][a-zA-Z0-9_.-]*", name): + raise ValueError("Unsupported Compose service name") + if service.get("build") is not None: + target_name = native_services.get(name) + # Exact names also support explicitly constructed single-image + # plans. Never guess Compose's name normalization or disregard + # a conflicting native service identity. + if ( + target_name is None + and name in targets + and "com.docker.compose.service" not in targets[name].get("labels", {}) + ): + target_name = name + if target_name is None: + raise ValueError("No native Bake target for build service: " + name) + target = targets[target_name] + else: + # Image-only inputs use the official Dockerfile frontend for format + # conversion, retaining OCI config. No Python layer reconstruction. + image = service.get("image") + if ( + not isinstance(image, str) + or not image + or any(c.isspace() for c in image) + ): + raise ValueError("Service requires an image or native build target") + target = {"dockerfile-inline": "FROM " + image + "\n"} + target_name = "akernel_image_" + str(index) + while target_name in targets: + target_name += "_" + targets[target_name] = target + service_targets[name] = target_name + # Publication names are private invocation-local identities, not + # lowercased service names. An ordinal is injective even for Foo/foo or + # dotted/underscored names, and bounds repository-component length. + reference = ( + registry + + "/" + + prefix.rstrip("/") + + "/service-" + + str(index) + + ":" + + publication_tag + ) + references[name] = reference + target.update( + { + "tags": [reference], + "platforms": ["linux/amd64"], + "attest": [], + "output": [ + "type=image,push=true,compression=nydus,force-compression=true," + "oci-mediatypes=true,fs-version=5" + + (",registry.insecure=true" if registry_insecure else "") + ], + } + ) + if no_cache: + target["no-cache"] = True + bake["group"] = {"default": {"targets": list(service_targets.values())}} + return bake, references, service_targets + + +def build(descriptor: dict, profile: dict, *, no_cache: bool = False) -> dict: + os.environ.setdefault("BUILDX_NO_DEFAULT_ATTESTATIONS", "1") + os.environ.update(descriptor.get("environment", {})) + command = compose_command(descriptor) + config = json.loads(execute([*command, "config", "--format", "json"], capture=True)) + printed = execute([*command, "build", "--print"], capture=True) + if printed.strip(): + bake = json.loads(printed) + elif any( + service.get("build") is not None for service in config["services"].values() + ): + raise ValueError("Compose returned an empty Bake plan for build services") + else: + bake = {} + bake, targets, service_targets = build_plan( + descriptor, + config, + bake, + profile["registry"], + profile["prefix"], + no_cache, + profile.get("registry_insecure", False), + "build-" + uuid.uuid4().hex, + ) + root = Path(descriptor["directory"] + ".build") + root.mkdir(mode=0o700, exist_ok=True) + plan_path = root / "bake.json" + plan_path.write_text(json.dumps(bake)) + (root / "outputs.json").write_text( + json.dumps( + { + name: {"target": service_targets[name], "reference": tagged} + for name, tagged in targets.items() + }, + indent=2, + ) + ) + metadata_path = root / "metadata.json" + metadata = bake_metadata( + [ + "docker", + "buildx", + "bake", + "--builder", + profile["builder"], + "--file", + str(plan_path), + "--metadata-file", + str(metadata_path), + "--progress", + "plain", + "default", + ], + bake, + metadata_path, + profile, + ) + references = {} + for name, tagged in targets.items(): + digest = metadata[service_targets[name]]["containerimage.digest"] + if not re.fullmatch(r"sha256:[0-9a-f]{64}", digest): + raise ValueError("BuildKit did not return an immutable image digest") + references[name] = tagged.rsplit(":", 1)[0] + "@" + digest + (root / "artifacts.json").write_text(json.dumps(references, indent=2)) + return references + + +def prepare(descriptor: dict, profile: dict) -> dict: + root = Path(descriptor["directory"] + ".build") + references = json.loads((root / "artifacts.json").read_text()) + token = Path(profile["token_file"]).read_text().strip() + opener = build_opener(ProxyHandler({})) + prepared = [] + for reference in dict.fromkeys(references.values()): + request = Request( + profile["provider"] + "/prepare", + data=json.dumps({"reference": reference}).encode(), + headers={ + "Authorization": "Bearer " + token, + "Content-Type": "application/json", + }, + ) + with opener.open(request, timeout=180) as response: + prepared.append(json.loads(response.read(16 * 1024))) + execute( + [ + "/opt/akernel-image-bridge", + "--mode", + "register", + "--plan", + profile["plan"], + "--containerd", + "/run/akernel-image-bridge/containerd.sock", + ] + ) + # Generated override; never mutate the supplied Compose or Dockerfiles. + lines = ["services:"] + for service, reference in references.items(): + lines.extend( + [ + " " + json.dumps(service) + ":", + " build: !reset null", + " image: " + json.dumps(reference), + " pull_policy: never", + ] + ) + overlay = str(root / "runtime-images.yaml") + Path(overlay).write_text("\n".join(lines) + "\n") + attached = dict(descriptor) + attached["files"] = [*descriptor["files"], overlay] + attached["managed_artifacts"] = references + (root / "attached.json").write_text(json.dumps(attached, indent=2)) + return {"artifacts": references, "prepared": prepared, "descriptor": attached} + + +def image_request(profile: dict, reference: str, operation: str) -> dict: + token = Path(profile["token_file"]).read_text().strip() + request = Request( + profile["provider"] + operation, + data=json.dumps({"reference": reference}).encode(), + headers={ + "Authorization": "Bearer " + token, + "Content-Type": "application/json", + }, + ) + with build_opener(ProxyHandler({})).open(request, timeout=180) as response: + return json.loads(response.read(1024 * 1024)) + + +def prepare_image(profile: dict, reference: str) -> dict: + return image_request(profile, reference, "/prepare") + + +def describe_image(profile: dict, reference: str) -> dict: + return image_request(profile, reference, "/describe") + + +def start_builder(profile: dict) -> None: + execute( + [ + "docker", + "buildx", + "create", + "--name", + profile["builder"], + "--driver", + "remote", + profile["endpoint"], + "--use", + ] + ) + + +def image_build(context: str, profile: dict, *, no_cache: bool = False) -> dict: + os.environ.setdefault("BUILDX_NO_DEFAULT_ATTESTATIONS", "1") + start_builder(profile) + metadata = context + ".metadata.json" + plan, targets, service_targets = build_plan( + {}, + {"services": {"single": {"build": {}}}}, + {"target": {"single": {"context": context}}}, + profile["registry"], + profile["prefix"], + no_cache, + profile.get("registry_insecure", False), + "build-" + uuid.uuid4().hex, + ) + plan_path = context + ".bake.json" + Path(plan_path).write_text(json.dumps(plan)) + values = bake_metadata( + [ + "docker", + "buildx", + "bake", + "--builder", + profile["builder"], + "--file", + plan_path, + "--progress", + "plain", + "--metadata-file", + metadata, + "default", + ], + plan, + Path(metadata), + profile, + ) + exported = service_targets["single"] + digest = values[exported]["containerimage.digest"] + if not re.fullmatch(r"sha256:[0-9a-f]{64}", digest): + raise ValueError("BuildKit did not return an immutable artifact") + return describe_image(profile, targets["single"].rsplit(":", 1)[0] + "@" + digest) + + +def expose_service(container: str, port: int) -> dict: + import sys + + sys.path.insert(0, "/usr/lib/akernel") + from docker_network_guest import expose + + data = json.loads(execute(["docker", "inspect", container], capture=True))[0] + seen = set() + for _ in range(8): + mode = data["HostConfig"]["NetworkMode"] + if not mode.startswith("container:"): + break + owner = mode.removeprefix("container:") + if not owner or owner in seen: + raise ValueError("Invalid network namespace ownership") + seen.add(owner) + data = json.loads(execute(["docker", "inspect", owner], capture=True))[0] + else: + raise ValueError("Network namespace ownership exceeds limit") + if data["HostConfig"]["NetworkMode"] == "host": + return {"url": f"http://127.0.0.1:{port}"} + connected = [] + for name, network in (data["NetworkSettings"].get("Networks") or {}).items(): + if not network.get("IPAddress"): + continue + network = dict(network) + if not network.get("Gateway"): + inspected = json.loads( + execute(["docker", "network", "inspect", name], capture=True) + )[0] + import ipaddress + + address = ipaddress.ip_address(network["IPAddress"]) + gateways = [ + v.get("Gateway") + for v in inspected.get("IPAM", {}).get("Config", []) + if v.get("Subnet") + and address in ipaddress.ip_network(v["Subnet"], strict=False) + ] + network["Gateway"] = next((v for v in gateways if v), None) + if network.get("Gateway"): + connected.append(network) + if not connected: + raise ValueError("Managed service requires an IPv4 bridge") + import uuid + + value = connected[0] + # The runtime helper owns the relay process; return its opaque endpoint. + return expose( + Path("/opt/akernel-docker-runtime/tunnels") / uuid.uuid4().hex, + { + "address": value["Gateway"], + "allowed": [value["IPAddress"]], + "target_port": port, + }, + ) + + +def start_runtime(profile: dict) -> None: + import sys + + sys.path.insert(0, "/usr/lib/akernel") + from docker_network_guest import configure + from image_bridge_guest import start_image_bridge + + if Path("/var/run/docker.sock").exists(): + raise RuntimeError("Runtime startup is fresh-only") + runtime = Path("/opt/akernel-docker-runtime") + runtime.mkdir(exist_ok=True) + flags = start_image_bridge( + runtime, + Path(profile["plan"]), + [], + writable_tmpfs_mb=profile["writable_mb"], + writable_backing="file", + ) + configure() + env = dict(os.environ) + env["PATH"] = str(runtime / "network-bin") + ":" + env["PATH"] + with Path("/var/log/akernel-dockerd.log").open("a") as logs: + process = subprocess.Popen( + [ + "dockerd", + *flags, + "--iptables=false", + "--ip6tables=false", + "--userland-proxy=true", + ], + stdout=logs, + stderr=logs, + env=env, + start_new_session=True, + ) + for _ in range(40): + if process.poll() is not None: + raise RuntimeError("Docker exited; inspect /var/log/akernel-dockerd.log") + if ( + subprocess.run( + ["docker", "info"], capture_output=True, check=False + ).returncode + == 0 + ): + break + time.sleep(0.5) + else: + raise RuntimeError("Docker readiness timed out") + start_builder(profile) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "operation", + choices=[ + "start", + "build", + "prepare", + "describe", + "image-build", + "reconnect", + "expose", + ], + ) + parser.add_argument("--profile", default="/run/akernel-build/profile.json") + parser.add_argument("--descriptor") + parser.add_argument("--no-cache", action="store_true") + parser.add_argument("--context") + parser.add_argument("--result") + parser.add_argument("--container") + parser.add_argument("--port", type=int) + args = parser.parse_args() + profile = json.loads(Path(args.profile).read_text()) + if args.operation == "start": + start_runtime(profile) + elif args.operation == "image-build": + if not args.context or not args.result: + parser.error("image-build requires context and result") + Path(args.result).write_text( + json.dumps(image_build(args.context, profile, no_cache=args.no_cache)) + ) + elif args.operation == "reconnect": + import sys + + sys.path.insert(0, "/usr/lib/akernel") + from docker_network_guest import configure + + configure(reconnect=True) + elif args.operation == "expose": + if not args.container or not args.port or not 1 <= args.port <= 65535: + parser.error("expose requires container and valid port") + result = expose_service(args.container, args.port) + if result: + print(json.dumps(result)) + else: + descriptor = json.loads(Path(args.descriptor).read_text()) + if args.operation == "describe": + result = describe(descriptor) + elif args.operation == "build": + result = build(descriptor, profile, no_cache=args.no_cache) + else: + result = prepare(descriptor, profile) + print(json.dumps(result)) diff --git a/builder/managed-docker/control-capacity.example.yaml b/builder/managed-docker/control-capacity.example.yaml new file mode 100644 index 00000000..b750c7e3 --- /dev/null +++ b/builder/managed-docker/control-capacity.example.yaml @@ -0,0 +1,12 @@ +# Operator overlay for the core chart, not a Harbor job configuration. +# Explicit budget for the large-catalog preview; this is not a concurrency SLA. +# For the umbrella chart, nest this mapping under `core`. +adx: + coordinator: + resources: + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "4" + memory: 6Gi diff --git a/builder/managed-docker/image_provider.py b/builder/managed-docker/image_provider.py new file mode 100644 index 00000000..75bf3f28 --- /dev/null +++ b/builder/managed-docker/image_provider.py @@ -0,0 +1,379 @@ +"""Node-side image preparation for a single owned managed Docker session. + +The existing sandboxd image manager owns distribution and the shared ROFS. +This adapter exports its read-only roots through a runtime-owned collection. +The SDK sees artifact references and command results, never layer metadata. +This source-preview service is not a replacement for platform authorization. +""" + +from __future__ import annotations + +import base64 +import hashlib +import hmac +import http.client +import json +import re +import socket +import subprocess +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + +DIGEST = re.compile(r"sha256:[0-9a-f]{64}\Z") +REFERENCE = re.compile( + r"(?P[^/]+)/(?P[a-z0-9][a-z0-9_./-]*)@(?Psha256:[0-9a-f]{64})\Z" +) + + +def digest(payload: bytes) -> str: + return "sha256:" + hashlib.sha256(payload).hexdigest() + + +def chain_id(values: list[str]) -> str: + if not values or any(not DIGEST.fullmatch(v) for v in values): + raise ValueError("Image requires valid rootfs diff IDs") + result = values[0] + for value in values[1:]: + result = digest((result + " " + value).encode()) + return result + + +def atomic_json(path: Path, value: object) -> None: + temporary = path.with_name(path.name + ".new") + temporary.write_text(json.dumps(value, separators=(",", ":"))) + temporary.chmod(0o444) + temporary.replace(path) + + +class UnixConnection(http.client.HTTPConnection): + def __init__(self, path: str): + super().__init__("localhost", timeout=180) + self.path = path + + def connect(self) -> None: + self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self.sock.settimeout(self.timeout) + self.sock.connect(self.path) + + +class ImageProvider: + def __init__( + self, + root: Path, + registry: str, + *, + image_socket: str = "/run/sandboxd/image-api/image.sock", + repository_prefix: str = "poc/", + scheme: str = "https", + owner: str | None = None, + ): + if scheme not in {"https", "http"}: + raise ValueError("Registry scheme must be http or https") + self.root = root + self.registry = registry + self.scheme = scheme + self.image_socket = image_socket + self.repository_prefix = repository_prefix + self.owner = owner + self.lock = threading.Lock() + self.metadata = root / ".metadata" + self.metadata.mkdir(mode=0o755, parents=True, exist_ok=True) + self.plan = self.metadata / "plan.json" + if not self.plan.exists(): + atomic_json(self.plan, {"images": [], "automatic": True}) + + def validate_reference(self, reference: str) -> tuple[str, str]: + match = REFERENCE.fullmatch(reference) + if not match or match["host"] != self.registry: + raise ValueError( + "Expected an immutable artifact in the authorized Registry" + ) + repository = match["repo"] + if not repository.startswith(self.repository_prefix) or any( + p in {"", ".", ".."} for p in repository.split("/") + ): + raise ValueError("Artifact is outside the session repository prefix") + return repository, match["digest"] + + def fetch( + self, repository: str, kind: str, selected: str, descriptor: dict | None = None + ) -> tuple[dict, dict]: + if kind not in {"manifests", "blobs"} or not DIGEST.fullmatch(selected): + raise ValueError("Expected a digest-addressed metadata object") + # Authentication belongs to Sandboxd's Registry client, including native + # Bearer challenges, scoped token reuse and refresh. Never parse Registry + # credentials or implement a second authentication protocol in this broker. + connection = UnixConnection(self.image_socket) + try: + connection.request( + "POST", + "/registry_metadata", + body=json.dumps( + { + "reference": f"{self.registry}/{repository}@{selected}", + "kind": kind, + "insecure": self.scheme == "http", + } + ), + headers={"Content-Type": "application/json"}, + ) + response = connection.getresponse() + if response.status != 200: + raise RuntimeError( + f"Registry metadata preparation failed: HTTP {response.status}" + ) + result = json.loads(response.read(24 * 1024**2)) + payload = base64.b64decode(result["data"], validate=True) + media_type = result["media_type"] + finally: + connection.close() + if len(payload) > 16 * 1024**2 or digest(payload) != selected: + raise ValueError("Metadata size or digest mismatch") + if descriptor and ( + descriptor["size"] != len(payload) or descriptor["digest"] != selected + ): + raise ValueError("Metadata descriptor mismatch") + obj = json.loads(payload) + desc = descriptor or { + "mediaType": obj.get("mediaType", media_type), + "digest": selected, + "size": len(payload), + } + name = selected.replace(":", "-") + ".json" + p = self.metadata / name + if p.exists() and p.read_bytes() != payload: + raise ValueError("Immutable metadata was replaced") + if not p.exists(): + p.write_bytes(payload) + p.chmod(0o444) + return obj, {"descriptor": desc, "file": name} + + def describe(self, reference: str) -> dict: + repository, selected = self.validate_reference(reference) + manifest, blob = self.fetch(repository, "manifests", selected) + target, blobs = blob["descriptor"], [blob] + for _ in range(4): + if "manifests" not in manifest: + break + matches = [ + d + for d in manifest["manifests"] + if d.get("platform", {}).get("os") == "linux" + and d.get("platform", {}).get("architecture") == "amd64" + ] + if len(matches) != 1: + raise ValueError("Expected exactly one linux/amd64 image") + descriptor = matches[0] + manifest, blob = self.fetch( + repository, "manifests", descriptor["digest"], descriptor + ) + blobs.append(blob) + if "manifests" in manifest: + raise ValueError("Image index nesting exceeds the limit") + if not any( + "nydus" in json.dumps(d).lower() for d in manifest.get("layers", []) + ): + raise ValueError("Managed artifact must contain Nydus layers") + config_descriptor = manifest["config"] + config, blob = self.fetch( + repository, "blobs", config_descriptor["digest"], config_descriptor + ) + blobs.append(blob) + if config.get("os") != "linux" or config.get("architecture") != "amd64": + raise ValueError("Image config platform mismatch") + return { + "reference": reference, + "root": "/opt/akernel-images/" + selected[7:], + "chain_id": chain_id(config["rootfs"]["diff_ids"]), + "target": target, + "config": blob["descriptor"], + "blobs": blobs, + } + + def mount_image(self, reference: str) -> str: + connection = UnixConnection(self.image_socket) + try: + connection.request( + "POST", + "/oci_mount", + body=json.dumps( + { + "image_url": reference, + **({"owner": self.owner} if self.owner else {}), + } + ), + headers={"Content-Type": "application/json"}, + ) + response = connection.getresponse() + payload = response.read(1024**2) + if response.status != 200: + raise RuntimeError( + f"sandboxd image preparation failed: HTTP {response.status}: " + + payload.decode()[-2000:] + ) + mounted = json.loads(payload)["mount_path"] + finally: + connection.close() + if not Path(mounted).is_dir(): + raise RuntimeError("Image manager returned an unavailable root") + return mounted + + def validate_mount(self, destination: Path) -> None: + result = subprocess.run( + [ + "findmnt", + "--noheadings", + "--output", + "OPTIONS", + "--mountpoint", + str(destination), + ], + check=True, + capture_output=True, + text=True, + timeout=10, + ) + if "ro" not in result.stdout.strip().split(","): + raise RuntimeError("Registered read-only root is unavailable or writable") + + def handle(self, image: dict, reused: bool) -> dict: + result = { + "reference": image["reference"], + "plan": "/opt/akernel-images/.metadata/plan.json", + "reused": reused, + } + result.update(self.process_defaults(image)) + return result + + def process_defaults(self, image: dict) -> dict: + config = image.get("config", {}).get("digest") + if config: + return { + "image_process": json.loads( + (self.metadata / (config.replace(":", "-") + ".json")).read_text() + ).get("config", {}) + } + return {} + + def describe_artifact(self, reference: str) -> dict: + """Return verified build metadata without acquiring a ROFS mount. + + A single-container build helper does not run the task image. Its + consumer acquires the final rootfs through Sandboxd's ordinary image + lifecycle; registering that root here would retain an unused daemon. + """ + with self.lock: + image = self.describe(reference) + return {"reference": reference, **self.process_defaults(image)} + + def prepare(self, reference: str) -> dict: + self.validate_reference(reference) + with self.lock: + plan = json.loads(self.plan.read_text()) + for image in plan["images"]: + if image["reference"] == reference: + self.validate_mount(self.root / reference.rsplit(":", 1)[1]) + return self.handle(image, True) + image = self.describe(reference) + for existing in plan["images"]: + if existing["root"] == image["root"]: + if any( + existing[key] != image[key] + for key in ["chain_id", "target", "config", "blobs"] + ): + raise ValueError("Immutable root metadata mismatch") + self.validate_mount(self.root / reference.rsplit(":", 1)[1]) + plan["images"].append(image) + atomic_json(self.plan, plan) + return self.handle(image, True) + mounted = self.mount_image(reference) + destination = self.root / reference.rsplit(":", 1)[1] + destination.mkdir(mode=0o755, exist_ok=True) + subprocess.run( + ["mount", "--bind", mounted, str(destination)], check=True, timeout=15 + ) + try: + subprocess.run( + ["mount", "-o", "remount,bind,ro", str(destination)], + check=True, + timeout=15, + ) + plan["images"].append(image) + atomic_json(self.plan, plan) + except BaseException: + subprocess.run(["umount", str(destination)], check=True, timeout=15) + raise + return self.handle(image, False) + + +def serve(provider: ImageProvider, token_file: Path, address: tuple[str, int]) -> None: + token = token_file.read_text().strip() + if len(token) < 32 or token_file.stat().st_mode & 0o077: + raise ValueError( + "Session capability must be private and at least 32 characters" + ) + + class Handler(BaseHTTPRequestHandler): + def do_POST(self) -> None: + if self.path not in {"/prepare", "/describe"}: + self.send_error(404) + return + if not hmac.compare_digest( + self.headers.get("Authorization", ""), "Bearer " + token + ): + self.send_error(401) + return + try: + size = int(self.headers.get("Content-Length", "0")) + if size < 1 or size > 16 * 1024: + raise ValueError("Invalid preparation request length") + body = json.loads(self.rfile.read(size)) + if set(body) != {"reference"} or not isinstance(body["reference"], str): + raise ValueError("Only an immutable artifact reference is accepted") + operation = ( + provider.prepare + if self.path == "/prepare" + else provider.describe_artifact + ) + value = operation(body["reference"]) + except ValueError as error: + self.reply(400, {"error": str(error), "phase": "validate_artifact"}) + return + except Exception as error: # noqa: BLE001 - HTTP boundary retains failure phase + self.reply(500, {"error": str(error), "phase": "prepare_artifact"}) + return + self.reply(200, value) + + def reply(self, status: int, value: object) -> None: + payload = json.dumps(value).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + ThreadingHTTPServer(address, Handler).serve_forever() + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("--root", type=Path, required=True) + parser.add_argument("--registry", required=True) + parser.add_argument("--token-file", type=Path, required=True) + parser.add_argument("--prefix", default="poc/") + parser.add_argument("--scheme", choices=["https", "http"], default="https") + parser.add_argument("--port", type=int, default=8840) + args = parser.parse_args() + serve( + ImageProvider( + args.root, + args.registry, + repository_prefix=args.prefix, + scheme=args.scheme, + ), + args.token_file, + ("0.0.0.0", args.port), + ) diff --git a/builder/managed-docker/profile.example.toml b/builder/managed-docker/profile.example.toml new file mode 100644 index 00000000..745553e5 --- /dev/null +++ b/builder/managed-docker/profile.example.toml @@ -0,0 +1,19 @@ +# Operator configuration only. Replace placeholders and explicitly enable the +# private broker service. Do not copy these settings into Harbor jobs. +[plugin.managed_runtime] +root = "/home/akernel/managed-runtime" + +[plugin.managed_runtime.profiles.harbor-v1] +image = "registry.example.internal/managed/tools@sha256:REPLACE_WITH_64_HEX_DIGEST" +bootstrap_root = "/opt/akernel/runtime/akernel-runtime-rootfs.img" +registry = "registry.example.internal" +repository_prefix = "managed" +# Build publisher credentials; the image manager uses its separate pull account. +registry_auth_file = "/home/akernel/sandboxd/config/build_registry_auths.json" +registry_insecure = false +buildkit_endpoint = "tcp://buildkit.example.internal:9094" +provider_url = "http://runtime-node.example.internal:8840" +allowed_tenants = ["example-tenant"] +writable_mb = 4096 +# Total scheduling floor is writable_mb + minimum_root_mb. +minimum_root_mb = 4096 diff --git a/builder/managed-docker/profile_metadata.py b/builder/managed-docker/profile_metadata.py new file mode 100644 index 00000000..2c010456 --- /dev/null +++ b/builder/managed-docker/profile_metadata.py @@ -0,0 +1,38 @@ +"""Operator-only resource metadata derived from Sandboxd's profile geometry. + +Merge the output into the ADXlet service's config.labels before starting the +worker. No credentials, Registry addresses or node paths enter scheduling. +""" + +import argparse +import json +import re +import tomllib +from pathlib import Path + +LABEL_PREFIX = "AKERNEL_RUNTIME_PROFILE_" +MAX_STORAGE_MB = (1 << 44) - 1 + + +def resource_labels(config: dict) -> dict[str, str]: + profiles = config.get("plugin", {}).get("managed_runtime", {}).get("profiles", {}) + labels = {} + for name, profile in profiles.items(): + if re.fullmatch(r"[a-z][a-z0-9-]{0,62}", name) is None: + raise ValueError("Invalid managed runtime profile name") + upper, root = profile.get("writable_mb"), profile.get("minimum_root_mb", 0) + if isinstance(root, int) and not isinstance(root, bool) and root == 0: + root = 4096 + if any(isinstance(v, bool) or not isinstance(v, int) or v < 1 for v in (upper, root)): + raise ValueError("Invalid managed runtime storage geometry") + if upper + root > MAX_STORAGE_MB: + raise ValueError("Managed runtime storage geometry overflows bytes") + labels[LABEL_PREFIX + name] = str(upper + root) + return labels + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("sandboxd_config", type=Path) + args = parser.parse_args() + print(json.dumps(resource_labels(tomllib.loads(args.sandboxd_config.read_text())))) diff --git a/builder/managed-docker/provider_service.py b/builder/managed-docker/provider_service.py new file mode 100644 index 00000000..1e89ed16 --- /dev/null +++ b/builder/managed-docker/provider_service.py @@ -0,0 +1,179 @@ +"""Node runtime broker for Sandboxd-authorized per-environment collections.""" + +from __future__ import annotations + +import hashlib +import fcntl +import hmac +import json +import logging +import os +import stat +import time +import re +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from contextlib import contextmanager +from weakref import WeakValueDictionary + +from image_provider import ImageProvider + +logger = logging.getLogger(__name__) + + +class RuntimeCollections: + def __init__(self, root: Path, image_socket: str): + self.root = root + self.image_socket = image_socket + self.providers: WeakValueDictionary[str, ImageProvider] = WeakValueDictionary() + self.lock = threading.Lock() + + def _record(self, authorization: str) -> dict: + if not authorization.startswith("Bearer "): + raise PermissionError("Runtime capability required") + token = authorization.removeprefix("Bearer ") + collection, _, _secret = token.partition(".") + if not re.fullmatch(r"[a-f0-9]{32}", collection): + raise PermissionError("Invalid runtime capability") + path = self.root / "records" / (collection + ".json") + try: + if path.stat().st_mode & 0o077: + raise PermissionError("Unsafe runtime authorization record") + record = json.loads(path.read_text()) + except FileNotFoundError as error: + raise PermissionError("Runtime capability unavailable") from error + if not hmac.compare_digest( + hashlib.sha256(token.encode()).hexdigest(), record["token_sha256"] + ): + raise PermissionError("Invalid runtime capability") + if record["collection_id"] != collection: + raise PermissionError("Runtime collection identity mismatch") + if record.get("lifecycle_version", 0) not in {0, 1}: + raise PermissionError("Unsupported runtime collection lifecycle") + return record + + def authorize(self, authorization: str) -> ImageProvider: + record = self._record(authorization) + collection = record["collection_id"] + with self.lock: + provider = self.providers.get(collection) + if provider is None: + provider = ImageProvider( + self.root / "collections" / collection, + record["registry"], + image_socket=self.image_socket, + repository_prefix=record["repository_prefix"], + scheme=record["scheme"], + owner=collection, + ) + self.providers[collection] = provider + return provider + + @contextmanager + def _lease(self, record: dict): + if record.get("lifecycle_version", 0) == 0: + # Old preview collections stay pinned until an operator drain. + yield + return + identity = record["collection_id"] + path = self.root / "leases" / identity + try: + fd = os.open(path, os.O_RDONLY | os.O_CLOEXEC | os.O_NOFOLLOW) + except FileNotFoundError as error: + raise PermissionError("Runtime collection unavailable") from error + try: + deadline = time.monotonic() + 180 + while True: + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + break + except BlockingIOError: + if time.monotonic() >= deadline: + raise TimeoutError( + "Runtime collection operation wait timed out" + ) + time.sleep(0.01) + try: + current = path.lstat() + except FileNotFoundError as error: + raise PermissionError("Runtime collection unavailable") from error + opened = os.fstat(fd) + if ( + not stat.S_ISREG(opened.st_mode) + or (current.st_dev, current.st_ino) != (opened.st_dev, opened.st_ino) + or opened.st_nlink <= 1 + or (self.root / "retired" / identity).exists() + ): + raise PermissionError("Runtime collection retired or unowned") + yield + finally: + os.close(fd) + + def perform(self, authorization: str, operation: str, reference: str) -> dict: + record = self._record(authorization) + with self._lease(record): + # Revalidate after waiting: retirement may have revoked the record. + provider = self.authorize(authorization) + function = ( + provider.prepare + if operation == "/prepare" + else provider.describe_artifact + ) + return function(reference) + + +def serve(collections: RuntimeCollections, port: int) -> None: + class Handler(BaseHTTPRequestHandler): + def do_POST(self): + if self.path not in {"/prepare", "/describe"}: + self.send_error(404) + return + try: + authorization = self.headers.get("Authorization", "") + collections._record(authorization) + except PermissionError: + self.send_error(401) + return + try: + self.connection.settimeout(180) + size = int(self.headers.get("Content-Length", "0")) + if not 0 < size <= 16 * 1024: + raise ValueError("Invalid request size") + body = json.loads(self.rfile.read(size)) + if set(body) != {"reference"} or not isinstance(body["reference"], str): + raise ValueError("Immutable artifact reference required") + result = collections.perform( + authorization, self.path, body["reference"] + ) + payload = json.dumps(result).encode() + except PermissionError: + self.send_error(401, "Runtime collection unavailable") + return + except ValueError as error: + self.send_error(400, str(error)) + return + except Exception: + logger.exception( + "Managed image preparation failed; inspect the node runtime" + ) + self.send_error(500, "Runtime image preparation failed") + return + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + ThreadingHTTPServer(("0.0.0.0", port), Handler).serve_forever() + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument("--root", type=Path, required=True) + parser.add_argument("--image-socket", default="/run/sandboxd/image-api/image.sock") + parser.add_argument("--port", type=int, default=8840) + args = parser.parse_args() + serve(RuntimeCollections(args.root, args.image_socket), args.port) diff --git a/builder/managed-docker/qualification-sources.json b/builder/managed-docker/qualification-sources.json new file mode 100644 index 00000000..437ae9be --- /dev/null +++ b/builder/managed-docker/qualification-sources.json @@ -0,0 +1,63 @@ +{ + "schema_version": 2, + "stage": "source-preview", + "scope": "Review/build provenance only; deployment versions and load results require separate runtime receipts.", + "upstream_baselines": { + "akernel": { + "branch": "develop", + "commit": "724238ccf2d1b582824100f312f2031250daa3dc" + }, + "sandboxd": { + "branch": "main", + "commit": "d23687d62ef0afd643b4028aedbcbc0e8bd346dc" + }, + "agent_dx": { + "branch": "master", + "commit": "2f61d44d9c56ae029a06698c337cbdc4aeaaa7af" + } + }, + "candidate_gitlinks": { + "sandboxd": "d23687d62ef0afd643b4028aedbcbc0e8bd346dc", + "agent_dx": "2f61d44d9c56ae029a06698c337cbdc4aeaaa7af" + }, + "pinned_runtime_artifacts": { + "gvisor": { + "source": "49590af731d7893e6c7cb33632068e5d4e4cf7fe", + "release": "release-20261012.0-akernel.4" + }, + "distill_fs": { + "source": "90227ac3a2f9fb919cfabc3e346b560ecae193e7", + "release": "v0.1.3" + } + }, + "native_adx_sdk": { + "package": "adx-environment", + "version": "0.1.0rc4", + "publication_commit": "6a4c6d872edd6596361da3cb0e27ca55c3f5e1b0", + "package_source_commit": "e611b9e4a83dcd6da5346bd17c2f6af28e244952", + "note": "Dependency resolution uses the published package pinned in sdk/python/pyproject.toml. Harbor client overrides and existing cluster qualification apply only to their separately recorded source and image revisions." + }, + "afs_enabled": false, + "envd_enabled": false, + "publication_requirement": "The default amd64 platform/Execd URL and checksum pairs select the same official Agent DX daily build and matching static Execd bytes. Verify architecture and complete manifests before packaging. This managed-runtime integration remains a preview; official dependency publication does not establish completed cluster qualification. Sandboxd is fixed to official main after merged #101 (including R1 checkpoint-lease preflight) and #102; gVisor uses the organization-published complete .4 bundle. The native Python SDK dependency remains index-resolved rc4; separately pinned Harbor source-client qualification is recorded outside this package.", + "agent_dx_platform_artifacts": { + "source_commit": "2f61d44d9c56ae029a06698c337cbdc4aeaaa7af", + "publication": "https://buildkite.com/agent-dx/agent-dx/builds/263", + "platform_url": "https://agent-dx.obs.cn-southwest-2.myhuaweicloud.com/adx/daily/20261106064716-2f61d44d9c56/linux/amd64/adx-release.tar.gz", + "platform_sha256": "e4390134ed9428936a241622dfd01a4420b0ddc494b3dab40bf0b9712238cf33", + "execd_url": "https://agent-dx.obs.cn-southwest-2.myhuaweicloud.com/adx/daily/20261106064716-2f61d44d9c56/linux/amd64/adx-execd.tar.gz", + "execd_sha256": "240bfcc19326faab9ea5ffa7587c9e6e5ca48241a86de74778b9a6c26403bbee", + "status": "Official OBS daily artifacts of merged upstream master. Complete manifests, source, architecture and matching static Execd bytes verified; cluster qualification remains separate." + }, + "integrated_upstream_changes": { + "sandboxd": { + "source_commit": "d23687d62ef0afd643b4028aedbcbc0e8bd346dc", + "merged_pull_requests": [ + "https://github.com/inclusionAI/sandboxd/pull/101", + "https://github.com/inclusionAI/sandboxd/pull/102" + ], + "includes_checkpoint_lease_preflight_review_fix": true, + "equivalent_qualified_candidate_tree": "bdc0b1f10c2a269fe59c13230346292f080e1b18" + } + } +} diff --git a/builder/managed-docker/test_compose.py b/builder/managed-docker/test_compose.py new file mode 100644 index 00000000..1889e3fc --- /dev/null +++ b/builder/managed-docker/test_compose.py @@ -0,0 +1,228 @@ +import importlib.util +import json +import sys +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock, patch + +spec = importlib.util.spec_from_file_location( + "runtime_compose", Path(__file__).with_name("compose.py") +) +compose = importlib.util.module_from_spec(spec) +spec.loader.exec_module(compose) + + +class NativeBuildContract(unittest.TestCase): + def test_single_build_returns_metadata_without_preparing_helper_rootfs(self): + with tempfile.TemporaryDirectory() as directory: + context = str(Path(directory) / "context") + Path(context).mkdir() + profile = { + "registry": "registry.test", + "prefix": "poc/job", + "builder": "remote", + "endpoint": "tcp://builder:9094", + } + identity = "sha256:" + "a" * 64 + artifact = { + "reference": "registry.test/poc/job/service-0@" + identity, + "image_process": {"WorkingDir": "/app"}, + } + + def bake(args, _plan): + metadata = args[args.index("--metadata-file") + 1] + Path(metadata).write_text( + json.dumps({"single": {"containerimage.digest": identity}}) + ) + + with ( + patch.object(compose, "start_builder"), + patch.object(compose, "execute_bake", side_effect=bake), + patch.object( + compose, "describe_image", create=True, return_value=artifact + ) as describe, + patch.object( + compose, + "prepare_image", + side_effect=AssertionError( + "build helper must not pin a task rootfs" + ), + ), + ): + self.assertEqual(compose.image_build(context, profile), artifact) + describe.assert_called_once_with(profile, artifact["reference"]) + + def test_task_image_is_exported_without_python_or_path_mutation(self): + original = { + "context": "/original", + "dockerfile": "Custom.Dockerfile", + "target": "release", + "args": {"MODE": "native"}, + "secret": ["id=build,src=/secret"], + } + plan, refs, _ = compose.build_plan( + {}, + {"services": {"main": {"build": {}}}}, + {"target": {"main": dict(original)}}, + "registry.test", + "poc/job", + ) + self.assertEqual(set(plan["target"]), {"main"}) + self.assertEqual(plan["group"]["default"]["targets"], ["main"]) + actual = plan["target"]["main"] + for key, value in original.items(): + self.assertEqual(actual[key], value) + self.assertEqual(actual["tags"], [refs["main"]]) + self.assertNotIn("dockerfile-inline", actual) + self.assertNotIn("contexts", actual) + + def test_native_target_names_are_not_reserved_by_harness_helpers(self): + native = { + "akernel_harness_python": {"context": "/user-helper"}, + "main": {"context": "/main"}, + } + plan, _, _ = compose.build_plan( + {}, + {"services": {"main": {"build": {}}}}, + {"target": native}, + "registry.test", + "poc/job", + True, + False, + ) + self.assertEqual( + plan["target"]["akernel_harness_python"], {"context": "/user-helper"} + ) + self.assertTrue(plan["target"]["main"]["no-cache"]) + self.assertEqual(set(plan["target"]), set(native)) + + def test_service_discovery_uses_official_compose_and_returns_only_names(self): + descriptor = { + "project_name": "test", + "directory": "/work", + "files": ["/work/base.yaml", "/work/override.yaml"], + } + config = {"services": {"main": {"environment": {"SECRET": "private"}}}} + with patch.object( + compose, "execute", return_value=json.dumps(config) + ) as execute: + self.assertEqual(compose.describe(descriptor), ["main"]) + self.assertEqual( + execute.call_args.args[0], + [ + "docker", + "compose", + "-p", + "test", + "--project-directory", + "/work", + "-f", + "/work/base.yaml", + "-f", + "/work/override.yaml", + "config", + "--format", + "json", + ], + ) + + def test_native_build_definition_and_cache_inputs_are_preserved(self): + native = { + "context": "/work/app", + "dockerfile": "Dockerfile", + "target": "release", + "args": {"MODE": "prod"}, + "secret": ["id=private,src=/run/session"], + "cache-from": ["type=registry,ref=registry.test/cache:step"], + } + plan, refs, _ = compose.build_plan( + {}, + {"services": {"main": {"build": {}}}}, + {"target": {"main": dict(native)}}, + "registry.test", + "poc/job", + ) + actual = plan["target"]["main"] + for key, value in native.items(): + self.assertEqual(actual[key], value) + self.assertNotIn("no-cache", actual) + self.assertNotIn("registry.insecure", actual["output"][0]) + self.assertIn("compression=nydus", actual["output"][0]) + self.assertEqual(refs, {"main": "registry.test/poc/job/service-0:built"}) + + def test_image_only_conversion_uses_native_dockerfile_frontend(self): + image = "registry.test/base@sha256:" + "a" * 64 + plan, _, services = compose.build_plan( + {}, + {"services": {"loadgen": {"image": image}}}, + {}, + "registry.test", + "poc/job", + True, + True, + ) + target = plan["target"][services["loadgen"]] + self.assertEqual(target["dockerfile-inline"], "FROM " + image + "\n") + self.assertTrue(target["no-cache"]) + self.assertIn("registry.insecure=true", target["output"][0]) + + def test_namespace_owner_and_internal_network_use_native_inspection(self): + relay = MagicMock(return_value={"url": "http://172.28.1.1:23456"}) + values = [ + [{"HostConfig": {"NetworkMode": "container:api-id"}}], + [ + { + "HostConfig": {"NetworkMode": "project-private"}, + "NetworkSettings": { + "Networks": { + "private": {"IPAddress": "172.28.1.2", "Gateway": ""} + } + }, + } + ], + [ + { + "IPAM": { + "Config": [{"Subnet": "172.28.1.0/24", "Gateway": "172.28.1.1"}] + } + } + ], + ] + with ( + patch.object( + compose, "execute", side_effect=[json.dumps(v) for v in values] + ) as execute, + patch.dict( + sys.modules, {"docker_network_guest": SimpleNamespace(expose=relay)} + ), + ): + self.assertEqual( + compose.expose_service("main-id", 8766), relay.return_value + ) + self.assertEqual( + execute.call_args_list[1].args[0], ["docker", "inspect", "api-id"] + ) + self.assertEqual( + relay.call_args.args[1], + {"address": "172.28.1.1", "allowed": ["172.28.1.2"], "target_port": 8766}, + ) + + def test_unsafe_repository_and_image_input_fail_before_build(self): + for prefix, image in [ + ("poc/../other", "alpine:3"), + ("poc/job", "ubuntu\nRUN bad"), + ]: + with self.subTest(prefix=prefix), self.assertRaises(ValueError): + compose.build_plan( + {}, + {"services": {"main": {"image": image}}}, + {}, + "registry.test", + prefix, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/builder/managed-docker/test_compose_native.py b/builder/managed-docker/test_compose_native.py new file mode 100644 index 00000000..08441dee --- /dev/null +++ b/builder/managed-docker/test_compose_native.py @@ -0,0 +1,149 @@ +"""Opt-in official Compose/Buildx plan validation without a daemon or builds.""" + +import hashlib +import importlib.util +import json +import os +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +spec = importlib.util.spec_from_file_location( + "runtime_compose_native", Path(__file__).with_name("compose.py") +) +compose = importlib.util.module_from_spec(spec) +spec.loader.exec_module(compose) + + +def digest(value): + return "sha256:" + hashlib.sha256(value.encode()).hexdigest() + + +class NativeComposePlansTest(unittest.TestCase): + def setUp(self): + self.compose = os.environ.get("AKERNEL_TEST_DOCKER_COMPOSE") + self.buildx = os.environ.get("AKERNEL_TEST_BUILDX") + if not self.compose or not self.buildx: + self.skipTest("Set AKERNEL_TEST_DOCKER_COMPOSE and AKERNEL_TEST_BUILDX") + for binary in [self.compose, self.buildx]: + self.assertTrue(Path(binary).is_file() and os.access(binary, os.X_OK)) + + def check_project(self, services): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + project = root / "project" + project.mkdir() + (project / "Dockerfile").write_text("FROM scratch\nCOPY marker /marker\n") + (project / "marker").write_text("Unchanged native build input\n") + filename = project / "compose.json" + filename.write_text(json.dumps({"services": services})) + original = {p.name: p.read_bytes() for p in project.iterdir()} + config = root / "docker-config" + plugins = config / "cli-plugins" + plugins.mkdir(parents=True) + (plugins / "docker-buildx").symlink_to(Path(self.buildx).resolve()) + env = { + k: v for k, v in os.environ.items() if k in ["PATH", "LANG", "LC_ALL"] + } + env.update( + HOME=str(root), + DOCKER_CONFIG=str(config), + DOCKER_HOST="unix:///nonexistent-akernel-native-plan-test.sock", + DOCKER_BUILDKIT="1", + ) + + def execute(args, *, capture=False): + self.assertEqual(args[:2], ["docker", "compose"]) + return subprocess.check_output( + [self.compose, *args[2:]], env=env, text=True, timeout=30 + ) + + def print_plan(_args, plan, path, _profile): + # Docker's parser validates the final native target/group names + # and Nydus exporter settings. No Solve or push is submitted. + parsed = json.loads( + subprocess.check_output( + [ + self.buildx, + "bake", + "--file", + str(path.with_name("bake.json")), + "--print", + "default", + ], + env=env, + text=True, + timeout=30, + ) + ) + active = parsed["group"]["default"]["targets"] + self.assertEqual(set(active), set(plan["group"]["default"]["targets"])) + result = {} + for name in active: + target = parsed["target"][name] + self.assertEqual(target["output"][0]["compression"], "nydus") + identity = target.get("labels", {}).get( + "com.docker.compose.service" + ) + result[name] = { + "containerimage.digest": digest( + identity or target["dockerfile-inline"] + ) + } + return result + + descriptor = { + "project_name": "native-plan-test", + "directory": str(project), + "files": [str(filename)], + } + with ( + patch.dict(os.environ, env, clear=True), + patch.object(compose, "execute", side_effect=execute), + patch.object(compose, "bake_metadata", side_effect=print_plan), + ): + references = compose.build( + descriptor, + { + "registry": "registry.test", + "prefix": "tenant/project", + "builder": "remote", + }, + ) + self.assertEqual(set(references), set(services)) + self.assertEqual( + len({ref.split("@")[0] for ref in references.values()}), len(services) + ) + for service, spec in services.items(): + identity = ( + service if "build" in spec else "FROM " + spec["image"] + "\n" + ) + self.assertTrue(references[service].endswith("@" + digest(identity))) + self.assertEqual( + original, {p.name: p.read_bytes() for p in project.iterdir()} + ) + + def test_image_only(self): + self.check_project( + {"api": {"image": "alpine:3"}, "worker": {"image": "ubuntu:24.04"}} + ) + + def test_dotted_build(self): + self.check_project({"foo.bar": {"build": "."}}) + + def test_dotted_build_with_an_image(self): + self.check_project({"foo.bar": {"build": ".", "image": "original/unused:tag"}}) + + def test_case_distinct_builds(self): + self.check_project({"Foo": {"build": "."}, "foo": {"build": "."}}) + + def test_dotted_build_and_underscore_image(self): + self.check_project( + {"foo.bar": {"build": "."}, "foo_bar": {"image": "alpine:3"}} + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/builder/managed-docker/test_compose_outputs.py b/builder/managed-docker/test_compose_outputs.py new file mode 100644 index 00000000..917dee30 --- /dev/null +++ b/builder/managed-docker/test_compose_outputs.py @@ -0,0 +1,192 @@ +"""Keep native Compose service identities through image publication.""" + +import copy +import importlib.util +import json +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +spec = importlib.util.spec_from_file_location( + "runtime_compose_outputs", Path(__file__).with_name("compose.py") +) +compose = importlib.util.module_from_spec(spec) +spec.loader.exec_module(compose) + +FIRST = "sha256:" + "1" * 64 +SECOND = "sha256:" + "2" * 64 +PROFILE = {"registry": "registry.test", "prefix": "tenant/job", "builder": "remote"} + + +def native_target(service, **values): + return { + "context": "/original/" + service, + "dockerfile": "Original.Dockerfile", + "labels": {"com.docker.compose.service": service}, + **values, + } + + +class ComposeOutputsTest(unittest.TestCase): + def run_build(self, services, native, metadata=None): + observed = {} + + def execute(args, *, capture=False): + if args[-3:] == ["config", "--format", "json"]: + return json.dumps({"services": services}) + self.assertEqual(args[-2:], ["build", "--print"]) + return native if isinstance(native, str) else json.dumps(native) + + def solve(_args, plan, _path, _profile): + observed["plan"] = copy.deepcopy(plan) + return metadata or { + name: {"containerimage.digest": FIRST} + for name in plan["group"]["default"]["targets"] + } + + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute", side_effect=execute), + patch.object(compose, "bake_metadata", side_effect=solve) as bake, + ): + directory = str(Path(root) / "project") + descriptor = { + "directory": directory, + "project_name": "original", + "files": [directory + "/compose.yaml"], + } + result = compose.build(descriptor, PROFILE) + observed["bake_calls"] = bake.call_count + observed["artifacts"] = json.loads( + Path(directory + ".build/artifacts.json").read_text() + ) + observed["outputs"] = json.loads( + Path(directory + ".build/outputs.json").read_text() + ) + return result, observed + + def test_image_only_project_accepts_successful_empty_native_plan(self): + for output in ["", " \n"]: + with self.subTest(output=output): + result, observed = self.run_build( + {"sidecar": {"image": "alpine:3"}}, output + ) + self.assertTrue(result["sidecar"].endswith("@" + FIRST)) + plan = observed["plan"] + targets = plan["group"]["default"]["targets"] + self.assertEqual(len(targets), 1) + self.assertEqual( + plan["target"][targets[0]]["dockerfile-inline"], + "FROM alpine:3\n", + ) + self.assertEqual(observed["bake_calls"], 1) + + def test_dotted_build_service_uses_native_identity_with_or_without_image(self): + for image in [None, "original/image:tag"]: + with self.subTest(image=image): + service = {"build": {"context": "."}} + if image: + service["image"] = image + original = native_target("foo.bar", args={"KEEP": "original"}) + result, observed = self.run_build( + {"foo.bar": service}, + {"target": {"foo_bar": original}}, + {"foo_bar": {"containerimage.digest": FIRST}}, + ) + self.assertTrue(result["foo.bar"].endswith("@" + FIRST)) + plan = observed["plan"] + self.assertEqual(plan["group"]["default"]["targets"], ["foo_bar"]) + for key, value in original.items(): + self.assertEqual(plan["target"]["foo_bar"][key], value) + self.assertNotIn("dockerfile-inline", plan["target"]["foo_bar"]) + + def test_case_distinct_services_keep_distinct_repositories_and_digests(self): + result, observed = self.run_build( + {"Foo": {"build": {}}, "foo": {"build": {}}}, + {"target": {name: native_target(name) for name in ["Foo", "foo"]}}, + { + "Foo": {"containerimage.digest": FIRST}, + "foo": {"containerimage.digest": SECOND}, + }, + ) + self.assertTrue(result["Foo"].endswith("@" + FIRST)) + self.assertTrue(result["foo"].endswith("@" + SECOND)) + self.assertNotEqual(result["Foo"].split("@")[0], result["foo"].split("@")[0]) + self.assertEqual(observed["artifacts"], result) + self.assertEqual(observed["outputs"]["Foo"]["target"], "Foo") + self.assertEqual(observed["outputs"]["foo"]["target"], "foo") + published = [t["tags"][0] for t in observed["plan"]["target"].values()] + self.assertEqual(len(set(published)), 2) + + def test_dotted_build_and_underscore_image_service_do_not_share_target(self): + result, observed = self.run_build( + {"foo.bar": {"build": {}}, "foo_bar": {"image": "alpine:3"}}, + {"target": {"foo_bar": native_target("foo.bar")}}, + ) + targets = observed["plan"]["group"]["default"]["targets"] + self.assertEqual(len(set(targets)), 2) + self.assertEqual(set(result), {"foo.bar", "foo_bar"}) + self.assertNotIn("dockerfile-inline", observed["plan"]["target"]["foo_bar"]) + + def test_missing_build_target_never_substitutes_the_service_image(self): + for output in ["", {"target": {}}]: + with self.subTest(output=output), self.assertRaises(ValueError): + self.run_build( + {"main": {"build": {}, "image": "stale/image:tag"}}, output + ) + + def test_ambiguous_native_service_labels_are_rejected(self): + with self.assertRaises(ValueError): + self.run_build( + {"main": {"build": {}}}, + { + "target": { + "first": native_target("main"), + "second": native_target("main"), + } + }, + ) + + def test_native_build_failure_is_not_interpreted_as_empty_plan(self): + error = subprocess.CalledProcessError( + 1, ["docker", "compose", "build", "--print"] + ) + with ( + patch.object( + compose, + "execute", + side_effect=[ + json.dumps({"services": {"main": {"image": "alpine:3"}}}), + error, + ], + ), + patch.object(compose, "bake_metadata") as bake, + self.assertRaises(subprocess.CalledProcessError) as caught, + ): + compose.build( + {"project_name": "x", "directory": "/x", "files": []}, PROFILE + ) + self.assertIs(caught.exception, error) + bake.assert_not_called() + + def test_malformed_nonempty_native_plan_is_not_converted(self): + with self.assertRaises(json.JSONDecodeError): + self.run_build({"main": {"image": "alpine:3"}}, "not a Bake plan") + + def test_native_build_context_dependencies_are_preserved(self): + dependency = native_target("foo.bar") + worker = native_target("worker", contexts={"base": "target:foo_bar"}) + _, observed = self.run_build( + {"foo.bar": {"build": {}}, "worker": {"build": {}}}, + {"target": {"foo_bar": dependency, "worker": worker}}, + ) + self.assertEqual( + observed["plan"]["target"]["worker"]["contexts"], + {"base": "target:foo_bar"}, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/builder/managed-docker/test_export_retry.py b/builder/managed-docker/test_export_retry.py new file mode 100644 index 00000000..bc68f405 --- /dev/null +++ b/builder/managed-docker/test_export_retry.py @@ -0,0 +1,192 @@ +"""Native export retries preserve diagnostics, cache policy and definite outcomes.""" + +import importlib.util +import io +import json +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock, patch + +spec = importlib.util.spec_from_file_location( + "runtime_export_retry", Path(__file__).with_name("compose.py") +) +compose = importlib.util.module_from_spec(spec) +spec.loader.exec_module(compose) + +COMMAND = ["docker", "buildx", "bake", "--file", "/frozen/bake.json", "default"] +MISSING = b'ERROR: failed to solve: failed to get compression blob "nydus": not found\n' +MERGE_MISSING = b'ERROR: failed to solve: merge nydus layers: get compression blob "nydus": not found\n' + + +def process(code, stderr): + result = MagicMock() + result.__enter__.return_value = result + result.stderr = io.BytesIO(stderr) + result.wait.return_value = code + return result + + +class NativeExportRetryTest(unittest.TestCase): + def test_real_process_pipe_streams_large_progress_and_retries_same_arguments(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + executable = root / "docker" + executable.write_text( + f"#!{sys.executable}\n" + "import json,os,pathlib,sys\n" + "p=pathlib.Path(os.environ['AKERNEL_EXPORT_TEST_COUNTER'])\n" + "rows=json.loads(p.read_text())if p.exists()else[]\n" + "rows.append(sys.argv[1:]);p.write_text(json.dumps(rows))\n" + "if len(rows)==1:\n" + " sys.stderr.buffer.write(b'x'*1000000+b'\\n'+" + + repr(MISSING) + + ");sys.stderr.flush();sys.exit(1)\n" + ) + executable.chmod(0o700) + counter = root / "calls.json" + output = io.BytesIO() + text = io.TextIOWrapper(output, encoding="utf-8", write_through=True) + with ( + patch.dict( + os.environ, + { + "PATH": str(root) + os.pathsep + os.environ["PATH"], + "AKERNEL_EXPORT_TEST_COUNTER": str(counter), + }, + ), + patch.object(compose.sys, "stderr", text), + patch.object(compose.time, "sleep"), + ): + compose.execute_bake(COMMAND, {"target": {"main": {}}}) + text.flush() + self.assertEqual( + json.loads(counter.read_text()), [COMMAND[1:], COMMAND[1:]] + ) + self.assertIn(MISSING, output.getvalue()) + self.assertIn(b"retry 2/3", output.getvalue()) + + def run_export(self, processes, targets=None): + output = io.BytesIO() + text = io.TextIOWrapper(output, encoding="utf-8", write_through=True) + with ( + patch.object(compose.subprocess, "Popen", side_effect=processes) as spawn, + patch.object(compose.sys, "stderr", text), + patch.object(compose.time, "sleep") as sleep, + ): + compose.execute_bake(COMMAND, {"target": targets or {"main": {}}}) + text.flush() + return spawn.call_args_list, output.getvalue(), sleep.call_count + + def test_known_completed_export_failure_reuses_exact_command_and_keeps_first_error( + self, + ): + calls, output, sleeps = self.run_export( + [process(1, MISSING), process(0, b"done\n")] + ) + self.assertEqual(len(calls), 2) + self.assertEqual(calls[0], calls[1]) + self.assertEqual(calls[0].args, (COMMAND,)) + self.assertIn(MISSING, output) + self.assertIn(b"retry 2/3", output) + self.assertEqual(sleeps, 1) + + def test_same_variant_failure_during_merge_is_also_retryable(self): + calls, output, sleeps = self.run_export( + [process(1, MERGE_MISSING), process(0, b"done\n")] + ) + self.assertEqual(len(calls), 2) + self.assertEqual(calls[0], calls[1]) + self.assertIn(MERGE_MISSING, output) + self.assertEqual(sleeps, 1) + + def test_persistent_export_failure_is_bounded_and_diagnostic_tail_is_bounded(self): + huge = b"x" * 100000 + b"\n" + MISSING + output = SimpleNamespace( + buffer=io.BytesIO(), write=lambda _: None, flush=lambda: None + ) + with ( + patch.object( + compose.subprocess, + "Popen", + side_effect=[process(1, huge) for _ in range(3)], + ) as spawn, + patch.object(compose.sys, "stderr", output), + patch.object(compose.time, "sleep"), + self.assertRaises(subprocess.CalledProcessError) as caught, + ): + compose.execute_bake(COMMAND, {"target": {"main": {}}}) + self.assertEqual(spawn.call_count, 3) + self.assertLessEqual(len(caught.exception.stderr), 65536) + self.assertIn(MISSING, caught.exception.stderr) + + def test_native_no_cache_on_any_target_disables_replay(self): + with ( + patch.object( + compose.subprocess, "Popen", return_value=process(1, MISSING) + ) as spawn, + patch.object(compose.sys, "stderr", SimpleNamespace(buffer=io.BytesIO())), + self.assertRaises(subprocess.CalledProcessError), + ): + compose.execute_bake( + COMMAND, {"target": {"main": {}, "dependency": {"no-cache": True}}} + ) + self.assertEqual(spawn.call_count, 1) + + def test_build_and_authorization_errors_are_not_retried(self): + for terminal in [ + b"ERROR: failed to solve: RUN did not complete successfully: exit code: 1\n", + b"ERROR: failed to push: 401 Unauthorized\n", + MISSING + b"ERROR: later failure is not the known export error\n", + ]: + with self.subTest(error=terminal): + with ( + patch.object( + compose.subprocess, "Popen", return_value=process(1, terminal) + ) as spawn, + patch.object( + compose.sys, "stderr", SimpleNamespace(buffer=io.BytesIO()) + ), + self.assertRaises(subprocess.CalledProcessError), + ): + compose.execute_bake(COMMAND, {"target": {"main": {}}}) + self.assertEqual(spawn.call_count, 1) + + def test_unknown_process_error_keeps_original_identity_without_replay(self): + error = OSError("unknown process outcome") + with patch.object(compose.subprocess, "Popen", side_effect=error) as spawn: + with self.assertRaises(OSError) as caught: + compose.execute_bake(COMMAND, {"target": {"main": {}}}) + self.assertIs(caught.exception, error) + self.assertEqual(spawn.call_count, 1) + + def test_process_signal_exit_is_not_an_export_retry(self): + for code in (-15, 130, 137): + with self.subTest(exit_code=code): + with ( + patch.object( + compose.subprocess, "Popen", return_value=process(code, MISSING) + ) as spawn, + patch.object( + compose.sys, "stderr", SimpleNamespace(buffer=io.BytesIO()) + ), + self.assertRaises(subprocess.CalledProcessError), + ): + compose.execute_bake(COMMAND, {"target": {"main": {}}}) + self.assertEqual(spawn.call_count, 1) + + def test_only_native_bake_is_eligible(self): + with patch.object(compose.subprocess, "Popen") as spawn: + with self.assertRaises(ValueError): + compose.execute_bake( + ["docker", "compose", "up"], {"target": {"main": {}}} + ) + spawn.assert_not_called() + + +if __name__ == "__main__": + unittest.main() diff --git a/builder/managed-docker/test_image_provider.py b/builder/managed-docker/test_image_provider.py new file mode 100644 index 00000000..1998d1c2 --- /dev/null +++ b/builder/managed-docker/test_image_provider.py @@ -0,0 +1,176 @@ +import base64 +import importlib.util +import json +import tempfile +import unittest +from pathlib import Path +from unittest.mock import MagicMock, patch + +module_spec = importlib.util.spec_from_file_location( + "provider", Path(__file__).with_name("image_provider.py") +) +provider = importlib.util.module_from_spec(module_spec) +module_spec.loader.exec_module(provider) + + +class ImageProviderTests(unittest.TestCase): + def test_metadata_uses_node_authentication_and_checks_digest(self): + data = b'{"mediaType":"application/vnd.oci.image.manifest.v1+json"}' + identity = provider.digest(data) + connection = MagicMock() + response = connection.getresponse.return_value + response.status = 200 + response.read.return_value = json.dumps( + {"data": base64.b64encode(data).decode(), "media_type": "application/json"} + ).encode() + with patch.object(provider, "UnixConnection", return_value=connection): + obj, _ = self.provider.fetch("poc/main", "manifests", identity) + self.assertEqual(obj["mediaType"], "application/vnd.oci.image.manifest.v1+json") + self.assertEqual( + connection.request.call_args.args[:2], ("POST", "/registry_metadata") + ) + self.assertNotIn( + "Authorization", connection.request.call_args.kwargs["headers"] + ) + connection.close.assert_called_once() + + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + self.provider = provider.ImageProvider( + self.root / "collection", "registry.test" + ) + + def test_only_authorized_digest_references_are_accepted(self): + digest = provider.digest(b"artifact") + self.assertEqual( + self.provider.validate_reference("registry.test/poc/main@" + digest), + ("poc/main", digest), + ) + for value in [ + "registry.test/poc/main:latest", + "foreign.test/poc/main@" + digest, + "registry.test/private/main@" + digest, + "registry.test/poc/../other@" + digest, + "registry.test/poc/main@sha256:bad", + ]: + with self.subTest(value=value), self.assertRaises(ValueError): + self.provider.validate_reference(value) + + def test_invalid_reference_cannot_start_mount(self): + self.provider.mount_image = lambda _: self.fail( + "invalid request reached privileged mount" + ) + with self.assertRaises(ValueError): + self.provider.prepare("registry.test/poc/main:latest") + + def test_single_image_artifact_reads_config_without_mount_or_registration(self): + config = { + "os": "linux", + "architecture": "amd64", + "rootfs": {"diff_ids": [provider.digest(b"root")]}, + "config": {"Env": ["MODE=native"], "WorkingDir": "/app", "User": "1000"}, + } + payload = json.dumps(config).encode() + config_desc = {"digest": provider.digest(payload), "size": len(payload)} + manifest = { + "config": config_desc, + "layers": [{"mediaType": "application/vnd.oci.image.layer.nydus.blob.v1"}], + } + reference = "registry.test/poc/main@" + provider.digest( + json.dumps(manifest).encode() + ) + + def fetch(_repository, kind, selected, descriptor=None): + if kind == "manifests": + return manifest, {"descriptor": {"digest": selected}} + self.assertEqual(descriptor, config_desc) + name = selected.replace(":", "-") + ".json" + (self.provider.metadata / name).write_bytes(payload) + return config, {"descriptor": config_desc, "file": name} + + with ( + patch.object(self.provider, "fetch", side_effect=fetch), + patch.object(self.provider, "mount_image") as mount, + patch.object(provider.subprocess, "run") as run, + ): + actual = self.provider.describe_artifact(reference) + self.assertEqual( + actual, {"reference": reference, "image_process": config["config"]} + ) + mount.assert_not_called() + run.assert_not_called() + self.assertEqual(json.loads(self.provider.plan.read_text())["images"], []) + + def test_digest_chain_matches_oci_definition(self): + first, second = provider.digest(b"one"), provider.digest(b"two") + self.assertEqual(provider.chain_id([first]), first) + self.assertEqual( + provider.chain_id([first, second]), + provider.digest((first + " " + second).encode()), + ) + with self.assertRaises(ValueError): + provider.chain_id([]) + + def test_plan_is_published_atomically_and_read_only(self): + plan = {"images": [{"reference": "kept"}], "automatic": True} + provider.atomic_json(self.provider.plan, plan) + self.assertEqual(json.loads(self.provider.plan.read_text()), plan) + self.assertFalse(self.provider.plan.with_name("plan.json.new").exists()) + self.assertEqual(self.provider.plan.stat().st_mode & 0o222, 0) + + def test_prepared_reference_reuses_existing_read_only_root(self): + reference = "registry.test/poc/main@" + provider.digest(b"artifact") + provider.atomic_json( + self.provider.plan, + {"images": [{"reference": reference}], "automatic": True}, + ) + self.provider.mount_image = lambda _: self.fail( + "existing registration was mounted twice" + ) + self.provider.validate_mount = lambda _: None + self.assertTrue(self.provider.prepare(reference)["reused"]) + + def test_same_digest_alias_does_not_stack_bind_mounts(self): + identity = provider.digest(b"artifact") + existing = { + "reference": "registry.test/poc/one@" + identity, + "root": "/opt/akernel-images/" + identity[7:], + "chain_id": "chain", + "target": {}, + "config": {}, + "blobs": [], + } + provider.atomic_json( + self.provider.plan, {"images": [existing], "automatic": True} + ) + reference = "registry.test/poc/two@" + identity + self.provider.describe = lambda _: dict(existing, reference=reference) + self.provider.validate_mount = lambda _: None + self.provider.mount_image = lambda _: self.fail( + "same immutable root was mounted twice" + ) + self.assertTrue(self.provider.prepare(reference)["reused"]) + self.assertEqual(len(json.loads(self.provider.plan.read_text())["images"]), 2) + + def test_missing_registered_mount_fails_without_replacing_live_root(self): + reference = "registry.test/poc/main@" + provider.digest(b"artifact") + provider.atomic_json( + self.provider.plan, + {"images": [{"reference": reference}], "automatic": True}, + ) + + def unavailable(_): + raise RuntimeError("missing root") + + self.provider.validate_mount = unavailable + self.provider.mount_image = lambda _: self.fail( + "missing live root was silently replaced" + ) + with self.assertRaisesRegex(RuntimeError, "missing root"): + self.provider.prepare(reference) + + +if __name__ == "__main__": + unittest.main() diff --git a/builder/managed-docker/test_profile_metadata.py b/builder/managed-docker/test_profile_metadata.py new file mode 100644 index 00000000..17afedfc --- /dev/null +++ b/builder/managed-docker/test_profile_metadata.py @@ -0,0 +1,17 @@ +import unittest + +from profile_metadata import LABEL_PREFIX, resource_labels + + +class ProfileMetadata(unittest.TestCase): + def labels(self, **geometry): + return resource_labels({"plugin": {"managed_runtime": {"profiles": {"harbor-v1": geometry}}}}) + + def test_floor_includes_upper_and_usable_root(self): + self.assertEqual(self.labels(writable_mb=4096), {LABEL_PREFIX + "harbor-v1": "8192"}) + self.assertEqual(self.labels(writable_mb=4096, minimum_root_mb=8192), {LABEL_PREFIX + "harbor-v1": "12288"}) + + def test_invalid_geometry_fails_before_advertisement(self): + for geometry in ({}, {"writable_mb": 0}, {"writable_mb": True}, {"writable_mb": 4096, "minimum_root_mb": -1}, {"writable_mb": 1 << 44}): + with self.subTest(geometry=geometry), self.assertRaises(ValueError): + self.labels(**geometry) diff --git a/builder/managed-docker/test_provider_service.py b/builder/managed-docker/test_provider_service.py new file mode 100644 index 00000000..52ea1222 --- /dev/null +++ b/builder/managed-docker/test_provider_service.py @@ -0,0 +1,135 @@ +"""A runtime capability cannot select another environment's collection.""" + +import hashlib +import fcntl +import json +import os +from pathlib import Path +import sys +import tempfile +import unittest +from unittest.mock import patch + +sys.path.insert(0, str(Path(__file__).parent)) +from provider_service import RuntimeCollections + + +class CollectionAuthorization(unittest.TestCase): + def managed_collection(self, root): + for part in ["records", "leases", "owners", "retired"]: + (root / part).mkdir() + identity = "a" * 32 + token = identity + ".private-secret" + record = root / "records" / (identity + ".json") + record.write_text( + json.dumps( + { + "collection_id": identity, + "lifecycle_version": 1, + "registry": "registry.test", + "repository_prefix": "managed/" + identity + "/", + "scheme": "https", + "token_sha256": hashlib.sha256(token.encode()).hexdigest(), + } + ) + ) + record.chmod(0o600) + anchor = root / "leases" / identity + anchor.touch(mode=0o600) + os.link(anchor, root / "owners" / "source") + return identity, token, anchor + + def test_metadata_only_operation_uses_same_capability_and_holds_retirement_lock( + self, + ): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + identity, token, anchor = self.managed_collection(root) + broker = RuntimeCollections(root, "/private/image.sock") + artifact = { + "reference": "registry.test/managed/" + + identity + + "/single@sha256:" + + "a" * 64 + } + + def describe(reference): + with anchor.open() as candidate: + with self.assertRaises(BlockingIOError): + fcntl.flock(candidate.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + return {"reference": reference} + + with patch("provider_service.ImageProvider") as factory: + factory.return_value.describe_artifact.side_effect = describe + self.assertEqual( + broker.perform( + "Bearer " + token, "/describe", artifact["reference"] + ), + artifact, + ) + self.assertEqual(factory.call_args.kwargs["owner"], identity) + factory.return_value.prepare.assert_not_called() + + def test_deleted_owner_and_retired_capability_cannot_mount(self): + for state in ["unowned", "retired", "missing"]: + with self.subTest(state=state), tempfile.TemporaryDirectory() as directory: + root = Path(directory) + identity, token, anchor = self.managed_collection(root) + if state == "unowned": + (root / "owners" / "source").unlink() + elif state == "retired": + (root / "retired" / identity).touch() + else: + anchor.unlink() + with patch("provider_service.ImageProvider") as factory: + with self.assertRaises(PermissionError): + RuntimeCollections(root, "/private/image.sock").perform( + "Bearer " + token, "/prepare", "immutable" + ) + factory.assert_not_called() + + def test_finished_operations_do_not_keep_collection_provider_cache_alive(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + _identity, token, _anchor = self.managed_collection(root) + broker = RuntimeCollections(root, "/private/image.sock") + with patch( + "provider_service.ImageProvider.describe_artifact", + return_value={"reference": "verified"}, + ): + self.assertEqual( + broker.perform("Bearer " + token, "/describe", "verified"), + {"reference": "verified"}, + ) + self.assertEqual(len(broker.providers), 0) + + def test_invalid_or_foreign_capability_cannot_prepare_mounts(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "records").mkdir() + identity = "a" * 32 + token = identity + ".private-secret" + record = root / "records" / (identity + ".json") + record.write_text( + json.dumps( + { + "collection_id": identity, + "token_sha256": hashlib.sha256(token.encode()).hexdigest(), + } + ) + ) + record.chmod(0o600) + broker = RuntimeCollections(root, "/private/image.sock") + for auth in [ + "", + "Basic secret", + "Bearer ../node.secret", + "Bearer " + identity + ".wrong", + "Bearer " + "b" * 32 + ".private-secret", + ]: + with self.subTest(auth=auth), self.assertRaises(PermissionError): + broker.authorize(auth) + self.assertEqual(broker.providers, {}) + record.chmod(0o644) + with self.assertRaises(PermissionError): + broker.authorize("Bearer " + token) diff --git a/builder/managed-docker/test_publication.py b/builder/managed-docker/test_publication.py new file mode 100644 index 00000000..ac861bcb --- /dev/null +++ b/builder/managed-docker/test_publication.py @@ -0,0 +1,397 @@ +"""Resolve a post-publication history error without replaying the native build.""" + +import base64 +import hashlib +import importlib.util +import json +import os +import subprocess +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +spec = importlib.util.spec_from_file_location( + "runtime_publication", Path(__file__).with_name("compose.py") +) +compose = importlib.util.module_from_spec(spec) +spec.loader.exec_module(compose) + +TAG = "build-" + "a" * 32 +REFERENCE = "registry.test/poc/job/main:" + TAG +RAW = json.dumps( + { + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "config": {"digest": "sha256:" + "b" * 64}, + "layers": [], + } +).encode() +DIGEST = "sha256:" + hashlib.sha256(RAW).hexdigest() +INSPECTION = { + "Descriptor": { + "digest": DIGEST, + "size": len(RAW), + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "platform": {"os": "linux", "architecture": "amd64"}, + }, + "Raw": base64.b64encode(RAW).decode(), +} +HISTORY = ( + b"ERROR: failed to solve: Internal: unknown blob sha256:" + + b"c" * 64 + + b" in history\n" +) +DEADLINE = b"ERROR: failed to solve: Internal: context deadline exceeded\n" +PUSH = f"#19 pushing manifest for {REFERENCE}@{DIGEST} 0.2s done\n".encode() +PLAN = { + "target": {"main": {"tags": [REFERENCE]}}, + "group": {"default": {"targets": ["main"]}}, +} +PROFILE = { + "registry": "registry.test", + "prefix": "poc/job", + "registry_insecure": True, + "builder": "remote-test", +} + + +class PublicationResultTest(unittest.TestCase): + def test_explicit_metadata_provenance_setting_is_preserved(self): + with ( + tempfile.TemporaryDirectory() as root, + patch.dict(os.environ, {"BUILDX_METADATA_PROVENANCE": "max"}), + patch.object(compose, "execute_bake") as solve, + ): + + def completed(*_): + self.assertEqual(os.environ["BUILDX_METADATA_PROVENANCE"], "max") + (Path(root) / "metadata.json").write_text( + json.dumps({"main": {"containerimage.digest": DIGEST}}) + ) + + solve.side_effect = completed + self.assertEqual(self.call(root)["main"]["containerimage.digest"], DIGEST) + + def test_deadline_adopts_only_an_acknowledged_matching_published_output(self): + error = subprocess.CalledProcessError(100, ["docker"], stderr=PUSH + DEADLINE) + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error) as solve, + patch.object(compose, "execute", return_value=json.dumps(INSPECTION)), + ): + result = self.call(root) + self.assertEqual(result["main"]["containerimage.digest"], DIGEST) + self.assertEqual(solve.call_count, 1) + receipt = json.loads( + (Path(root) / "metadata.json.publication.json").read_text() + ) + self.assertEqual( + receipt["failure_kind"], "acknowledged-publication-deadline" + ) + self.assertFalse(receipt["native_build_replayed"]) + + def test_deadline_without_a_complete_matching_push_acknowledgement_fails_closed( + self, + ): + for log in [ + DEADLINE, + PUSH.replace(b" done", b" pending") + DEADLINE, + PUSH.replace(REFERENCE.encode(), b"registry.test/other:built") + DEADLINE, + ]: + error = subprocess.CalledProcessError(100, ["docker"], stderr=log) + with ( + self.subTest(log=log), + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute") as inspect, + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root) + self.assertIs(caught.exception, error) + inspect.assert_not_called() + + def test_deadline_preserves_failure_on_push_digest_mismatch(self): + error = subprocess.CalledProcessError( + 100, + ["docker"], + stderr=PUSH.replace(DIGEST.encode(), ("sha256:" + "d" * 64).encode()) + + DEADLINE, + ) + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute", return_value=json.dumps(INSPECTION)), + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root) + self.assertIs(caught.exception, error) + self.assertFalse((Path(root) / "metadata.json").exists()) + + def test_deadline_partial_group_and_explicit_attestations_are_not_adopted(self): + partial = { + "target": { + "main": {"tags": [REFERENCE]}, + "loadgen": {"tags": ["registry.test/poc/job/loadgen:" + TAG]}, + }, + "group": {"default": {"targets": ["main", "loadgen"]}}, + } + attested = { + "target": {"main": {"tags": [REFERENCE], "attest": ["type=provenance"]}}, + "group": {"default": {"targets": ["main"]}}, + } + for plan in [partial, attested]: + error = subprocess.CalledProcessError( + 100, ["docker"], stderr=PUSH + DEADLINE + ) + with ( + self.subTest(plan=plan), + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute") as inspect, + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root, plan) + self.assertIs(caught.exception, error) + inspect.assert_not_called() + + def test_deadline_does_not_adopt_signals_or_other_internal_failures(self): + for code, log in [ + (1, PUSH + DEADLINE), + (137, PUSH + DEADLINE), + (100, PUSH + b"ERROR: failed to solve: Internal: other error\n"), + ]: + error = subprocess.CalledProcessError(code, ["docker"], stderr=log) + with ( + self.subTest(code=code), + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute") as inspect, + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root) + self.assertIs(caught.exception, error) + inspect.assert_not_called() + + def call(self, root, plan=PLAN, profile=PROFILE): + metadata = Path(root) / "metadata.json" + command = [ + "docker", + "buildx", + "bake", + "--metadata-file", + str(metadata), + "default", + ] + return compose.bake_metadata(command, plan, metadata, profile) + + def test_history_failure_adopts_hash_verified_published_image_without_second_solve( + self, + ): + error = subprocess.CalledProcessError(100, ["docker"], stderr=HISTORY) + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error) as solve, + patch.object( + compose, "execute", return_value=json.dumps(INSPECTION) + ) as inspect, + ): + result = self.call(root) + self.assertEqual(result["main"]["containerimage.digest"], DIGEST) + self.assertTrue(result["main"]["akernel.publication-reconciled"]) + self.assertEqual(solve.call_count, 1) + self.assertEqual( + inspect.call_args.args[0], + ["docker", "manifest", "inspect", "--verbose", "--insecure", REFERENCE], + ) + self.assertTrue((Path(root) / "metadata.json.publication.json").exists()) + + def test_success_uses_fresh_native_metadata_without_registry_lookup(self): + with tempfile.TemporaryDirectory() as root: + metadata = Path(root) / "metadata.json" + metadata.write_text( + json.dumps({"main": {"containerimage.digest": "sha256:" + "f" * 64}}) + ) + + def solve(*_): + self.assertFalse(metadata.exists()) + metadata.write_text( + json.dumps({"main": {"containerimage.digest": DIGEST}}) + ) + + with ( + patch.object(compose, "execute_bake", side_effect=solve), + patch.object(compose, "execute") as inspect, + ): + self.assertEqual( + self.call(root)["main"]["containerimage.digest"], DIGEST + ) + inspect.assert_not_called() + + def test_non_history_authorization_and_signal_failures_keep_original_error(self): + for code, message in [ + (1, HISTORY), + (100, b"ERROR: unauthorized\n"), + (137, HISTORY), + (100, HISTORY + b"ERROR: final error is a different failure\n"), + ]: + error = subprocess.CalledProcessError(code, ["docker"], stderr=message) + with ( + self.subTest(code=code, message=message), + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute") as inspect, + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root) + self.assertIs(caught.exception, error) + inspect.assert_not_called() + + def test_unknown_process_outcome_is_not_reconciled_or_replayed(self): + error = OSError("unknown process outcome") + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute") as inspect, + ): + with self.assertRaises(OSError) as caught: + self.call(root) + self.assertIs(caught.exception, error) + inspect.assert_not_called() + + def test_reused_mutable_tag_and_foreign_repository_cannot_authorize_recovery(self): + for reference in [ + "registry.test/poc/job/main:built", + "other.test/poc/job/main:" + TAG, + ]: + error = subprocess.CalledProcessError(100, ["docker"], stderr=HISTORY) + plan = { + "target": {"main": {"tags": [reference]}}, + "group": {"default": {"targets": ["main"]}}, + } + with ( + self.subTest(reference=reference), + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute") as inspect, + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root, plan) + self.assertIs(caught.exception, error) + inspect.assert_not_called() + + def test_invalid_registry_content_never_becomes_success(self): + variants = [ + [], + {**INSPECTION, "Raw": base64.b64encode(b"wrong bytes").decode()}, + { + **INSPECTION, + "Descriptor": { + **INSPECTION["Descriptor"], + "digest": "sha256:" + "d" * 64, + }, + }, + { + **INSPECTION, + "Descriptor": {**INSPECTION["Descriptor"], "size": len(RAW) + 1}, + }, + { + **INSPECTION, + "Descriptor": { + **INSPECTION["Descriptor"], + "platform": {"os": "linux", "architecture": "arm64"}, + }, + }, + ] + for value in variants: + error = subprocess.CalledProcessError(100, ["docker"], stderr=HISTORY) + with ( + self.subTest(value=value), + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute", return_value=json.dumps(value)), + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root) + self.assertIs(caught.exception, error) + self.assertFalse((Path(root) / "metadata.json").exists()) + + def test_partial_compose_publication_preserves_failure_and_original_diagnostics( + self, + ): + second = "registry.test/poc/job/loadgen:" + TAG + plan = { + "target": {"main": {"tags": [REFERENCE]}, "loadgen": {"tags": [second]}}, + "group": {"default": {"targets": ["main", "loadgen"]}}, + } + error = subprocess.CalledProcessError(100, ["docker"], stderr=HISTORY) + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object( + compose, + "execute", + side_effect=[json.dumps(INSPECTION), RuntimeError("manifest missing")], + ), + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root, plan) + self.assertIs(caught.exception, error) + self.assertEqual(caught.exception.stderr, HISTORY) + self.assertFalse((Path(root) / "metadata.json").exists()) + + def test_no_cache_does_not_replay_a_solve_during_publication_recovery(self): + plan = { + "target": {"main": {"tags": [REFERENCE], "no-cache": True}}, + "group": {"default": {"targets": ["main"]}}, + } + error = subprocess.CalledProcessError(100, ["docker"], stderr=HISTORY) + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error) as solve, + patch.object(compose, "execute", return_value=json.dumps(INSPECTION)), + ): + self.assertEqual( + self.call(root, plan)["main"]["containerimage.digest"], DIGEST + ) + self.assertEqual(solve.call_count, 1) + + def test_receipt_write_failure_preserves_original_build_error(self): + error = subprocess.CalledProcessError(100, ["docker"], stderr=HISTORY) + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "execute_bake", side_effect=error), + patch.object(compose, "execute", return_value=json.dumps(INSPECTION)), + patch.object(Path, "write_text", side_effect=OSError("disk full")), + ): + with self.assertRaises(subprocess.CalledProcessError) as caught: + self.call(root) + self.assertIs(caught.exception, error) + self.assertIsInstance(caught.exception.__cause__, OSError) + self.assertFalse((Path(root) / "metadata.json").exists()) + + def test_single_build_calls_allocate_distinct_native_publication_tags(self): + with ( + tempfile.TemporaryDirectory() as root, + patch.object(compose, "start_builder"), + patch.object( + compose, "describe_image", side_effect=lambda _, reference: reference + ), + ): + tags = [] + + def solve(_args, plan, _path, _profile): + tags.append(plan["target"]["single"]["tags"][0]) + return {"single": {"containerimage.digest": DIGEST}} + + with patch.object(compose, "bake_metadata", side_effect=solve): + first = compose.image_build(root, PROFILE) + second = compose.image_build(root, PROFILE) + self.assertEqual(first, second) + self.assertNotEqual(tags[0], tags[1]) + for tag in tags: + self.assertRegex(tag, r":build-[0-9a-f]{32}$") + + +if __name__ == "__main__": + unittest.main() diff --git a/builder/node.Dockerfile b/builder/node.Dockerfile index a01bdb1a..bf0b8abf 100644 --- a/builder/node.Dockerfile +++ b/builder/node.Dockerfile @@ -38,8 +38,8 @@ ARG OTELCOL_CONTRIB_SHA256= ARG AKERNEL_VERSION=unknown ARG AKERNEL_REVISION=unknown ARG ADX_VERSION=v0.1.0rc2 -ARG ADX_RELEASE_URL=https://agent-dx.obs.cn-southwest-2.myhuaweicloud.com/adx/daily/20261106003204-e611b9e4a83d/linux/amd64/adx-release.tar.gz -ARG ADX_RELEASE_SHA256=f53816d8378dc43e62755dc173e093d7237105e1b190fd8297dd934692cdb403 +ARG ADX_RELEASE_URL=https://agent-dx.obs.cn-southwest-2.myhuaweicloud.com/adx/daily/20261106064716-2f61d44d9c56/linux/amd64/adx-release.tar.gz +ARG ADX_RELEASE_SHA256=e4390134ed9428936a241622dfd01a4420b0ddc494b3dab40bf0b9712238cf33 FROM ${AKERNEL_NODE_BASE_IMAGE} AS adx-release ARG ADX_RELEASE_URL @@ -473,6 +473,7 @@ COPY ./builder/scripts/prepare-adx-config.py /usr/local/libexec/prepare-adx-conf COPY ./builder/scripts/adx-service.sh /usr/local/bin/adx-service COPY ./builder/scripts/otel-collector-service.sh /usr/local/bin/otel-collector-service COPY ./builder/scripts/sandboxd_network_prepare.sh /usr/local/bin/sandboxd-network-prepare +COPY ./builder/managed-docker/image_provider.py ./builder/managed-docker/provider_service.py /usr/local/lib/akernel/managed-docker/ ARG FIRECRACKER_KERNEL_PROFILE COPY ./builder/config/adx-standalone.yaml /etc/akernel/adx-standalone.yaml RUN case "${FIRECRACKER_KERNEL_PROFILE}" in \ diff --git a/builder/runtime.Dockerfile b/builder/runtime.Dockerfile index aa7ff7f0..50b361aa 100644 --- a/builder/runtime.Dockerfile +++ b/builder/runtime.Dockerfile @@ -4,8 +4,8 @@ ARG AKERNEL_RUNTIME_BASE_IMAGE=ubuntu:24.04 ARG ADX_VERSION=v0.1.0rc2 -ARG ADX_EXECD_URL=https://agent-dx.obs.cn-southwest-2.myhuaweicloud.com/adx/daily/20261106003204-e611b9e4a83d/linux/amd64/adx-execd.tar.gz -ARG ADX_EXECD_SHA256=3fca4c022eea8e35f4ec72111682484d7b7af0ccbad2723c920868a68c53a9ba +ARG ADX_EXECD_URL=https://agent-dx.obs.cn-southwest-2.myhuaweicloud.com/adx/daily/20261106064716-2f61d44d9c56/linux/amd64/adx-execd.tar.gz +ARG ADX_EXECD_SHA256=240bfcc19326faab9ea5ffa7587c9e6e5ca48241a86de74778b9a6c26403bbee FROM ${AKERNEL_RUNTIME_BASE_IMAGE} AS adx-execd ARG ADX_EXECD_URL diff --git a/builder/systemd_services/akernel-managed-images.service b/builder/systemd_services/akernel-managed-images.service new file mode 100644 index 00000000..cdb22154 --- /dev/null +++ b/builder/systemd_services/akernel-managed-images.service @@ -0,0 +1,15 @@ +[Unit] +Description=AKernel managed Docker image collection broker (opt-in) +After=sandboxd.service +Requires=sandboxd.service +PartOf=sandboxd.service +ConditionPathExists=/etc/akernel/managed-runtime.env + +[Service] +EnvironmentFile=/etc/akernel/managed-runtime.env +ExecStart=/usr/bin/python3 /usr/local/lib/akernel/managed-docker/provider_service.py --root ${MANAGED_RUNTIME_ROOT} --image-socket ${MANAGED_IMAGE_SOCKET} --port ${MANAGED_IMAGE_PORT} +Restart=on-failure +UMask=0077 + +[Install] +WantedBy=sandboxd.service diff --git a/src/agent-dx b/src/agent-dx index e611b9e4..2f61d44d 160000 --- a/src/agent-dx +++ b/src/agent-dx @@ -1 +1 @@ -Subproject commit e611b9e4a83dcd6da5346bd17c2f6af28e244952 +Subproject commit 2f61d44d9c56ae029a06698c337cbdc4aeaaa7af diff --git a/src/sandboxd b/src/sandboxd index fbe69e56..d23687d6 160000 --- a/src/sandboxd +++ b/src/sandboxd @@ -1 +1 @@ -Subproject commit fbe69e566859718414a207fa780b8dab363b9381 +Subproject commit d23687d62ef0afd643b4028aedbcbc0e8bd346dc From f14d13514e0085f7b2f4b732beab1fffab62fd17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9C=88=E5=8F=AF?= Date: Fri, 6 Nov 2026 17:37:52 +0800 Subject: [PATCH 3/5] feat(sdk): expose managed build and checkpointable Compose handles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add image, project, service and process handles with managed profile admission, network forwarding and restore attachment without replay. Preserve command timeout identity and partial results, and coalesce authenticated resource discovery. Include SDK regressions, integration probes, examples and API guidance. Signed-off-by: 月可 --- sdk/python/README.md | 268 +++++ sdk/python/akernel_sdk/__init__.py | 27 + sdk/python/akernel_sdk/_addresses.py | 5 + sdk/python/akernel_sdk/_backends/adx.py | 181 +++- sdk/python/akernel_sdk/_backends/base.py | 2 + sdk/python/akernel_sdk/_backends/errors.py | 13 +- sdk/python/akernel_sdk/_compose.py | 945 ++++++++++++++++++ .../akernel_sdk/_docker_network_guest.py | 161 +++ sdk/python/akernel_sdk/_docker_runtime.py | 253 +++++ sdk/python/akernel_sdk/_image_bridge_guest.py | 240 +++++ sdk/python/akernel_sdk/_image_plan.py | 175 ++++ sdk/python/akernel_sdk/_image_resolver.py | 246 +++++ sdk/python/akernel_sdk/_managed_build.py | 186 ++++ sdk/python/akernel_sdk/_service_process.py | 191 ++++ .../akernel_sdk/_service_process_guest.py | 170 ++++ sdk/python/akernel_sdk/commands.py | 5 +- sdk/python/akernel_sdk/sandbox.py | 8 + sdk/python/docs/managed-compose.md | 160 +++ sdk/python/examples/compose_checkpoint.py | 67 ++ sdk/python/examples/managed_compose.py | 31 + sdk/python/pyproject.toml | 5 +- .../tests/integration/test_compose_network.py | 335 +++++++ sdk/python/tests/unit/test_addresses.py | 50 +- sdk/python/tests/unit/test_adx_backend.py | 396 +++++++- sdk/python/tests/unit/test_command_compat.py | 19 +- sdk/python/tests/unit/test_compose.py | 337 +++++++ sdk/python/tests/unit/test_compose_network.py | 285 ++++++ sdk/python/tests/unit/test_docker_runtime.py | 396 ++++++++ sdk/python/tests/unit/test_fork.py | 4 + sdk/python/tests/unit/test_image_plan.py | 139 +++ sdk/python/tests/unit/test_managed_compose.py | 165 +++ .../unit/test_managed_runtime_profile.py | 220 ++++ sdk/python/tests/unit/test_service_process.py | 217 ++++ sdk/python/tests/unit/test_snapshot_create.py | 5 + sdk/python/tests/unit/test_snapshots.py | 13 +- sdk/python/tests/unit/test_types.py | 48 +- 36 files changed, 5902 insertions(+), 66 deletions(-) create mode 100644 sdk/python/akernel_sdk/_compose.py create mode 100644 sdk/python/akernel_sdk/_docker_network_guest.py create mode 100644 sdk/python/akernel_sdk/_docker_runtime.py create mode 100644 sdk/python/akernel_sdk/_image_bridge_guest.py create mode 100644 sdk/python/akernel_sdk/_image_plan.py create mode 100644 sdk/python/akernel_sdk/_image_resolver.py create mode 100644 sdk/python/akernel_sdk/_managed_build.py create mode 100644 sdk/python/akernel_sdk/_service_process.py create mode 100644 sdk/python/akernel_sdk/_service_process_guest.py create mode 100644 sdk/python/docs/managed-compose.md create mode 100644 sdk/python/examples/compose_checkpoint.py create mode 100644 sdk/python/examples/managed_compose.py create mode 100644 sdk/python/tests/integration/test_compose_network.py create mode 100644 sdk/python/tests/unit/test_compose.py create mode 100644 sdk/python/tests/unit/test_compose_network.py create mode 100644 sdk/python/tests/unit/test_docker_runtime.py create mode 100644 sdk/python/tests/unit/test_image_plan.py create mode 100644 sdk/python/tests/unit/test_managed_compose.py create mode 100644 sdk/python/tests/unit/test_managed_runtime_profile.py create mode 100644 sdk/python/tests/unit/test_service_process.py diff --git a/sdk/python/README.md b/sdk/python/README.md index 17b95788..6eab24e5 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -1,5 +1,7 @@ # AKernel Python SDK +The experimental [managed remote-build runtime](../../builder/managed-docker/README.md) uses official Buildx remote and native Compose rather than the legacy Dockerfile interpreter. Its operator-prepared fresh descriptor binds with `ComposeProject.bind_runtime(sandbox, descriptor)`; this performs no upload, daemon startup, image preparation or build. After checkpoint restore, use `ComposeProject.attach()` so captured services and tunnels are reused. Docker `network_mode: service:...` follows the native namespace owner when exposing a bridge-accessible model tunnel. + `akernel-sdk` is the Python interface for creating and managing remote AKernel sandboxes. Applications use one stable API for commands, files, interactive PTYs, port forwarding, and reverse tunnels. @@ -88,6 +90,15 @@ Address behavior is deterministic: use 443 for HTTPS/WSS and 80 for HTTP/WS; an explicit port is preserved. Exec and file transfer continue to use `AKERNEL_SERVER_ADDRESS`. +For a shared HTTPS control and tunnel entrypoint, set both addresses explicitly: + +```bash +export AKERNEL_SERVER_ADDRESS="https://akernel.example.com" +export AKERNEL_GATEWAY_ADDRESS="$AKERNEL_SERVER_ADDRESS" +``` + +The shell expands the double-quoted variable in this example. The SDK does not expand `$NAME` or `${NAME}` within address values and rejects unresolved variables before creating a sandbox. Use literal URLs in configuration files whose loader does not expand variables. + ### TLS certificate verification `AKERNEL_VERIFY_TLS` controls certificate-chain and hostname verification for @@ -480,6 +491,13 @@ observation. The default continues returning `CommandResult`. Authentication, permission, missing-resource, timeout, rate-limit and service-busy exceptions subclass `BackendOperationError` and retain structured service context. +An ADX foreground command deadline raises `TimeoutException` with its native +error code and any available partial `command_result`. Its message identifies +the deadline; stdout/stderr remain command output. A wait deadline does not +establish a terminal remote exit. Managed builds and Compose operations retain +available partial output in their configured logs, preserving the original +exception if local log persistence fails. + Foreground commands return a backend-neutral `CommandResult`. Background commands return an AKernel `CommandHandle`; its lifecycle operations are delegated to the selected backend. @@ -1428,6 +1446,8 @@ are `None`; the original exception remains in `__cause__`. `cleanup_errors` is a tuple of rollback failures (empty when absent). Known-created IDs and these errors are also attached to interruption exceptions without changing their type. +`first_error` retains the native client's first failed attempt when a later retry reports a different failure; otherwise it is `None`. The final exception's `code`, `retry` and `outcome` still govern retry and cleanup decisions. The original final native exception remains in `__cause__`. + ```python from akernel_sdk import BackendOperationError, Sandbox @@ -1448,3 +1468,251 @@ results retain `outcome="unknown"` and `retry="same_operation"`; preserve the original identity when querying or retrying. Do not infer execution or cleanup from a timeout message. `status_code` describes the service failure (including an SSE final event's error code); it need not equal the stream's HTTP status. + + +### Client-managed Docker/Compose runtime + +For a platform-owned deployment, use the [managed profile](#platform-managed-native-docker-builds). The client-managed API below requires a compatible runsc deployment with Sandboxd managed file-backed tmpfs support. Select reflink checkpoint mode and durable sync in deployment configuration; the SDK never modifies cluster settings. See [runtime prerequisites](../../builder/image-bridge/README.md). + +```python +from pathlib import Path +from akernel_sdk import DockerRuntime, Sandbox, prepare_compose_images + +images = prepare_compose_images( + "./project", ["./project/docker-compose.yaml"], + output_dir="./diagnostics/images", + bridge_binary="/opt/tools/akernel-image-bridge", +) +runtime = DockerRuntime( + images=images, bridge_binary=Path("/opt/tools/akernel-image-bridge"), + writable_mb=2048, writable_backing="file", +) +sandbox = Sandbox( + image=DockerRuntime.default_image, cpu=4000, memory=8192, + storage_mb=30720, cwd="/", failover=False, + **runtime.sandbox_options(), +) +try: + runtime.start(sandbox, network="bridge") + # Upload the original project; run ordinary docker compose build/up here. + # Persist any client-side state that must stay aligned with the snapshot. + snapshot = sandbox.create_snapshot(timeout=300) + sandbox.kill() # Fence the source before creating a running clone. + restored = Sandbox.from_snapshot( + snapshot, cwd="/", failover=False, **runtime.sandbox_options(), + # Explicitly rebind any reverse_tunnel and current network_policy here. + ) + try: + runtime.reconnect_network(restored) # new outer TAP; no daemon restart + print(restored.commands.run("docker ps", check=True).stdout) + # Continue using restored, which wraps the captured processes. Do not + # call runtime.start(), rerun builds, or restart the agent command. + finally: + restored.kill() + Sandbox.delete_snapshot(snapshot.snapshot_id) +finally: + sandbox.kill() +``` + +The SDK prepares manifest/config metadata only. Sandboxd's existing OCI/ROFS image manager supplies immutable shared lower roots; containerd uses a private managed upper. Credentials for metadata discovery and node layer fetching are separate. No task image build/push or user image plan is required. Original definitions are read without rewriting them. Managed bridge networking preserves original default/custom IPv4 bridge networks, service DNS and TCP/UDP userland port mappings; Harbor continues to own its normal service, bind, resource and verifier overlays. + +`from_snapshot(..., extra_config=...)` and `create(snapshot_id=..., extra_config=...)` forward validated, defensively copied runtime configuration. Supply the same managed-upper geometry and image mounts on restore. Native ADX DTOs, client connection/TLS handling and reconstructed commands/files/tunnel handles stay inside the SDK. Snapshot restore does not capture local Python harness state or external services; callers own alignment, fencing the previous execution and deleting snapshots. + +`DockerRuntime.default_image` is the public Docker Hub image +`docker.io/library/docker:28.3.3-dind`. Fresh runtime startup installs missing +Alpine support packages (Python 3, Bash, util-linux and related tools). This +requires access to Docker Hub and Alpine package repositories; no custom runtime +image build or registry push is needed. A compatible prebuilt image can still be +selected explicitly to avoid installing packages during startup. Restore never +repeats runtime installation or starts another Docker daemon. + +### Compose projects and service processes + +`ComposeProject`, `ServiceHandle` and `ServiceProcessHandle` expose project, service and process operations. They use the same managed Docker runtime and existing ROFS provider described above. Their backend-neutral operations use the Agent DX API. The managed file-backed upper profile additionally requires compatible Sandboxd support; installing this Python SDK alone does not enable it. The framework/runtime boundary and prerequisites are described in [managed Compose integration](docs/managed-compose.md). A complete client-side example is in [compose_checkpoint.py](examples/compose_checkpoint.py). + +`ComposeProject.prepare()` freezes a private local copy of the selected context and Compose bytes and produces sandbox options. The platform-managed profile resolves services through official Compose after `start_runtime()`; client-managed mode discovers image inputs locally. Put `output_dir` outside the input context. Keep build contexts inside the selected directory. The default `network="bridge"` preserves original Compose network definitions, Docker IPAM, DNS and published TCP/UDP ports. The explicit `network="host"` compatibility profile still adds host/loopback aliases and rejects explicit Compose networks. Scaled service process handles are rejected. Task files are not rewritten. Generic SDK defaults contain no Harbor service or artifact names. + +In client-managed automatic image mode, both Dockerfile bases and image-only services use pinned ROFS mounts. `pull()` verifies their Docker references and committed snapshots; `up()` verifies them again and prevents native repeat pulls. This includes `image: name:tag@sha256:...`, for which native Compose's missing policy can still request layer extraction. Tags are resolved once during preparation; missing or changed registrations fail without fallback. Without automatic images, native Compose pull behavior is retained. + +Client-managed discovery requires an image bridge reporting `discover_schema: 2` +via `--mode capabilities`. Discovery +confines build/Dockerfile and `.env` reads to the frozen context and receives +only explicitly supplied project variables. `include` / `extends.file` and +remote contexts are not supported in client-managed discovery. Valid shared chains +across different manifests/configs are supported after OCI configuration checks. + +```python +from pathlib import Path +from akernel_sdk import ComposeProject, DockerRuntime, Sandbox + +project = ComposeProject.prepare( + "./project", ["./project/docker-compose.yaml"], + runtime=DockerRuntime(bridge_binary=Path("/opt/tools/akernel-image-bridge"), + writable_mb=2048, writable_backing="file"), + auto_images=True, output_dir="./artifacts/compose", +) +sandbox = Sandbox(image=DockerRuntime.default_image, cpu=4000, + memory=8192, storage_mb=30720, cwd="/", failover=False, + **project.sandbox_options()) +try: + project.start_runtime(sandbox) # fresh creation only + project.build(timeout=600) + project.pull(timeout=600) + # Apply the application's running network policy here if required. + project.up(wait_timeout=600) + worker = project.service("worker") + print(worker.exec("uname -s").stdout) # POSIX sh by default + process = worker.start_command("exec ./worker", cwd="/app") + snapshot = sandbox.create_snapshot(timeout=300) + project_state, command_id = project.descriptor, process.command_id + # Persist these JSON descriptors with application/harness state as needed. + sandbox.kill() # fence source before creating running clone + restored = Sandbox.from_snapshot(snapshot, cwd="/", failover=False, + **project.sandbox_options()) + try: + attached = ComposeProject.attach(restored, project_state) + attached.reconnect_network() # rebind owned NAT/forwarding rules only + original = restored.commands.connect(command_id) # no build/up/relaunch + finally: + restored.kill() + Sandbox.delete_snapshot(snapshot.snapshot_id) +finally: + sandbox.kill() +``` + +`ServiceHandle` provides exec, upload/download (including directories), stop and process start. `start_command()` starts native Compose exec through the sandbox command API without installing a service monitor or requiring Python in the service. Its handle PID belongs to the outer Docker CLI; persist its stable `command_id` and reconnect through `restored.commands.connect(command_id)` after whole-sandbox restore. The runtime captures the Docker CLI, dockerd and inner process together. Do not use the outer PID to signal an inner service process. Whole-sandbox checkpointing requires no application stop/continue or model-idle barrier. + +The separate explicit `start_process()` API returns a `ServiceProcessHandle` for callers that need inner PID/tree signalling. Select shell, login behavior, cwd, user and environment explicitly; the SDK does not assign a special main service. `ServiceProcessHandle` uses a stable process token, Docker container ID/start time and Linux process start time. Its in-service monitor and identity record are captured with the sandbox. It supports inspect, tree stop/continue/terminate/kill and exit-code wait. Python 3 is required only for managed service processes, not ordinary service exec. The process is detached from the short-lived docker-exec submission; outer command PIDs are not used to signal the inner process. + +Unknown start failures retain `error.process_descriptor`; attach using that token to observe the original execution instead of submitting it again. A restarted/replaced container or changed process identity is rejected. A partial runtime startup is fresh-only and cannot be replayed on the same project object; delete the owned sandbox after failure. Compose failures retain phase, exit code, stdout/stderr and diagnostic logs. The SDK never silently retries a side-effecting Compose operation with an unknown outcome. + +`ComposeProject.attach()` only reconstructs client bindings. It does not upload files, start daemons, build images or recreate services. Snapshot restore must retain the same mount/upper options, rebind tunnels and the current network policy, and fence the previous execution before resuming it. Application conversation state and checkpoint timing remain with the application. Descriptors may contain explicit project environment values; store them with the same protection as application configuration. + + +### Native Compose networking and model tunnels + +The Linux/amd64 runsc Docker profile requires Docker 28's operator-owned +`--net-raw=true` and `--allow-packet-socket-write=true` flags. The outer runtime +image supplies `ip`, `iptables-legacy` and `iptables-legacy-restore`. No task +Dockerfile or Compose edit is needed. These are node/runtime prerequisites; +the SDK never changes a deployment. Existing shared ROFS lower and file-backed +upper mounts are independent of the network choice. + +The SDK enables IPv4 forwarding/SNAT inside the sandbox and keeps independent +bridges from routing directly to each other. Docker owns bridge creation, IPAM, +embedded DNS and TCP/UDP userland port proxies. Global dockerd iptables handling +is disabled because this runsc lacks Docker's `addrtype` matcher. A daemon-only +PATH selects legacy tools for Docker's per-container DNS DNAT; system symlinks +and original user files are preserved. IPv6-only egress, non-bridge drivers and +arbitrary Docker firewall/driver options are outside the supported profile. + +After restoring to a new outer TAP/address, call `project.reconnect_network()` +(or `DockerRuntime.reconnect_network(restored)` for the low-level API) before +resuming work. It replaces only SDK-owned forwarding/NAT chains, retaining +conntrack and the captured bridges, containers, DNS/proxies and inner sockets. +`attach()` itself remains free of remote operations. Established connections +inside the sandbox, including through its published-port proxy, are retained. +Connections to external peers do not acquire address-migration guarantees. + +An `HttpReverseTunnel` listens on the **outer** loopback. For a service on its +own bridge, use the service-specific controlled entry point: + +```python +# Pass the same HttpReverseTunnel on fresh creation and Sandbox.from_snapshot. +url = project.service("worker").expose_tunnel(tunnel) +process = project.service("worker").start_process( + "exec python3 worker.py", envs={"MODEL_BASE_URL": url}, +) +``` + +`expose_tunnel()` binds an ephemeral port only on an attached IPv4 gateway and +accepts only that service's IP. It relays TCP bytes without changing URLs, +authentication headers, HTTP or SSE. An internal bridge can use its local +gateway without gaining general egress. Host-network services retain the +original loopback URL. The relay and JSON descriptor enter the sandbox snapshot; +restored projects reuse the entry point without starting another listener. +Container replacement, a missing relay descriptor on restore, or ambiguous +launch outcomes fail closed. Network-none and IPv6-only services cannot use this +IPv4 entry point. + +Run the opt-in real network regression only against an owned test instance: + +```sh +AKERNEL_RUN_COMPOSE_TESTS=1 \ +AKERNEL_TEST_DIND_IMAGE='' \ +AKERNEL_TEST_IMAGE_BRIDGE_BINARY='/absolute/path/akernel-image-bridge' \ +python -m unittest tests.integration.test_compose_network -v +``` + +It checks default/custom DNS, aliases, disjoint-network isolation, TCP/UDP port +mappings, peer-restricted model HTTP/auth/SSE, new egress, and original service +and published-port TCP sockets across two snapshot/delete/restore cycles. All +owned environments and snapshots are deleted; no deployment is changed. + +Explicit service process-tree control freezes waiters before descendants and thaws descendants before waiters, preserving Bash job-control wait semantics. Numeric PID order is not used as ancestry. Whole-sandbox checkpointing does not require these signals. + +### Platform-managed native Docker builds + +When an operator enables a named managed-runtime profile, clients submit its +capability name. They do not need Docker, Kubernetes access, a Registry route, +Registry credentials, a node path or an image-bridge executable: + +```python +from akernel_sdk import ComposeProject, DockerRuntime, Sandbox, build_image + +artifact = build_image("./environment", profile="harbor-v1") +sandbox = Sandbox(image=artifact.reference, env=artifact.envs) +# Use artifact.cwd / artifact.user as the task's execution defaults. +sandbox.kill() + +project = ComposeProject.prepare( + "./environment", ["./environment/docker-compose.yaml"], + runtime=DockerRuntime(profile="harbor-v1"), network="bridge", +) +sandbox = Sandbox(**project.sandbox_options()) +project.start_runtime(sandbox) +project.build() # Official Buildx remote; native Dockerfile frontend and cache. +project.pull() # Runtime prepares ROFS and registers the immutable references. +project.up() +service = project.service("main") +# Existing service/process and snapshot interfaces apply unchanged. +sandbox.kill() +``` + +The node authorizes the ADX tenant and supplies the pinned tool image, official +Buildx endpoint, private Registry account, collection capability and Docker +runtime configuration. The runtime pushes native Nydus OCI artifacts directly; +Dragonfly and Sandboxd's existing ROFS image manager distribute shared lowers. +The SDK only transfers task files, submits operations and returns artifact/service +handles. It does not interpret Dockerfiles or start a local Docker daemon. + +`no_cache=True` is an explicit rebuild choice; fresh environments retain native +BuildKit cache use by default. Build failure stdout/stderr can be retained using +`build_image(..., log_path=...)`. Cleanup failures retain environment identity. +`artifact.has_startup_command` reports whether the effective OCI ENTRYPOINT/CMD +is nonempty. Pass it as `Sandbox(..., inherit_entrypoint=artifact.has_startup_command)` +when the task should run the image's startup command. Images that clear both +commands still support interactive sandbox creation with inheritance disabled. +Optional `cpu_request` (millicores) and `memory_request` (MiB) change only the +temporary preparation sandbox's scheduling reservation. Its existing limits +remain 1000 millicores and 2048 MiB; each request must be positive and no greater +than its limit. Omit these options to preserve the default reservations. Lower +requests do not provide additional physical capacity for concurrent workloads. +The managed resource-geometry probe coalesces concurrent reads within one +authenticated SDK backend and reuses successful views for at most one second. +Errors are not cached. Creation still reaches the scheduler individually, and +node authorization and storage validation remain authoritative on every request. +Restore uses `ComposeProject.attach()` and saved descriptors; it never rebuilds +or reruns Docker startup. Snapshot metadata contains an opaque server-created +collection identity, checked against the tenant and profile on restore. + +This opt-in profile requires same-node retained collections and the same mount +view. Version-1 collections use durable local live/checkpoint leases and are +retired after their last reference disappears; retained checkpoints keep their +ROFS mounts pinned. Service restart recovers these leases before admission. +Legacy collections require an operator drain. This local mount cleanup does not +delete Registry artifacts. Cross-node reconstruction, Pod replacement with a +changed provider identity, and automatic Registry artifact GC remain outside +the qualified contract. +Operators own Registry TLS and repository-scoped authorization. A shared +Registry account does not provide per-tenant image isolation. diff --git a/sdk/python/akernel_sdk/__init__.py b/sdk/python/akernel_sdk/__init__.py index df3ba4cc..8a934648 100644 --- a/sdk/python/akernel_sdk/__init__.py +++ b/sdk/python/akernel_sdk/__init__.py @@ -15,6 +15,15 @@ """Public API for the AKernel Python SDK.""" import importlib +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from ._compose import ComposeOperationError, ComposeProject, ServiceHandle + from ._docker_runtime import DockerRuntime + from ._image_plan import ImageMountPlan + from ._image_resolver import prepare_compose_images + from ._managed_build import BuiltImage, build_image + from ._service_process import ServiceProcessHandle from ._backends.errors import ( AuthenticationException, @@ -106,9 +115,27 @@ "DockerfileBuildError", "DockerfileParseError", "BuildInstruction", + "ComposeProject", + "ComposeOperationError", + "ServiceHandle", + "ServiceProcessHandle", + "DockerRuntime", + "ImageMountPlan", + "prepare_compose_images", + "BuiltImage", + "build_image", ] _LAZY_IMPORTS = { + "BuiltImage": ("._managed_build", "BuiltImage"), + "build_image": ("._managed_build", "build_image"), + "ComposeProject": ("._compose", "ComposeProject"), + "ComposeOperationError": ("._compose", "ComposeOperationError"), + "ServiceHandle": ("._compose", "ServiceHandle"), + "ServiceProcessHandle": ("._service_process", "ServiceProcessHandle"), + "DockerRuntime": ("._docker_runtime", "DockerRuntime"), + "ImageMountPlan": ("._image_plan", "ImageMountPlan"), + "prepare_compose_images": ("._image_resolver", "prepare_compose_images"), "Client": ("._client", "Client"), "Sandbox": (".sandbox", "Sandbox"), "CommandHandle": (".commands", "CommandHandle"), diff --git a/sdk/python/akernel_sdk/_addresses.py b/sdk/python/akernel_sdk/_addresses.py index 5bfd781a..68c6825c 100644 --- a/sdk/python/akernel_sdk/_addresses.py +++ b/sdk/python/akernel_sdk/_addresses.py @@ -76,6 +76,11 @@ def _parse_endpoint( value = raw.strip() if not value: raise RuntimeError("address is empty") + if "$" in value: + raise RuntimeError( + "address contains an unexpanded environment variable; " + "set an explicit host or URL before creating a sandbox" + ) parsed = urlparse(value if "://" in value else f"//{value}") if not parsed.hostname: diff --git a/sdk/python/akernel_sdk/_backends/adx.py b/sdk/python/akernel_sdk/_backends/adx.py index 01c078ec..71fa1934 100644 --- a/sdk/python/akernel_sdk/_backends/adx.py +++ b/sdk/python/akernel_sdk/_backends/adx.py @@ -17,10 +17,13 @@ from __future__ import annotations import logging +import os +import re import threading import time from collections.abc import Mapping -from dataclasses import dataclass +from concurrent.futures import Future +from dataclasses import dataclass, replace from typing import Any, cast import adx_environment @@ -31,6 +34,7 @@ from adx_environment.tunnel_client import TunnelClient from .._fork import ForkCleanupError, retain_fork_failure +from .._resource_api import RESOURCE_PATH, parse_resource_nodes from .._sandboxes import SandboxQuery from .._timeouts import resolve_create_timeout from ..types import ( @@ -42,6 +46,7 @@ Mount, NetworkPolicy, NetworkRule, + NodeInfo, PortRange, SandboxInfo, SnapshotInfo, @@ -73,6 +78,7 @@ _NAMESPACE = "default" _DEFAULT_LISTEN_PORT = 8766 +_MANAGED_PROFILE_LABEL = "AKERNEL_RUNTIME_PROFILE_" logger = logging.getLogger(__name__) @@ -135,7 +141,11 @@ def _log_creation_budgets( logger.debug( "sandbox lifecycle budgets operation=%s schedule_timeout=%s " "init_timeout=%s create_timeout=%s idle_timeout=%s policy_source=request", - operation, spec.schedule_timeout, spec.init_timeout, timeout, spec.idle_timeout, + operation, + spec.schedule_timeout, + spec.init_timeout, + timeout, + spec.idle_timeout, ) @@ -304,6 +314,8 @@ def _convert_error(operation: str, error: Exception) -> BackendOperationError: request_forwarded=getattr(error, "request_forwarded", None), cleanup_errors=getattr(error, "cleanup_errors", ()) or (), command_id=getattr(error, "command_id", None), + first_error=getattr(error, "first_error", None), + command_result=getattr(error, "command_result", None), ) @@ -345,9 +357,13 @@ def _command_timeout_message(value: Any, timeout: int) -> str | None: ) ): return None - return str(getattr(value, "error_message", None) or stderr).strip() or ( - f"Command timed out after {timeout} seconds" + message = str(getattr(value, "error_message", None) or "").strip() + label = ( + f"Command wait timed out after {timeout} seconds" + if error_code == "WAIT_TIMEOUT" + else f"Command timed out after {timeout} seconds" ) + return label + (": " + message if message else "") def _command_info(value: Any) -> CommandInfo: @@ -387,7 +403,8 @@ def _remember(self, handle: Any, pid: int) -> CommandReference: def _handle(self, command: str | int) -> Any: handle = ( self._command_handles.get(command) - if isinstance(command, str) else self._handles.get(command) + if isinstance(command, str) + else self._handles.get(command) ) if handle is None: raise NotFoundException(f"no connected command for {command!r}") @@ -431,8 +448,12 @@ def run( timeout=timeout, ) if timeout_message := _command_timeout_message(value, timeout): - raise BackendOperationError( - f"command execution failed: {timeout_message}" + raise TimeoutException( + timeout_message, + code=getattr(value, "error_code", None) or "COMMAND_TIMEOUT", + retry=getattr(value, "retry", None), + outcome=getattr(value, "outcome", None), + command_result=_command_result(value), ) result = _command_result(value) except Exception as error: @@ -739,7 +760,9 @@ def fork( self._spec.init_timeout if init_timeout is None else init_timeout ) timeout = resolve_create_timeout( - self._spec.schedule_timeout, child_init_timeout, restore_timeout, + self._spec.schedule_timeout, + child_init_timeout, + restore_timeout, name="restore_timeout", ) # The pinned ADX API has snapshot/create/delete primitives, not a batch @@ -760,6 +783,7 @@ def fork( command_cwd=self._spec.command_cwd, port_forwardings=self._spec.port_forwardings, mounts=self._spec.mounts, + extra_config=self._spec.extra_config, reverse_tunnel=self._spec.reverse_tunnel, detached=False, failover=self._spec.failover, @@ -786,7 +810,8 @@ def fork( and error.outcome == "not_started" ): lease = _ForkSnapshotLease( - snapshot.snapshot_id, {child.id for child in children}, + snapshot.snapshot_id, + {child.id for child in children}, self._connection, ) for child in children: @@ -828,7 +853,8 @@ def _cleanup_interrupted_fork( pending.append(child) errors.append(error) logger.warning( - "failed to clean interrupted fork child %s", child.id, + "failed to clean interrupted fork child %s", + child.id, exc_info=True, ) try: @@ -836,7 +862,8 @@ def _cleanup_interrupted_fork( except BaseException as error: errors.append(error) retain_fork_failure( - failure, snapshot_id=snapshot_id, + failure, + snapshot_id=snapshot_id, created_environment_ids=tuple( child.id for child in (*children, *unreturned) ), @@ -1061,7 +1088,10 @@ def __init__(self, *args: Any, **kwargs: Any) -> None: ) if identity and not deleted: _retain_pending_creation( - error, self, kwargs.get("connection"), None, + error, + self, + kwargs.get("connection"), + None, closed=getattr(self, "_closed", False) is True, ) raise @@ -1125,15 +1155,21 @@ def _rollback_created( def _retain_pending_creation( - failure: BaseException, sandbox: Any, connection: Any, - session: BackendSession | None, *, closed: bool, + failure: BaseException, + sandbox: Any, + connection: Any, + session: BackendSession | None, + *, + closed: bool, ) -> None: """Keep a known bootstrap resource without inferring ownership from errors.""" context: Any = failure pending = tuple(getattr(failure, "_fork_pending_sessions", ())) if all(child.id != str(sandbox.id) for child in pending): - handle = session if session is not None else _Session._cleanup_handle( - sandbox, connection, closed=closed + handle = ( + session + if session is not None + else _Session._cleanup_handle(sandbox, connection, closed=closed) ) context._fork_pending_sessions = (*pending, handle) @@ -1159,6 +1195,51 @@ def __init__(self, config: BackendConfig) -> None: gateway_use_tls=config.gateway_endpoint.use_tls, verify_tls=config.verify_tls, ) + self._reset_managed_resource_probe() + + def _reset_managed_resource_probe(self) -> None: + self._managed_resource_lock = threading.Lock() + self._managed_resource_cache: tuple[float, list[NodeInfo]] | None = None + self._managed_resource_pending: Future[list[NodeInfo]] | None = None + self._managed_resource_pid = os.getpid() + + def _managed_resource_nodes(self) -> list[NodeInfo]: + # This advisory geometry view is scoped to one authenticated backend. + # Do not send one identical TCP/TLS resource query per concurrent create. + # Scheduling and node authorization remain authoritative on every create. + if self._managed_resource_pid != os.getpid(): + self._reset_managed_resource_probe() + with self._managed_resource_lock: + cached = self._managed_resource_cache + if cached is not None and time.monotonic() < cached[0]: + return cached[1] + pending = self._managed_resource_pending + leader = pending is None + if pending is None: + pending = Future() + self._managed_resource_pending = pending + if not leader: + return pending.result() + try: + client = EnvironmentClient(connection=self._connection) + try: + response = client._http.get(client._origin + RESOURCE_PATH, timeout=15) + if response.status_code >= 400: + raise client._http_error(response) + nodes = parse_resource_nodes(response.json()) + finally: + client.close() + with self._managed_resource_lock: + self._managed_resource_cache = (time.monotonic() + 1.0, nodes) + pending.set_result(nodes) + return nodes + except BaseException as error: + pending.set_exception(error) + raise + finally: + with self._managed_resource_lock: + if self._managed_resource_pending is pending: + self._managed_resource_pending = None def pty_for(self, instance_id: str) -> Any: """Attach a PTY to an existing sandbox with the backend connection.""" @@ -1326,6 +1407,68 @@ def _validate(self, spec: SandboxSpec) -> None: "Backend 'adx' requires reverse_port to equal listen_port - 1." ) + def _managed_storage( + self, spec: SandboxSpec + ) -> tuple[SandboxSpec, list[dict[str, Any]] | None]: + profile = spec.extra_config.get("managedRuntimeProfile") + if profile is None: + return spec, None + if ( + not isinstance(profile, str) + or re.fullmatch(r"[a-z][a-z0-9-]{0,62}", profile) is None + ): + raise ValueError("Invalid managed runtime profile") + key = _MANAGED_PROFILE_LABEL + profile + try: + nodes = self._managed_resource_nodes() + except Exception as error: + raise _convert_error("resolve managed runtime resources", error) from error + floors: set[int] = set() + for node in nodes: + if node.status != 0 or (spec.node_id and node.id != spec.node_id): + continue + for value in node.labels.get(key, []): + if ( + len(value) <= 14 + and value.isdecimal() + and 0 < int(value) <= (1 << 44) - 1 + ): + floors.add(int(value)) + if not floors: + raise ValueError( + f"Managed runtime profile {profile!r} " + "has no ready node resource contract" + ) + eligible = sorted( + floor + for floor in floors + if (spec.storage_mb is None or spec.storage_mb >= floor) + and (spec.storage_limit_mb == 0 or spec.storage_limit_mb >= floor) + ) + if not eligible: + raise ValueError( + f"Managed runtime profile {profile!r} " + f"requires at least {min(floors)} MiB " + "reserved and total writable storage (upper plus root)" + ) + # Reserve the platform floor before scheduling; constrain placement to + # workers with compatible geometry. Ordinary image sandboxes are unchanged. + budget = spec.storage_mb if spec.storage_mb is not None else eligible[0] + spec = replace(spec, storage_mb=budget) + return spec, [ + { + "kind": 0, + "affinity": 2, + "labelOps": [ + { + "type": 0, + "labelKey": key, + "labelValues": [str(v) for v in eligible if v <= budget], + } + ], + } + ] + def create(self, spec: SandboxSpec) -> BackendSession: self._validate(spec) create_timeout = resolve_create_timeout( @@ -1333,6 +1476,7 @@ def create(self, spec: SandboxSpec) -> BackendSession: ) _log_creation_budgets("create", spec, create_timeout) create_started = time.monotonic() + spec, affinities = self._managed_storage(spec) rootfs = None if spec.rootfs is not None: rootfs = adx_environment.S3Config( @@ -1391,6 +1535,8 @@ def create(self, spec: SandboxSpec) -> BackendSession: ), connection=self._connection, ) + if affinities is not None: + create_args["schedule_affinities"] = affinities try: sandbox = _OwnedSandbox(**create_args) except UnsupportedBackendFeatureError: @@ -1457,6 +1603,7 @@ def _restore_snapshot(spec: SnapshotRestoreSpec, connection: Any) -> BackendSess cwd=spec.command_cwd, port_forwardings=list(spec.port_forwardings), mounts=[_native_mount(mount) for mount in spec.mounts], + extra_config=dict(spec.extra_config), upstream=tunnel.target if tunnel else None, proxy_port=tunnel.listen_port if tunnel else _DEFAULT_LISTEN_PORT, tunnel_connect_timeout=tunnel.connect_timeout if tunnel else None, @@ -1473,7 +1620,7 @@ def _restore_snapshot(spec: SnapshotRestoreSpec, connection: Any) -> BackendSess port_forward_mode="tls", ), connection=connection, - ) + ) except UnsupportedBackendFeatureError: raise except Exception as error: diff --git a/sdk/python/akernel_sdk/_backends/base.py b/sdk/python/akernel_sdk/_backends/base.py index b7a8953e..cd93f828 100644 --- a/sdk/python/akernel_sdk/_backends/base.py +++ b/sdk/python/akernel_sdk/_backends/base.py @@ -104,6 +104,7 @@ class SnapshotRestoreSpec: failover: bool node_id: str | None network_policy: NetworkPolicy | None + extra_config: Mapping[str, object] = field(default_factory=dict) xpu: str | None = None storage_mb: int | None = None inherit_entrypoint: bool = False @@ -124,6 +125,7 @@ class SandboxConnectSpec: schedule_timeout: int = 30 network_policy: NetworkPolicy | None = None mounts: tuple[Mount, ...] = () + extra_config: Mapping[str, object] = field(default_factory=dict) xpu: str | None = None storage_mb: int | None = None inherit_entrypoint: bool = False diff --git a/sdk/python/akernel_sdk/_backends/errors.py b/sdk/python/akernel_sdk/_backends/errors.py index 715c3fd4..29796779 100644 --- a/sdk/python/akernel_sdk/_backends/errors.py +++ b/sdk/python/akernel_sdk/_backends/errors.py @@ -14,7 +14,10 @@ """Stable errors raised at the AKernel backend boundary.""" -from typing import Any +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from ..types import CommandResult class InvalidBackendError(ValueError): @@ -36,6 +39,10 @@ class BackendOperationError(RuntimeError): instructions; callers must not infer execution state from the message or an HTTP status. The original backend error remains available as the cause. ``cleanup_errors`` retains rollback failures without replacing that cause. + ``first_error`` retains the native client's earliest failed attempt without + changing the final error's retry/outcome guidance. + ``command_result`` retains available command output on an execution/wait + timeout without implying that the remote command reached a terminal exit. """ def __init__( @@ -55,6 +62,8 @@ def __init__( request_forwarded: bool | None = None, cleanup_errors: tuple[BaseException, ...] = (), command_id: str | None = None, + first_error: BaseException | None = None, + command_result: "CommandResult | None" = None, ) -> None: super().__init__(message) self.code = code @@ -70,6 +79,8 @@ def __init__( self.request_forwarded = request_forwarded self.cleanup_errors = cleanup_errors self.command_id = command_id + self.first_error = first_error + self.command_result = command_result class AuthenticationException(BackendOperationError): diff --git a/sdk/python/akernel_sdk/_compose.py b/sdk/python/akernel_sdk/_compose.py new file mode 100644 index 00000000..a9936f37 --- /dev/null +++ b/sdk/python/akernel_sdk/_compose.py @@ -0,0 +1,945 @@ +"""Compose workloads inside a Sandbox; independent of any evaluation harness.""" + +from __future__ import annotations + +import ipaddress +import json +import re +import shlex +import shutil +import tempfile +import time +from dataclasses import replace +from pathlib import Path +from threading import Event +from typing import TYPE_CHECKING, Any +from urllib.parse import urlsplit +from uuid import uuid4 + +import yaml + +from ._backends.errors import ( + BackendOperationError, + TimeoutException, + retain_failure_context, +) +from ._docker_runtime import DockerRuntime +from ._image_plan import ImageMountPlan +from ._image_resolver import prepare_compose_images +from .types import CommandResult, HttpReverseTunnel, Mount + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + + from ._service_process import ServiceProcessHandle + from .commands import CommandHandle + from .sandbox import Sandbox + + +class ComposeOperationError(BackendOperationError): + """Compose returned a failure, retaining phase and command output.""" + + def __init__(self, phase: str, result: CommandResult) -> None: + super().__init__( + f"Compose {phase} failed (exit_code={result.exit_code})", + code="COMPOSE_FAILED", + outcome="unknown" if result.exit_code is None else "terminal", + ) + self.phase = phase + self.result = result + self.stdout, self.stderr = result.stdout, result.stderr + self.exit_code = result.exit_code + + +def _services(contents: Sequence[str]) -> dict[str, Any]: + services: dict[str, Any] = {} + for content in contents: + document = yaml.safe_load(content) or {} + for name, config in document.get("services", {}).items(): + services.setdefault(name, {}).update(config or {}) + if not services: + raise ValueError("Compose project requires at least one service") + return services + + +def host_network_overlay( + services: Mapping[str, Any], *, prebuilt: Sequence[str] = () +) -> dict[str, Any]: + """Explicit runsc host-network compatibility profile, preserving task files.""" + if any(config.get("networks") for config in services.values()): + raise ValueError("Host-network Compose does not support explicit networks") + hostnames = { + name: config.get("hostname") or name for name, config in services.items() + } + aliases = list(dict.fromkeys([*services, *hostnames.values()])) + return { + "services": { + name: { + "network_mode": "host", + "hostname": hostnames[name], + "extra_hosts": [f"{alias}:127.0.0.1" for alias in aliases], + **( + {"build": {"network": "host"}} + if config.get("build") is not None and name not in prebuilt + else {} + ), + } + for name, config in services.items() + } + } + + +class ComposeProject: + """Prepared client descriptor and remote Compose operations. + + prepare() freezes the project context and Compose bytes before resolving + image inputs. No server is changed until start_runtime(). Restore with + attach(): it performs no upload, build, up, or daemon startup. The caller + owns the outer Sandbox and its deletion after a partial startup failure. + """ + + _descriptor: dict[str, Any] + _runtime: DockerRuntime | None + _context: Path | None + _output: Path | None + _contents: list[str] + _sandbox: Sandbox | None + _startup_attempted: bool + + @classmethod + def prepare( + cls, + directory: str | Path, + files: Sequence[str | Path] = (), + *, + project_name: str = "akernel", + base: Mapping[str, Any] | None = None, + overrides: Sequence[Mapping[str, Any]] = (), + runtime: DockerRuntime | None = None, + auto_images: bool = False, + output_dir: str | Path | None = None, + environment: Mapping[str, str] | None = None, + network: str = "bridge", + buildkit: bool = True, + prebuilt_images: Mapping[str, str] | None = None, + remote_directory: str = "/opt/akernel-compose/project", + directories: Mapping[str, int] | None = None, + ) -> ComposeProject: + if re.fullmatch(r"[a-z0-9][a-z0-9_-]*", project_name) is None: + raise ValueError("Invalid Compose project_name") + if auto_images and not buildkit: + raise ValueError("Automatic images require BuildKit") + if network not in {"host", "bridge"}: + raise ValueError("Compose network must be host or bridge") + if not remote_directory.startswith("/") or ".." in Path(remote_directory).parts: + raise ValueError("remote_directory must be an absolute normalized path") + for path, mode in (directories or {}).items(): + if ( + not path.startswith("/") + or ".." in Path(path).parts + or isinstance(mode, bool) + or not isinstance(mode, int) + or not 0 <= mode <= 0o777 + ): + raise ValueError("Invalid bind directory initialization") + source = Path(directory).resolve(strict=True) + output = ( + Path(output_dir).resolve() + if output_dir + else Path(tempfile.mkdtemp(prefix="akernel-compose-")) + ) + if output == source or source in output.parents: + raise ValueError("Compose output_dir must be outside the source context") + output.mkdir(parents=True, exist_ok=True, mode=0o700) + context = output / "context" + if context.exists(): + raise ValueError("Compose output_dir already contains a prepared context") + contents: list[str] = [] + names: list[str] = [] + if base is not None: + names.append("base.json") + contents.append(json.dumps(base)) + for index, file in enumerate(files): + names.append(f"source-{index}.yaml") + contents.append(Path(file).read_text()) + for index, override in enumerate(overrides): + names.append(f"override-{index}.json") + contents.append(json.dumps(override)) + runtime = runtime or DockerRuntime() + # Managed inputs are interpreted only by the bundled official Compose + # CLI. safe_load cannot implement native merge tags, include or profiles. + services = {} if runtime.profile is not None else _services(contents) + # Preserve original definitions; compatibility policy is an extra file. + selected = set(prebuilt_images or {}) & services.keys() + if network == "host" and runtime.profile is None: + names.append("host.json") + contents.append( + json.dumps(host_network_overlay(services, prebuilt=sorted(selected))) + ) + # A private local copy makes discovery and upload see the same context. + shutil.copytree(source, context, symlinks=True) + local_files = [] + for name, content in zip(names, contents, strict=True): + local_path = output / name + local_path.write_text(content) + local_files.append(local_path) + if runtime.profile is not None and (auto_images or prebuilt_images): + raise ValueError( + "Managed runtime profile owns image discovery and publication" + ) + if auto_images: + if runtime.bridge_binary is None: + raise ValueError("Automatic images require a matching bridge binary") + images = prepare_compose_images( + context, + local_files, + output_dir=output / "images", + bridge_binary=runtime.bridge_binary, + env=environment, + ) + runtime = replace(runtime, images=images) + if selected: + lines = ["services:"] + for name in sorted(selected): + lines += [ + f" {json.dumps(name)}:", + " build: !reset null", + f" image: {json.dumps((prebuilt_images or {})[name])}", + ] + names.append("prebuilt.yaml") + contents.append("\n".join(lines) + "\n") + result = cls.__new__(cls) + result._runtime = runtime + result._context = context + result._contents = contents + result._output = output + result._sandbox = None + result._startup_attempted = False + result._descriptor = { + "schema_version": 1, + "project_name": project_name, + "directory": remote_directory, + "files": [remote_directory + ".config/" + n for n in names], + "services": list(services), + "network": network, + "buildkit": buildkit, + "prebuilt_only": runtime.images is not None and not auto_images, + "automatic_images": auto_images, + "environment": dict(environment or {}), + "directories": dict(directories or {}), + "managed_runtime_profile": runtime.profile, + } + options = runtime.sandbox_options() + result._descriptor["sandbox_options"] = { + **options, + "mounts": [ + {"target": m.target, "image_url": m.image_url} + for m in options.get("mounts", []) + ], + } + return result + + @classmethod + def attach(cls, sandbox: Sandbox, descriptor: Mapping[str, Any]) -> ComposeProject: + """Rebind captured remote state only, with no fresh-boot operations.""" + data = json.loads(json.dumps(descriptor, allow_nan=False)) + if data.get("schema_version") != 1 or not isinstance( + data.get("services"), list + ): + raise ValueError("Invalid Compose project descriptor") + if re.fullmatch(r"[a-z0-9][a-z0-9_-]*", data.get("project_name", "")) is None: + raise ValueError("Invalid Compose project name") + paths = [data.get("directory"), *data.get("files", [])] + if not all( + isinstance(p, str) and p.startswith("/") and ".." not in Path(p).parts + for p in paths + ): + raise ValueError("Invalid Compose project paths") + result = cls.__new__(cls) + result._descriptor = data + result._sandbox = sandbox + result._startup_attempted = True + result._output = None + result._runtime = None + result._context = None + result._contents = [] + return result + + @property + def descriptor(self) -> dict[str, Any]: + return json.loads(json.dumps(self._descriptor)) + + @classmethod + def bind_runtime( + cls, sandbox: Sandbox, descriptor: Mapping[str, Any] + ) -> ComposeProject: + """Bind a newly prepared managed runtime's opaque Compose descriptor. + + Runtime preparation, builds, image registration and daemon startup must + already have succeeded. This method performs no remote operation. It + permits initial service tunnel creation. Use attach(), never this + method, after checkpoint restore: attach only reuses captured tunnels. + The operator owns runtime-profile authorization and artifact leases. + """ + result = cls.attach(sandbox, descriptor) + result._runtime = DockerRuntime() + return result + + @property + def images(self) -> ImageMountPlan | None: + return self._runtime.images if self._runtime else None + + def sandbox_options(self) -> dict[str, Any]: + options = json.loads(json.dumps(self._descriptor["sandbox_options"])) + options["mounts"] = [Mount(**value) for value in options.get("mounts", [])] + return options + + def _require_sandbox(self) -> Sandbox: + if self._sandbox is None: + raise RuntimeError("Compose project is not bound to a Sandbox") + return self._sandbox + + def start_runtime(self, sandbox: Sandbox) -> None: + if ( + self._sandbox is not None + or self._runtime is None + or self._startup_attempted + ): + raise RuntimeError("Compose runtime startup is fresh-only") + self._startup_attempted = True + self._runtime.start(sandbox, network=self._descriptor["network"]) + self._sandbox = sandbox + directory = self._descriptor["directory"] + if self._descriptor.get("managed_runtime_profile"): + from ._managed_build import upload_context + + if self._context is None: + raise RuntimeError("Managed project context is unavailable") + upload_context(sandbox, self._context, directory) + else: + sandbox.files.copy_from_local(str(self._context), directory) + sandbox.files.make_dir(directory + ".config") + for remote, content in zip( + self._descriptor["files"], self._contents, strict=True + ): + sandbox.files.write(remote, content) + for path, mode in self._descriptor["directories"].items(): + sandbox.commands.run( + shlex.join(["install", "-d", "-m", oct(mode)[2:], path]), + timeout=20, + check=True, + ) + if self._descriptor.get("managed_runtime_profile"): + sandbox.files.write( + directory + ".config/descriptor.json", json.dumps(self._descriptor) + ) + result = sandbox.commands.run( + shlex.join( + [ + "python3", + "/usr/lib/akernel/compose.py", + "describe", + "--descriptor", + directory + ".config/descriptor.json", + ] + ), + timeout=60, + check=True, + ) + services = json.loads(result.stdout) + if ( + not isinstance(services, list) + or not services + or any(not isinstance(name, str) or not name for name in services) + ): + raise ValueError("Runtime Compose returned an invalid service list") + self._descriptor["services"] = services + sandbox.files.write( + directory + ".config/descriptor.json", json.dumps(self._descriptor) + ) + + def reconnect_network(self) -> None: + """Rebind managed egress rules after restoring to a new outer TAP. + + Call after attach and before resuming work. Does not start a daemon, + build/up a project, replace containers or recreate tunnel listeners. + Only SDK-owned IPv4 forwarding/NAT chains can change. Inner network + identities and existing connections inside the sandbox remain intact; + connections to external peers do not gain IP-migration guarantees. + """ + if self._descriptor.get("managed_runtime_profile"): + self._require_sandbox().commands.run( + "python3 /usr/lib/akernel/compose.py reconnect", timeout=20, check=True + ) + return + DockerRuntime.reconnect_network( + self._require_sandbox(), network=self._descriptor["network"] + ) + + def command(self, args: Sequence[str]) -> str: + if not args: + raise ValueError("Compose subcommand is required") + parts = [ + "docker", + "compose", + "-p", + self._descriptor["project_name"], + "--project-directory", + self._descriptor["directory"], + ] + for file in self._descriptor["files"]: + parts += ["-f", file] + prefix = ( + f"DOCKER_BUILDKIT={int(self._descriptor['buildkit'])} " + if args[0] == "build" + else "" + ) + # Interpolation gets only explicit project variables, never host secrets. + if self._descriptor["environment"]: + prefix += ( + "env " + + shlex.join( + [f"{k}={v}" for k, v in self._descriptor["environment"].items()] + ) + + " " + ) + return prefix + shlex.join([*parts, *args]) + + def run( + self, + args: Sequence[str], + *, + timeout: int = 60, + check: bool = False, + cancellation: Event | None = None, + ) -> CommandResult: + result = self._execute( + self.command(args), timeout=timeout, cancellation=cancellation + ) + if check and result.exit_code != 0: + raise ComposeOperationError(args[0], result) + return result + + def _execute( + self, command: str, *, timeout: int, cancellation: Event | None = None + ) -> CommandResult: + sandbox = self._require_sandbox() + if cancellation is None and timeout != 0: + result = sandbox.commands.run(command, cwd="/", timeout=timeout) + else: + # A zero execution budget means no deadline. Waits remain bounded + # so an async caller can cancel without abandoning remote work. + handle = sandbox.commands.run(command, cwd="/", background=True) + deadline = time.monotonic() + timeout if timeout > 0 else None + while True: + cancelled = cancellation is not None and cancellation.is_set() + expired = deadline is not None and time.monotonic() >= deadline + if cancelled or expired: + failure = TimeoutException( + "Compose command cancelled" + if cancelled + else "Compose command timed out", + command_id=handle.command_id, + outcome="unknown", + ) + try: + handle.kill() + except BaseException as cleanup_error: + retain_failure_context(failure, sandbox.id, cleanup_error) + raise failure + result = handle.wait(timeout=1) + if result.exit_code is not None: + break + return result + + def _operation(self, args: Sequence[str], timeout: int) -> CommandResult: + try: + result = self.run(args, timeout=timeout) + except BackendOperationError as error: + # Unknown outcomes are never retried by the Compose layer. + self._retain_partial_log(args[0] + ".log", error) + raise + if self._output is not None: + (self._output / (args[0] + ".log")).write_text( + result.stdout + result.stderr + ) + if result.exit_code != 0: + if args[0] == "up": + try: + logs = self.run(["logs", "--no-color", "--tail", "200"], timeout=30) + if self._output is not None: + (self._output / "services.log").write_text( + logs.stdout + logs.stderr + ) + except Exception: + pass + raise ComposeOperationError(args[0], result) + return result + + def _retain_partial_log(self, name: str, error: BackendOperationError) -> None: + if self._output is None or error.command_result is None: + return + try: + partial = error.command_result + (self._output / name).write_text(partial.stdout + partial.stderr) + except OSError as logging_error: + add_note = getattr(error, "add_note", None) + if callable(add_note): + add_note( + "Could not preserve partial operation log: " + + type(logging_error).__name__ + ) + + def build(self, *, no_cache: bool = False, timeout: int = 600) -> None: + if self._descriptor.get("managed_runtime_profile"): + command = shlex.join( + [ + "python3", + "/usr/lib/akernel/compose.py", + "build", + "--descriptor", + self._descriptor["directory"] + ".config/descriptor.json", + *(["--no-cache"] if no_cache else []), + ] + ) + try: + result = self._execute(command, timeout=timeout) + except BackendOperationError as error: + self._retain_partial_log("build.log", error) + raise + if self._output is not None: + (self._output / "build.log").write_text(result.stdout + result.stderr) + if result.exit_code != 0: + raise ComposeOperationError("build", result) + return + if not self._descriptor["prebuilt_only"]: + self._operation(["build", *(["--no-cache"] if no_cache else [])], timeout) + + def pull(self, *, timeout: int = 600) -> None: + """Prepare service images, verifying existing ROFS inputs in auto mode. + + Automatic discovery already resolved and mounted all external inputs. + Native Compose pull can still unpack digest references despite its + missing policy, so use the checked mounted images rather than pull again. + """ + if self._descriptor.get("managed_runtime_profile"): + result = self._execute( + shlex.join( + [ + "python3", + "/usr/lib/akernel/compose.py", + "prepare", + "--descriptor", + self._descriptor["directory"] + ".config/descriptor.json", + ] + ), + timeout=timeout, + ) + if self._output is not None: + (self._output / "pull.log").write_text(result.stdout + result.stderr) + if result.exit_code != 0: + raise ComposeOperationError("prepare", result) + attached = self._require_sandbox().commands.run( + shlex.join( + ["cat", self._descriptor["directory"] + ".build/attached.json"] + ), + timeout=20, + check=True, + ) + self._descriptor = ComposeProject.attach( + self._require_sandbox(), json.loads(attached.stdout) + ).descriptor + self._descriptor["prebuilt_only"] = True + self._descriptor["automatic_images"] = False + return + if self._descriptor["automatic_images"]: + self._verify_images("pull", timeout) + elif not self._descriptor["prebuilt_only"]: + self._operation(["pull", "--ignore-buildable"], timeout) + + def _verify_images(self, phase: str, timeout: int) -> None: + result = self._execute( + "python3 /opt/akernel-docker-runtime/start.py --verify " + "--plan /opt/akernel-docker-runtime/metadata/plan.json", + timeout=timeout, + ) + if self._output is not None: + (self._output / (phase + "-images.log")).write_text( + result.stdout + result.stderr + ) + if phase == "pull": + (self._output / "pull.log").write_text(result.stdout + result.stderr) + if result.exit_code != 0: + raise ComposeOperationError(phase, result) + + def up(self, *, wait_timeout: int = 600, timeout: int = 600) -> None: + if self._descriptor["automatic_images"]: + self._verify_images("up", timeout) + self._operation( + [ + "up", + "-d", + "--wait", + "--wait-timeout", + str(wait_timeout), + *( + ["--no-build", "--pull", "never"] + if self._descriptor["prebuilt_only"] + else ["--pull", "never"] + if self._descriptor["automatic_images"] + else [] + ), + ], + timeout, + ) + + def ps(self) -> list[dict[str, Any]]: + output = self.run(["ps", "-a", "--format", "json"], check=True).stdout + rows = [] + for line in output.splitlines(): + parsed = json.loads(line) + rows.extend(parsed if isinstance(parsed, list) else [parsed]) + return rows + + def logs(self, *, tail: int = 200) -> str: + return self.run(["logs", "--no-color", "--tail", str(tail)], check=True).stdout + + def down(self, *, timeout: int = 60) -> None: + self.run(["down"], timeout=timeout, check=True) + + def service(self, name: str) -> ServiceHandle: + if ( + self._descriptor.get("managed_runtime_profile") + and not self._descriptor["services"] + ): + raise RuntimeError( + "Managed Compose services are resolved by start_runtime()" + ) + if name not in self._descriptor["services"]: + raise ValueError(f"Unknown Compose service: {name}") + return ServiceHandle(self, name) + + +class ServiceHandle: + """Operations in one Compose service; no hardcoded main or log paths.""" + + def __init__(self, project: ComposeProject, name: str) -> None: + self.project, self.name = project, name + + def identity(self) -> dict[str, str]: + ids = self.project.run(["ps", "-a", "-q", self.name], check=True).stdout.split() + if len(ids) != 1: + raise BackendOperationError( + "Service must have exactly one container", code="SERVICE_UNAVAILABLE" + ) + result = self.project._require_sandbox().commands.run( + shlex.join(["docker", "inspect", ids[0]]), timeout=20, check=True + ) + data = json.loads(result.stdout)[0] + return {"container_id": data["Id"], "started_at": data["State"]["StartedAt"]} + + def expose_tunnel(self, tunnel: HttpReverseTunnel) -> str: + """Return this service's reachable URL for an outer reverse tunnel. + + Host-network services use the existing loopback listener. For bridge + networks, a private TCP relay binds only to an attached gateway and + accepts only this service's IP. HTTP, credentials and SSE pass through + unchanged. The daemon and descriptor survive whole-sandbox snapshots; + attach never restarts a relay. Requires IPv4 on one attached network. + An ambiguous launch is never retried automatically. + """ + if not isinstance(tunnel, HttpReverseTunnel): + raise TypeError("tunnel must be an HttpReverseTunnel") + identity = self.identity() + key = f"{self.name}:{tunnel.listen_port}" + endpoints = self.project._descriptor.setdefault("tunnel_endpoints", {}) + if key in endpoints: + record = endpoints[key] + if any(record[k] != identity[k] for k in identity): + raise BackendOperationError( + "Tunnel service was replaced or restarted", + code="SERVICE_TUNNEL_STALE", + outcome="not_started", + ) + if "url" not in record: + raise BackendOperationError( + "Tunnel launch outcome unknown; do not resubmit", + code="SERVICE_TUNNEL_UNKNOWN", + outcome="unknown", + ) + return str(record["url"]) + if self.project._runtime is None: + raise RuntimeError("Restored projects can only attach existing tunnels") + sandbox = self.project._require_sandbox() + if self.project._descriptor.get("managed_runtime_profile"): + operation_id = uuid4().hex + endpoints[key] = {**identity, "operation_id": operation_id} + result = sandbox.commands.run( + shlex.join( + [ + "python3", + "/usr/lib/akernel/compose.py", + "expose", + "--container", + identity["container_id"], + "--port", + str(tunnel.listen_port), + ] + ), + timeout=30, + check=True, + ) + value = json.loads(result.stdout) + url = urlsplit(value["url"]) + if ( + url.scheme != "http" + or not url.hostname + or not url.port + or url.username + or url.query + or url.path + ): + raise ValueError("Runtime returned an invalid service tunnel endpoint") + endpoints[key]["url"] = value["url"] + return str(value["url"]) + result = sandbox.commands.run( + shlex.join(["docker", "inspect", identity["container_id"]]), + timeout=20, + check=True, + ) + data = json.loads(result.stdout)[0] + # Compose network_mode: service:api becomes container: in Docker's + # native inspect result. Resolve the namespace owner, keeping the task + # network definition and the original service's identity unchanged. + visited: set[str] = set() + for _ in range(8): + mode = data["HostConfig"]["NetworkMode"] + if not mode.startswith("container:"): + break + owner = mode.removeprefix("container:") + if not owner or owner in visited: + raise ValueError("Cyclic or missing service network namespace owner") + visited.add(owner) + inspected = sandbox.commands.run( + shlex.join(["docker", "inspect", owner]), timeout=20, check=True + ) + data = json.loads(inspected.stdout)[0] + if data.get("State", {}).get("Running") is False: + raise ValueError("Service network namespace owner is not running") + else: + raise ValueError("Service network namespace nesting exceeds the limit") + if data["HostConfig"]["NetworkMode"] == "host": + endpoints[key] = {**identity, "url": tunnel.url} + return tunnel.url + networks = data["NetworkSettings"].get("Networks") or {} + connected = [] + for name, value in sorted(networks.items()): + if not value.get("IPAddress"): + continue + gateway = value.get("Gateway") + if not gateway: + # Internal Docker networks intentionally omit a default route; + # their local bridge gateway can still host a controlled tunnel. + inspect = sandbox.commands.run( + shlex.join(["docker", "network", "inspect", name]), + timeout=20, + check=True, + ) + config = json.loads(inspect.stdout)[0]["IPAM"]["Config"] + gateway = next( + ( + v["Gateway"] + for v in config + if v.get("Gateway") and ":" not in v["Gateway"] + ), + None, + ) + if gateway: + connected.append({"Gateway": gateway, "IPAddress": value["IPAddress"]}) + if not connected: + raise ValueError("Service tunnel requires an attached IPv4 bridge network") + network = connected[0] + address = str(ipaddress.IPv4Address(network["Gateway"])) + peer = str(ipaddress.IPv4Address(network["IPAddress"])) + if ( + ipaddress.IPv4Address(address).is_unspecified + or ipaddress.IPv4Address(address).is_loopback + ): + raise ValueError("Service tunnel requires a non-loopback bridge gateway") + operation_id = uuid4().hex + # Reserve before dispatch so a lost reply cannot create a second relay. + record = {**identity, "operation_id": operation_id} + endpoints[key] = record + script = Path(__file__).with_name("_docker_network_guest.py").read_text() + result = sandbox.commands.run( + shlex.join( + [ + "python3", + "-c", + script, + "expose", + "/opt/akernel-docker-runtime/tunnels/" + operation_id, + json.dumps( + { + "address": address, + "allowed": [peer], + "target_port": tunnel.listen_port, + } + ), + ] + ), + timeout=20, + check=True, + ) + value = json.loads(result.stdout) + url = urlsplit(value["url"]) + if ( + url.scheme != "http" + or url.hostname != address + or not url.port + or url.username + or url.path + or url.query + or url.fragment + ): + raise ValueError("Runtime returned an invalid service tunnel endpoint") + record["url"] = value["url"] + return str(record["url"]) + + def _args( + self, + command: str, + *, + cwd: str | None, + envs: Mapping[str, str] | None, + user: str | int | None, + shell: str, + login: bool = False, + ) -> list[str]: + args = ["exec", "-T"] + if cwd: + args += ["-w", cwd] + if user is not None: + args += ["-u", str(user)] + for key, value in (envs or {}).items(): + args += ["-e", f"{key}={value}"] + return [*args, self.name, shell, "-lc" if login else "-c", command] + + def exec( + self, + command: str, + *, + cwd: str | None = None, + envs: Mapping[str, str] | None = None, + user: str | int | None = None, + shell: str = "/bin/sh", + login: bool = False, + timeout: int = 60, + cancellation: Event | None = None, + ) -> CommandResult: + return self.project.run( + self._args( + command, cwd=cwd, envs=envs, user=user, shell=shell, login=login + ), + timeout=timeout, + cancellation=cancellation, + ) + + def _copy(self, source: str, target: str, timeout: int) -> None: + self.project.run(["cp", source, target], timeout=timeout, check=True) + + def upload( + self, + source: str | Path, + target: str, + *, + directory: bool = False, + timeout: int = 120, + ) -> None: + sandbox = self.project._require_sandbox() + temp = "/tmp/akernel-compose-" + uuid4().hex + failure: BaseException | None = None + try: + sandbox.files.copy_from_local(str(source), temp) + self._copy( + temp + ("/." if directory else ""), f"{self.name}:{target}", timeout + ) + except BaseException as error: + failure = error + raise + finally: + try: + sandbox.files.remove(temp) + except Exception as cleanup_error: + if failure is None: + raise + retain_failure_context(failure, sandbox.id, cleanup_error) + + def download( + self, + source: str, + target: str | Path, + *, + directory: bool = False, + timeout: int = 120, + ) -> None: + sandbox = self.project._require_sandbox() + temp = "/tmp/akernel-compose-" + uuid4().hex + failure: BaseException | None = None + try: + if directory: + sandbox.files.make_dir(temp) + self._copy( + f"{self.name}:{source}" + ("/." if directory else ""), temp, timeout + ) + Path(target).parent.mkdir(parents=True, exist_ok=True) + sandbox.files.copy_to_local(temp, str(target)) + except BaseException as error: + failure = error + raise + finally: + try: + sandbox.files.remove(temp) + except Exception as cleanup_error: + if failure is None: + raise + retain_failure_context(failure, sandbox.id, cleanup_error) + + def stop(self, *, timeout: int = 60) -> None: + self.project.run(["stop", self.name], timeout=timeout, check=True) + + def start_process( + self, + command: str, + *, + cwd: str | None = None, + envs: Mapping[str, str] | None = None, + user: str | int | None = None, + shell: str = "/bin/bash", + ) -> ServiceProcessHandle: + from ._service_process import ServiceProcessHandle + + return ServiceProcessHandle.start( + self, command, cwd=cwd, envs=envs, user=user, shell=shell + ) + + def start_command( + self, + command: str, + *, + cwd: str | None = None, + envs: Mapping[str, str] | None = None, + user: str | int | None = None, + shell: str = "/bin/sh", + ) -> CommandHandle: + """Start native Compose exec through the sandbox's command API. + + No monitor or Python interpreter is installed in the service. The + returned PID belongs to the outer Docker CLI; its stable command_id + can be reconnected with restored_sandbox.commands.connect() after a + whole-sandbox restore. This does not relaunch the service command. + """ + args = self._args(command, cwd=cwd, envs=envs, user=user, shell=shell) + return self.project._require_sandbox().commands.run( + self.project.command(args), cwd="/", background=True + ) diff --git a/sdk/python/akernel_sdk/_docker_network_guest.py b/sdk/python/akernel_sdk/_docker_network_guest.py new file mode 100644 index 00000000..8120892f --- /dev/null +++ b/sdk/python/akernel_sdk/_docker_network_guest.py @@ -0,0 +1,161 @@ +"""Dependency-free networking helper, executed in the outer sandbox only. + +The runtime owns forwarding/NAT; Docker owns bridges, IPAM, DNS and its +userland port proxies. No rules or listeners are installed on the real host. +""" + +import ipaddress +import json +import os +import shutil +import socket +import subprocess +import sys +import threading +from pathlib import Path + +STATE = Path("/opt/akernel-docker-runtime/network.json") + + +def configure(*, reconnect=False): + route = subprocess.check_output( + ["ip", "-4", "route", "get", "1.1.1.1"], text=True + ).split() + device = route[route.index("dev") + 1] + source = str(ipaddress.IPv4Address(route[route.index("src") + 1])) + executable = shutil.which("iptables-legacy") + restore = shutil.which("iptables-legacy-restore") + if executable is None or restore is None: + raise RuntimeError( + "Managed bridge networking requires iptables-legacy " + "and iptables-legacy-restore" + ) + record = {"uplink": device, "source": source, "iptables": executable} + if reconnect: + previous = json.loads(STATE.read_text()) + if previous == record: + return record + else: + # Docker's embedded DNS still programs per-container DNAT rules with + # --iptables=false. Put legacy tools in only the daemon's PATH. + directory = STATE.parent / "network-bin" + directory.mkdir(parents=True, exist_ok=True) + for name in ("iptables", "ip6tables"): + legacy = shutil.which(name + "-legacy") + if legacy is not None: + (directory / name).symlink_to(legacy) + Path("/proc/sys/net/ipv4/ip_forward").write_text("1\n") + subprocess.run([executable, "-w", "-P", "FORWARD", "DROP"], check=True) + for table, hook, chain in ( + ("filter", "FORWARD", "AKERNEL_DIND_FWD"), + ("nat", "POSTROUTING", "AKERNEL_DIND_NAT"), + ): + prefix = [executable, "-w", "-t", table] + subprocess.run([*prefix, "-N", chain], check=True) + subprocess.run([*prefix, "-A", hook, "-j", chain], check=True) + # A restored sandbox has a new outer TAP name/address. Replace only owned + # chains, atomically per table, keeping Docker's bridges, DNS, proxies and + # existing inner connections alive. Never restart the daemon or Compose. + rules = ( + "*filter\n:AKERNEL_DIND_FWD - [0:0]\n" + f"-A AKERNEL_DIND_FWD -i {device} -j ACCEPT\n" + f"-A AKERNEL_DIND_FWD -o {device} -j ACCEPT\nCOMMIT\n" + "*nat\n:AKERNEL_DIND_NAT - [0:0]\n" + f"-A AKERNEL_DIND_NAT -o {device} -p tcp -j SNAT --to-source {source}\n" + f"-A AKERNEL_DIND_NAT -o {device} -p udp -j SNAT --to-source {source}\n" + "COMMIT\n" + ) + subprocess.run([restore, "--wait", "--noflush"], input=rules, text=True, check=True) + temporary = STATE.with_suffix(".tmp") + temporary.write_text(json.dumps(record)) + temporary.replace(STATE) + return record + + +def forward(peer, target): + """Relay bytes and half-closes without interpreting HTTP, auth or SSE.""" + try: + upstream = socket.create_connection(target, timeout=10) + except OSError: + peer.close() + return + + def pump(source, destination): + try: + while chunk := source.recv(65536): + destination.sendall(chunk) + except OSError: + pass + finally: + try: + destination.shutdown(socket.SHUT_WR) + except OSError: + pass + + with peer, upstream: + peer.settimeout(None) + upstream.settimeout(None) + other = threading.Thread(target=pump, args=(upstream, peer), daemon=True) + other.start() + pump(peer, upstream) + other.join() + + +def serve(listener, target, allowed): + while True: + try: + peer, address = listener.accept() + except OSError: + return + if address[0] not in allowed: + peer.close() + continue + threading.Thread(target=forward, args=(peer, target), daemon=True).start() + + +def expose(root, config): + # The caller reserves this identity before dispatch. Never replay an + # ambiguous start, or expose a wildcard/unrestricted outer listener. + root.mkdir(mode=0o700, parents=True, exist_ok=False) + address = str(ipaddress.IPv4Address(config["address"])) + allowed = {str(ipaddress.IPv4Address(v)) for v in config["allowed"]} + if not allowed or address in {"0.0.0.0", "127.0.0.1"}: + raise ValueError("A service gateway and explicit peer addresses are required") + port = config["target_port"] + if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535: + raise ValueError("Invalid loopback tunnel port") + listener = socket.socket() + listener.bind((address, 0)) + listener.listen(64) + record = {"url": f"http://{address}:{listener.getsockname()[1]}"} + pid = os.fork() + if pid == 0: + os.setsid() + # The listener must outlive the SDK command's stdin/stdout transport. + if os.fork() != 0: + os._exit(0) + descriptor = os.open("/dev/null", os.O_RDWR) + for number in (0, 1, 2): + os.dup2(descriptor, number) + if descriptor > 2: + os.close(descriptor) + serve(listener, ("127.0.0.1", port), allowed) + os._exit(0) + listener.close() + _, status = os.waitpid(pid, 0) + if status != 0: + raise RuntimeError("Tunnel relay could not detach; launch outcome unknown") + (root / "endpoint.json").write_text(json.dumps(record)) + return record + + +if __name__ == "__main__": + if sys.argv[1] == "bootstrap": + result = configure() + elif sys.argv[1] == "reconnect": + result = configure(reconnect=True) + elif sys.argv[1] == "expose": + result = expose(Path(sys.argv[2]), json.loads(sys.argv[3])) + else: + raise ValueError("Unknown Docker networking operation") + print(json.dumps(result)) diff --git a/sdk/python/akernel_sdk/_docker_runtime.py b/sdk/python/akernel_sdk/_docker_runtime.py new file mode 100644 index 00000000..b8103501 --- /dev/null +++ b/sdk/python/akernel_sdk/_docker_runtime.py @@ -0,0 +1,253 @@ +"""Managed Docker runtime using sandboxd's shared ROFS and private upper mounts.""" + +from __future__ import annotations + +import json +import re +import shlex +import time +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING, Any, ClassVar + +from ._image_plan import ImageMountPlan +from .types import Mount + +if TYPE_CHECKING: + from .sandbox import Sandbox + + +_PREPARE_RUNTIME = r"""set -eu +for tool in docker dockerd containerd ctr; do + command -v "$tool" >/dev/null 2>&1 || { + echo "Docker runtime image is missing $tool" >&2 + exit 127 + } +done +ready=true +for tool in bash python3 findmnt mount tar xz realpath usermod; do + command -v "$tool" >/dev/null 2>&1 || ready=false +done +if [ "$ready" = false ]; then + if ! command -v apk >/dev/null 2>&1; then + printf '%s\n' 'Missing Docker support tools; install bash, python3,' \ + 'util-linux, coreutils, shadow, tar and xz in the runtime image.' >&2 + exit 127 + fi + apk add --no-cache bash python3 coreutils util-linux shadow tar xz +fi +docker compose version >/dev/null +""" + + +@dataclass(frozen=True) +class DockerRuntime: + """Prepare a private Docker daemon inside a fresh sandbox. + + Pass sandbox_options() to both fresh creation and from_snapshot(). Call + start() only on fresh creation: snapshot restore retains the actual daemon, + containers, sockets and bridge processes. The public default_image provides + Docker, containerd and Compose; fresh startup installs missing Alpine support + packages. Custom images must provide these tools. The matching helper is + source-built from builder/image-bridge, or preinstalled by the operator at + /opt/akernel-image-bridge. No code is downloaded at runtime. + """ + + default_image: ClassVar[str] = "docker.io/library/docker:28.3.3-dind" + + images: ImageMountPlan | None = None + bridge_binary: Path | None = None + writable_mb: int = 1024 + writable_backing: str = "file" + log_dir: Path | None = None + profile: str | None = None + + def __post_init__(self) -> None: + if self.profile is not None: + if ( + not isinstance(self.profile, str) + or re.fullmatch(r"[a-z][a-z0-9-]{0,62}", self.profile) is None + ): + raise ValueError("Invalid managed runtime profile") + if self.images is not None or self.bridge_binary is not None: + raise ValueError( + "Managed runtime profile owns image preparation and helpers" + ) + if ( + isinstance(self.writable_mb, bool) + or not isinstance(self.writable_mb, int) + or self.writable_mb < 1 + ): + raise ValueError("writable_mb must be a positive integer") + if self.writable_backing not in {"file", "memory"}: + raise ValueError("writable_backing must be file or memory") + if self.bridge_binary is not None and not Path(self.bridge_binary).is_file(): + raise ValueError("bridge_binary must be an existing matching executable") + + def sandbox_options(self) -> dict[str, Any]: + """Return public SDK mount/config options; also required during restore.""" + if self.profile is not None: + return {"extra_config": {"managedRuntimeProfile": self.profile}} + if self.images is None: + return {} + options: dict[str, Any] = {"mounts": self.images.mounts(Mount)} + if self.writable_backing == "file": + options["extra_config"] = { + "fileBackedTmpfsMounts": [ + { + "target": "/var/lib/akernel-image-snapshots", + "sizeMB": self.writable_mb, + } + ] + } + return options + + def _log(self, name: str, content: str) -> None: + if self.log_dir is not None: + self.log_dir.mkdir(parents=True, exist_ok=True) + (self.log_dir / name).write_text(content) + + def _bridge(self, sandbox: Sandbox) -> list[str]: + plan = self.images + if plan is None: + return ["--storage-driver=vfs"] + remote = "/opt/akernel-docker-runtime" + sandbox.files.make_dir(remote + "/metadata") + sandbox.files.write( + remote + "/metadata/plan.json", + json.dumps( + { + "images": plan.images, + "services": plan.services, + "aliases": plan.aliases, + "automatic": plan.automatic, + } + ), + ) + for name in sorted( + {b["file"] for image in plan.images for b in image["blobs"]} + ): + sandbox.files.copy_from_local( + str(plan.directory / name), remote + "/metadata/" + name + ) + if self.bridge_binary is not None: + sandbox.files.copy_from_local( + str(self.bridge_binary), "/opt/akernel-image-bridge" + ) + sandbox.commands.run( + "chmod 755 /opt/akernel-image-bridge", timeout=20, check=True + ) + # A packaged module is a wheel/sdist resource, not a Harbor source path. + sandbox.files.copy_from_local( + str(Path(__file__).with_name("_image_bridge_guest.py")), + remote + "/start.py", + ) + command = shlex.join( + [ + "python3", + remote + "/start.py", + "--root", + remote, + "--plan", + remote + "/metadata/plan.json", + "--writable-tmpfs-mb", + str(self.writable_mb), + "--writable-backing", + self.writable_backing, + ] + ) + try: + result = sandbox.commands.run(command, timeout=150, check=True) + self._log("startup.log", result.stdout + result.stderr) + except Exception: + try: + logs = sandbox.commands.run( + f"tail -100 {remote}/snapshotter.log {remote}/containerd.log " + f"{remote}/image-registration.log", + timeout=20, + check=True, + ) + self._log("services.log", logs.stdout + logs.stderr) + except Exception: + pass + raise + flags = json.loads(result.stdout) + if not isinstance(flags, list) or not all(isinstance(x, str) for x in flags): + raise ValueError("Image bridge returned invalid Docker arguments") + return flags + + def start(self, sandbox: Sandbox, *, network: str = "bridge") -> None: + """Start bridge, containerd and dockerd once; caller owns sandbox cleanup. + + Never call on a connected/restored sandbox. Fail closed on an existing + daemon so a retry cannot replace its mounted image/upper state. + """ + if self.profile is not None: + if network != "bridge": + raise ValueError( + "Managed runtime profile owns Docker network configuration" + ) + result = sandbox.commands.run( + "python3 /usr/lib/akernel/compose.py start", timeout=180, check=True + ) + self._log("startup.log", result.stdout + result.stderr) + return + if network not in {"host", "bridge"}: + raise ValueError("Docker network must be host or bridge") + sandbox.commands.run("test ! -S /var/run/docker.sock", timeout=20, check=True) + sandbox.commands.run( + shlex.join(["sh", "-c", _PREPARE_RUNTIME]), timeout=300, check=True + ) + flags = self._bridge(sandbox) + if network == "host": + flags += ["--iptables=false", "--ip6tables=false", "--bridge=none"] + else: + # gVisor supports legacy SNAT, but not all of Docker's netfilter + # matchers. Keep Docker's bridges/IPAM/DNS and userland port proxies; + # configure forwarding inside this sandbox rather than rewriting + # service networks or installing rules on the node. + script = Path(__file__).with_name("_docker_network_guest.py").read_text() + sandbox.commands.run( + shlex.join(["python3", "-c", script, "bootstrap"]), + timeout=20, + check=True, + ) + flags += ["--iptables=false", "--ip6tables=false", "--userland-proxy=true"] + sandbox.commands.run( + "PATH=/opt/akernel-docker-runtime/network-bin:" + "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin " + + shlex.join(["dockerd", *flags]) + + " > /var/log/akernel-dockerd.log 2>&1", + background=True, + ) + for _ in range(30): + try: + sandbox.commands.run("docker info", timeout=10, check=True) + return + except Exception: + time.sleep(2) + logs = sandbox.commands.run( + "cat /var/log/akernel-dockerd.log", timeout=10, check=True + ) + self._log("dockerd.log", logs.stdout + logs.stderr) + raise RuntimeError( + "Docker daemon readiness failed; inspect /var/log/akernel-dockerd.log" + ) + + @staticmethod + def reconnect_network(sandbox: Sandbox, *, network: str = "bridge") -> None: + """Refresh owned egress rules after a new outer TAP is assigned. + + Call after snapshot restore, before resuming work. This never starts + dockerd, containers, image helpers or listeners. It preserves inner + connections; external connections do not gain IP-migration guarantees. + """ + if network not in {"host", "bridge"}: + raise ValueError("Docker network must be host or bridge") + if network == "bridge": + script = Path(__file__).with_name("_docker_network_guest.py").read_text() + sandbox.commands.run( + shlex.join(["python3", "-c", script, "reconnect"]), + timeout=20, + check=True, + ) diff --git a/sdk/python/akernel_sdk/_image_bridge_guest.py b/sdk/python/akernel_sdk/_image_bridge_guest.py new file mode 100644 index 00000000..1a5ee4e3 --- /dev/null +++ b/sdk/python/akernel_sdk/_image_bridge_guest.py @@ -0,0 +1,240 @@ +"""Start a private containerd and snapshotter for sandboxd-mounted images.""" + +import json +import subprocess +import time +from pathlib import Path + + +def verify_image_bridge(plan): + """Check pinned image records and mounted roots without pulling or unpacking. + + Docker's digest pull path does not treat metadata-only registration as a + complete local content store. The committed snapshot is our filesystem + source; missing aliases or snapshots must fail instead of falling back. + """ + data = json.loads(plan.read_text()) + references = {} + roots_by_chain: dict[str, set[str]] = {} + for image in data["images"]: + references[image["reference"]] = image["target"]["digest"] + roots_by_chain.setdefault(image["chain_id"], set()).add(image["root"]) + for chain, roots in roots_by_chain.items(): + result = subprocess.run( + [ + "ctr", + "--address", + "/run/akernel-image-bridge/containerd.sock", + "--namespace", + "moby", + "snapshots", + "--snapshotter", + "akernel", + "info", + chain, + ], + check=True, + capture_output=True, + text=True, + timeout=30, + ) + info = json.loads(result.stdout) + if ( + info.get("Kind") != "Committed" + or info.get("Labels", {}).get( + "containerd.io/snapshot/agent-service.image-root" + ) + not in roots + ): + raise RuntimeError("Prepared image snapshot does not match its ROFS mount") + for alias, reference in data.get("aliases", {}).items(): + references[alias] = references[reference] + for reference, expected in references.items(): + result = subprocess.run( + [ + "docker", + "image", + "inspect", + "--format", + "{{.Descriptor.Digest}}", + reference, + ], + check=True, + capture_output=True, + text=True, + timeout=30, + ) + if result.stdout.strip() != expected: + raise RuntimeError("Prepared image reference changed: " + reference) + print("Verified pinned ROFS images; no layer pull or extraction required") + + +def start_image_bridge( + root, plan, children, writable_tmpfs_mb=0, writable_backing="memory" +): + if writable_tmpfs_mb < 0: + raise ValueError("writable tmpfs size must be nonnegative") + if writable_backing not in {"memory", "file"}: + raise ValueError("writable backing must be memory or file") + snapshots = Path("/var/lib/akernel-image-snapshots") + use_snapshots = writable_tmpfs_mb > 0 or writable_backing == "file" + if writable_backing == "file": + result = subprocess.run( + [ + "findmnt", + "--noheadings", + "--output", + "FSTYPE", + "--mountpoint", + str(snapshots), + ], + check=True, + capture_output=True, + text=True, + ) + if result.stdout.strip() != "tmpfs": + raise RuntimeError("sandboxd must provide the file-backed tmpfs mount") + if writable_tmpfs_mb and writable_backing == "memory": + # This POC uses a separate tmpfs to avoid using the root overlay as upper. + # Writable host binds also work on the tested runsc, but need a separate + # storage snapshot/rollback integration before replacing this path. + # This is within the Sandbox memory budget and survives whole-Sandbox C/R. + # Docker bind-mounts its data root at startup; keep the tmpfs outside it. + snapshots = Path("/var/lib/akernel-image-snapshots") + snapshots.mkdir(parents=True, exist_ok=True) + if any(snapshots.iterdir()): + raise RuntimeError( + "refusing to hide existing snapshotter state under tmpfs" + ) + subprocess.run( + [ + "mount", + "-t", + "tmpfs", + "-o", + f"size={writable_tmpfs_mb}m", + "worker-image-writes", + str(snapshots), + ], + check=True, + ) + directory = Path("/run/akernel-image-bridge") + directory.mkdir(mode=0o700, exist_ok=True) + config = root / "containerd-image-bridge.toml" + config.write_text("""version = 2 +root = "/var/lib/docker/containerd-bridge" +state = "/run/akernel-image-bridge/state" +disabled_plugins = ["io.containerd.grpc.v1.cri"] +[grpc] + address = "/run/akernel-image-bridge/containerd.sock" +[proxy_plugins.akernel] + type = "snapshot" + address = "/run/akernel-image-bridge/snapshotter.sock" +""") + for name, command in [ + ( + "snapshotter", + [ + "/opt/akernel-image-bridge", + "--plan", + str(plan), + *(["--root", str(snapshots)] if use_snapshots else []), + ], + ), + ("containerd", ["containerd", "--config", str(config)]), + ]: + with (root / (name + ".log")).open("ab") as log: + process = subprocess.Popen( + command, stdout=log, stderr=subprocess.STDOUT, start_new_session=True + ) + children.append(process) + deadline = time.monotonic() + 60 + while True: + if any(p.poll() is not None for p in children): + raise RuntimeError( + "image bridge exited; inspect snapshotter/containerd logs" + ) + try: + result = subprocess.run( + ["ctr", "--address", str(directory / "containerd.sock"), "version"], + check=False, + capture_output=True, + text=True, + timeout=3, + ) + if result.returncode == 0 and (directory / "snapshotter.sock").exists(): + break + except subprocess.TimeoutExpired: + pass + if time.monotonic() > deadline: + raise TimeoutError("image bridge readiness timeout") + time.sleep(0.2) + with (root / "image-registration.log").open("ab") as log: + subprocess.run( + ["/opt/akernel-image-bridge", "--mode", "register", "--plan", str(plan)], + stdout=log, + stderr=subprocess.STDOUT, + check=True, + timeout=120, + ) + for alias, reference in json.loads(plan.read_text()).get("aliases", {}).items(): + subprocess.run( + [ + "ctr", + "--address", + str(directory / "containerd.sock"), + "--namespace", + "moby", + "images", + "tag", + "--force", + reference, + alias, + ], + stdout=log, + stderr=subprocess.STDOUT, + check=True, + timeout=30, + ) + config = root / "docker-image-bridge.json" + config.write_text( + json.dumps( + { + "features": {"containerd-snapshotter": True}, + "containerd": str(directory / "containerd.sock"), + "containerd-namespace": "moby", + "storage-driver": "akernel", + } + ) + ) + return ["--config-file", str(config)] + + +if __name__ == "__main__": + import argparse + + parser = argparse.ArgumentParser() + parser.add_argument( + "--root", type=Path, default=Path("/opt/akernel-docker-runtime") + ) + parser.add_argument("--plan", type=Path, required=True) + parser.add_argument("--writable-tmpfs-mb", type=int, default=0) + parser.add_argument("--verify", action="store_true") + parser.add_argument( + "--writable-backing", choices=["memory", "file"], default="memory" + ) + args = parser.parse_args() + if args.verify: + verify_image_bridge(args.plan) + else: + print( + json.dumps( + start_image_bridge( + args.root, + args.plan, + [], + args.writable_tmpfs_mb, + args.writable_backing, + ) + ) + ) diff --git a/sdk/python/akernel_sdk/_image_plan.py b/sdk/python/akernel_sdk/_image_plan.py new file mode 100644 index 00000000..9e646639 --- /dev/null +++ b/sdk/python/akernel_sdk/_image_plan.py @@ -0,0 +1,175 @@ +"""Startup-declared, immutable Compose images mounted by AKernel. + +The snapshotter and guest bootstrap originate from tianyuzhou95's image-bridge +POC, agent-worker commit 9b3434d36086d920b6bb7f3322da642381a254b8. +Docker keeps ownership of container lifecycle; this module stages the mount plan. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from collections.abc import Callable + + +def _digest(value: Any) -> str: + if ( + not isinstance(value, str) + or re.fullmatch(r"sha256:[0-9a-f]{64}", value) is None + ): + raise ValueError("Image bridge requires sha256 image and metadata digests") + return value + + +def _chain_id(diffs: list[str]) -> str: + if not diffs: + raise ValueError("Mounted external images require a nonempty rootfs") + chain = _digest(diffs[0]) + for diff in diffs[1:]: + chain = ( + "sha256:" + hashlib.sha256(f"{chain} {_digest(diff)}".encode()).hexdigest() + ) + return chain + + +def _validate_image_config(image: dict[str, Any], payloads: dict[str, Any]) -> None: + """Bind the mounted chain to the selected immutable OCI configuration.""" + try: + manifest = payloads[image["target"]["digest"]] + for _ in range(4): + if "manifests" not in manifest: + break + matches = [ + d + for d in manifest["manifests"] + if d.get("platform", {}).get("os") == "linux" + and d.get("platform", {}).get("architecture") == "amd64" + ] + if len(matches) != 1: + raise ValueError("Image plan requires one linux/amd64 manifest") + manifest = payloads[_digest(matches[0]["digest"])] + expected = manifest["config"] + if any( + expected[k] != image["config"][k] for k in ("digest", "size", "mediaType") + ): + raise ValueError("Image config is not selected by its target manifest") + config = payloads[_digest(expected["digest"])] + if config.get("os") != "linux" or config.get("architecture") != "amd64": + raise ValueError("Image config must match linux/amd64") + rootfs = config["rootfs"] + diffs = rootfs["diff_ids"] + if ( + rootfs.get("type") != "layers" + or not isinstance(diffs, list) + or len(diffs) != len(manifest["layers"]) + ): + raise ValueError("Image rootfs and manifest layer list disagree") + if _chain_id(diffs) != image["chain_id"]: + raise ValueError("Image chain ID does not match its rootfs configuration") + except (KeyError, TypeError) as error: + raise ValueError( + "Image plan has incomplete OCI manifest/config metadata" + ) from error + + +@dataclass(frozen=True) +class ImageMountPlan: + directory: Path + images: list[dict[str, Any]] + services: dict[str, str] + aliases: dict[str, str] = field(default_factory=dict) + automatic: bool = False + + @classmethod + def load(cls, path: str | Path, *, bases_only: bool = False) -> ImageMountPlan: + path = Path(path).resolve() + data = json.loads(path.read_text()) + images, services = data.get("images"), data.get("services") + automatic = data.get("automatic", False) + if not isinstance(automatic, bool): + raise ValueError("Image plan automatic marker must be a boolean") + if not isinstance(images, list) or ( + not images and not (bases_only and automatic) + ): + raise ValueError("Image plan requires a nonempty images list") + if not isinstance(services, dict) or ( + "main" not in services and not bases_only + ): + raise ValueError("Image plan requires a services map including main") + by_reference = {} + for image in images: + reference = image["reference"] + digest = _digest(image["target"]["digest"]) + if ( + not reference.endswith("@" + digest) + or image["root"] != "/opt/akernel-images/" + digest[7:] + or reference in by_reference + ): + raise ValueError("Image reference, digest and immutable mount disagree") + _digest(image["chain_id"]) + by_reference[reference] = image + descriptors = set() + payloads = {} + for blob in image["blobs"]: + name = blob["file"] + if not isinstance(name, str) or Path(name).name != name: + raise ValueError("Image metadata must be a sibling filename") + metadata = path.parent / name + if metadata.is_symlink(): + raise ValueError("Image metadata must not be a symlink") + payload = metadata.read_bytes() + descriptor = blob["descriptor"] + actual = "sha256:" + hashlib.sha256(payload).hexdigest() + if ( + actual != _digest(descriptor["digest"]) + or len(payload) != descriptor["size"] + ): + raise ValueError("Image metadata digest or size mismatch") + descriptors.add(actual) + payloads[actual] = json.loads(payload) + if not {digest, _digest(image["config"]["digest"])} <= descriptors: + raise ValueError("Image plan is missing its manifest or config") + _validate_image_config(image, payloads) + for service, reference in services.items(): + if ( + not isinstance(service, str) + or not service + or reference not in by_reference + ): + raise ValueError( + "Every planned service must reference a declared image" + ) + aliases = data.get("aliases", {}) + if not isinstance(aliases, dict) or any( + not isinstance(alias, str) or not alias or reference not in by_reference + for alias, reference in aliases.items() + ): + raise ValueError("Image aliases must reference declared immutable images") + selected = ( + list(by_reference) if bases_only else list(dict.fromkeys(services.values())) + ) + return cls( + path.parent, + [by_reference[r] for r in selected], + dict(services), + { + alias: reference + for alias, reference in aliases.items() + if reference in selected + }, + automatic, + ) + + def mounts(self, factory: Callable[..., Any]) -> list[Any]: + # Different registry names can refer to the same immutable image root. + by_root = {i["root"]: i for i in self.images} + return [ + factory(target=i["root"], image_url=i["reference"]) + for i in by_root.values() + ] diff --git a/sdk/python/akernel_sdk/_image_resolver.py b/sdk/python/akernel_sdk/_image_resolver.py new file mode 100644 index 00000000..8d152fc3 --- /dev/null +++ b/sdk/python/akernel_sdk/_image_resolver.py @@ -0,0 +1,246 @@ +"""Automatically prepare immutable image roots; Docker still performs the build. + +The bridge's host-side discovery command uses compose-go and BuildKit's actual +Dockerfile parser. Only OCI manifests/configs pass through the SDK client. +Layer data is supplied by AKernel's image manager through read-only mounts. +""" + +from __future__ import annotations + +import base64 +import hashlib +import json +import os +import re +import subprocess +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any +from urllib.parse import urlsplit + +import httpx + +from ._image_plan import ImageMountPlan, _chain_id, _digest + +ACCEPT = ( + "application/vnd.oci.image.index.v1+json, " + "application/vnd.oci.image.manifest.v1+json, " + "application/vnd.docker.distribution.manifest.list.v2+json, " + "application/vnd.docker.distribution.manifest.v2+json" +) + + +def _docker_credentials(host: str) -> tuple[str, str] | None: + config = Path(os.environ.get("DOCKER_CONFIG", str(Path.home() / ".docker"))) + path = config / "config.json" + data = json.loads(path.read_text()) if path.exists() else {} + server = "https://index.docker.io/v1/" if host == "docker.io" else host + helper = data.get("credHelpers", {}).get(host) or data.get("credsStore") + if helper: + result = subprocess.run( + ["docker-credential-" + helper, "get"], + input=server, + text=True, + capture_output=True, + timeout=20, + check=False, + ) + if result.returncode: + raise RuntimeError(f"Docker credential helper failed for {host}") + credential = json.loads(result.stdout) + username, secret = credential["Username"], credential["Secret"] + if not isinstance(username, str) or not isinstance(secret, str): + raise ValueError("Invalid Docker credential helper response") + return username, secret + for name in (host, server, "https://" + host): + auth = data.get("auths", {}).get(name, {}).get("auth") + if auth: + username, password = ( + base64.b64decode(auth, validate=True).decode().split(":", 1) + ) + return username, password + return None + + +class RegistryMetadata: + def __init__(self, client: httpx.Client, directory: Path): + self.client = client + self.directory = directory + self.tokens: dict[tuple[str, str], str] = {} + + def fetch(self, host: str, repository: str, kind: str, selector: str) -> bytes: + endpoint = "registry-1.docker.io" if host == "docker.io" else host + url = f"https://{endpoint}/v2/{repository}/{kind}/{selector}" + key = (host, repository) + headers = {"Accept": ACCEPT} + if key in self.tokens: + headers["Authorization"] = "Bearer " + self.tokens[key] + response = self.client.get(url, headers=headers) + if response.status_code == 401: + challenge = response.headers.get("www-authenticate", "") + credentials = _docker_credentials(host) + if challenge.lower().startswith("bearer "): + fields = dict(re.findall(r'(\w+)="([^"]*)"', challenge)) + realm = fields.get("realm", "") + if urlsplit(realm).scheme != "https": + raise ValueError("Registry token service must use HTTPS") + token = self.client.get( + realm, + params={k: fields[k] for k in ("service", "scope") if k in fields}, + auth=credentials, + ) + token.raise_for_status() + value = token.json().get("token") or token.json().get("access_token") + if not isinstance(value, str) or not value: + raise ValueError("Registry returned no bearer token") + self.tokens[key] = value + headers["Authorization"] = "Bearer " + value + response = self.client.get(url, headers=headers) + elif challenge.lower().startswith("basic ") and credentials: + response = self.client.get(url, headers=headers, auth=credentials) + response.raise_for_status() + payload = response.content + actual = "sha256:" + hashlib.sha256(payload).hexdigest() + expected = ( + selector + if selector.startswith("sha256:") + else response.headers.get("docker-content-digest") + ) + if expected and actual != _digest(expected): + raise ValueError("Registry metadata digest mismatch") + return payload + + def prepare(self, reference: str) -> dict[str, Any]: + repository, separator, selector = reference.partition("@") + if not separator: + repository, selector = reference.rsplit(":", 1) + elif ":" in repository.rsplit("/", 1)[-1]: + repository = repository.rsplit(":", 1)[0] + host, name = repository.split("/", 1) + blobs = [] + + def fetch(kind: str, selected: str, descriptor: dict[str, Any] | None = None): + payload = self.fetch(host, name, kind, selected) + obj = json.loads(payload) + digest = "sha256:" + hashlib.sha256(payload).hexdigest() + if descriptor is not None and len(payload) != descriptor["size"]: + raise ValueError("Registry metadata size mismatch") + media_type = (descriptor or {}).get("mediaType") or obj["mediaType"] + result = {"mediaType": media_type, "digest": digest, "size": len(payload)} + filename = digest.replace(":", "-") + ".json" + (self.directory / filename).write_bytes(payload) + blobs.append({"descriptor": result, "file": filename}) + return obj, result + + manifest, target = fetch("manifests", selector) + for _ in range(4): + if "manifests" not in manifest: + break + matches = [ + d + for d in manifest["manifests"] + if d.get("platform", {}).get("os") == "linux" + and d.get("platform", {}).get("architecture") == "amd64" + ] + if len(matches) != 1: + raise ValueError("Image must contain exactly one linux/amd64 manifest") + manifest, _ = fetch("manifests", _digest(matches[0]["digest"]), matches[0]) + if "manifests" in manifest: + raise ValueError("Image index nesting is too deep") + descriptor = manifest["config"] + config, config_descriptor = fetch( + "blobs", _digest(descriptor["digest"]), descriptor + ) + if config.get("os") != "linux" or config.get("architecture") != "amd64": + raise ValueError("Image config must match linux/amd64") + diffs = [_digest(d) for d in config["rootfs"]["diff_ids"]] + if not diffs: + raise ValueError("An external image with an empty rootfs cannot be mounted") + chain = _chain_id(diffs) + return { + "reference": repository + "@" + target["digest"], + "root": "/opt/akernel-images/" + target["digest"][7:], + "chain_id": chain, + "target": target, + "config": config_descriptor, + "blobs": blobs, + } + + +def prepare_compose_images( + directory: str | Path, + files: Sequence[str | Path], + *, + output_dir: str | Path, + bridge_binary: str | Path, + env: Mapping[str, str] | None = None, +) -> ImageMountPlan: + """Discover Compose/Dockerfile image inputs without building or pulling layers. + + The operator supplies a source-built, version-matched image-bridge helper. + Credentials stay on the client. Sandboxd's existing image manager owns layer + distribution, ROFS mounts and shared caching; this stores metadata only. + Original project files are read without modification. + """ + output = Path(output_dir).resolve() + output.mkdir(parents=True, exist_ok=True) + request = { + "directory": str(Path(directory).resolve()), + "files": [str(Path(p).resolve()) for p in files], + } + binary = str(Path(bridge_binary).resolve()) + # Probe without project data before invoking a possibly older helper. The + # schema version requires client-file containment before reading Dockerfiles. + capability = subprocess.run( + [binary, "--mode", "capabilities"], + env=dict(env or {}), + text=True, + capture_output=True, + timeout=10, + check=False, + ) + try: + supported = ( + capability.returncode == 0 + and json.loads(capability.stdout).get("discover_schema") == 2 + ) + except (ValueError, AttributeError): + supported = False + if not supported: + raise ValueError( + "Image bridge helper must support discovery schema 2; " + "install the helper matching this SDK" + ) + result = subprocess.run( + [binary, "--mode", "discover"], + input=json.dumps(request), + env=dict(env or {}), + text=True, + capture_output=True, + timeout=60, + check=False, + ) + if result.returncode: + raise ValueError("Automatic image discovery failed: " + result.stderr[-1500:]) + references = json.loads(result.stdout)["references"] + aliases: dict[str, str] = {} + images: dict[str, dict[str, Any]] = {} + with httpx.Client(follow_redirects=True, timeout=60) as client: + registry = RegistryMetadata(client, output) + for reference in references: + image = registry.prepare(reference) + images.setdefault(image["reference"], image) + aliases[reference] = image["reference"] + path = output / "plan.json" + path.write_text( + json.dumps( + { + "images": list(images.values()), + "services": {}, + "aliases": aliases, + "automatic": True, + }, + indent=2, + ) + ) + return ImageMountPlan.load(path, bases_only=True) diff --git a/sdk/python/akernel_sdk/_managed_build.py b/sdk/python/akernel_sdk/_managed_build.py new file mode 100644 index 00000000..6992d9a1 --- /dev/null +++ b/sdk/python/akernel_sdk/_managed_build.py @@ -0,0 +1,186 @@ +"""Build submission and opaque runtime artifact handles; no Dockerfile parser.""" + +from __future__ import annotations + +import json +import re +import shlex +import tarfile +import tempfile +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING + +from ._backends.errors import BackendOperationError, retain_failure_context + +if TYPE_CHECKING: + from .sandbox import Sandbox + + +@dataclass(frozen=True) +class BuiltImage: + """Immutable image and public process defaults returned by the runtime. + + ``has_startup_command`` indicates whether its effective OCI ENTRYPOINT/CMD + is nonempty. An image with neither can still host an interactive sandbox; + leave ``Sandbox.inherit_entrypoint`` disabled for that image. + """ + + reference: str + envs: dict[str, str] + cwd: str | None = None + user: str | None = None + has_startup_command: bool = True + + +def upload_context(sandbox: Sandbox, source: Path, destination: str) -> None: + """Transfer a directory as ordinary files/links, retaining native build inputs.""" + source = source.resolve(strict=True) + if not source.is_dir(): + raise ValueError("Build context must be a directory") + archive = destination + ".upload.tar.gz" + with tempfile.TemporaryDirectory(prefix="akernel-context-") as temporary: + local = Path(temporary) / "context.tar.gz" + with tarfile.open(local, "w:gz", dereference=False) as output: + output.add(source, arcname=".") + sandbox.files.copy_from_local(str(local), archive) + sandbox.commands.run( + shlex.join(["mkdir", "-p", destination]), timeout=20, check=True + ) + sandbox.commands.run( + shlex.join(["tar", "-xzf", archive, "-C", destination]), timeout=120, check=True + ) + sandbox.commands.run(shlex.join(["rm", "-f", archive]), timeout=20, check=True) + + +def build_image( + context: str | Path, + *, + profile: str, + no_cache: bool = False, + timeout: int = 1800, + schedule_timeout: int = 180, + log_path: str | Path | None = None, + cpu_request: int | None = None, + memory_request: int | None = None, +) -> BuiltImage: + """Submit an original context through an authorized platform build profile. + + The platform supplies the builder image, endpoint, publication credentials + and image provider. No host Docker daemon, helper binary or Registry route + is needed on this client. The temporary preparation Sandbox is deleted on + success/failure; unknown outcomes are not replayed. Cleanup errors preserve + the preparation Sandbox identity. The returned image lifetime follows the + platform profile's retention contract. Timeouts are seconds. + + Optional preparation CPU/memory requests use millicores/MiB. The existing + preparation limits remain 1000 millicores and 2048 MiB; these settings do + not change task resources or pretend the builder can use unlimited memory. + """ + from ._docker_runtime import DockerRuntime + from .sandbox import Sandbox + + options = DockerRuntime(profile=profile).sandbox_options() + source = Path(context).resolve(strict=True) + if not source.is_dir() or not (source / "Dockerfile").is_file(): + raise ValueError("Native build requires a context containing Dockerfile") + if isinstance(timeout, bool) or not isinstance(timeout, int) or timeout < 1: + raise ValueError("Build timeout must be a positive integer") + for name, request, limit in ( + ("cpu_request", cpu_request, 1000), + ("memory_request", memory_request, 2048), + ): + if request is not None and ( + isinstance(request, bool) + or not isinstance(request, int) + or not 1 <= request <= limit + ): + raise ValueError(f"{name} must be an integer between 1 and {limit}") + failure: BaseException | None = None + builder = Sandbox( + cpu=1000 if cpu_request is None else cpu_request, + memory=2048 if memory_request is None else memory_request, + cpu_limit=1000, + mem_limit=2048, + idle_timeout=max(300, timeout + 60), + schedule_timeout=schedule_timeout, + **options, + ) + try: + upload_context(builder, source, "/work/context") + result = builder.commands.run( + shlex.join( + [ + "python3", + "/usr/lib/akernel/compose.py", + "image-build", + "--context", + "/work/context", + "--result", + "/work/artifact.json", + *(["--no-cache"] if no_cache else []), + ] + ), + timeout=timeout, + ) + if log_path is not None: + target = Path(log_path) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(result.stdout + result.stderr) + from .commands import _checked_result + + _checked_result(result, True) + handle = json.loads( + builder.commands.run( + "cat /work/artifact.json", timeout=20, check=True + ).stdout + ) + reference = handle["reference"] + if ( + not isinstance(reference, str) + or re.fullmatch(r"[^\s]+@sha256:[0-9a-f]{64}", reference) is None + ): + raise ValueError("Runtime returned an invalid artifact reference") + process = handle.get("image_process", {}) + return BuiltImage( + reference=reference, + envs=dict( + value.split("=", 1) for value in process.get("Env", []) if "=" in value + ), + cwd=process.get("WorkingDir") or None, + user=process.get("User") or None, + has_startup_command=bool(process.get("Entrypoint") or process.get("Cmd")), + ) + except BaseException as error: + failure = error + if ( + isinstance(error, BackendOperationError) + and error.command_result is not None + and log_path is not None + ): + try: + target = Path(log_path) + target.parent.mkdir(parents=True, exist_ok=True) + partial = error.command_result + target.write_text(partial.stdout + partial.stderr) + except OSError as logging_error: + add_note = getattr(error, "add_note", None) + if callable(add_note): + add_note( + "Could not preserve partial build log: " + + type(logging_error).__name__ + ) + raise + finally: + try: + builder.kill() + except BaseException as cleanup: + if failure is not None: + retain_failure_context(failure, builder.id, cleanup) + else: + cleanup_failure = BackendOperationError( + "Managed build preparation cleanup failed", + environment_id=builder.id, + ) + retain_failure_context(cleanup_failure, builder.id, cleanup) + raise cleanup_failure from cleanup diff --git a/sdk/python/akernel_sdk/_service_process.py b/sdk/python/akernel_sdk/_service_process.py new file mode 100644 index 00000000..ff39db24 --- /dev/null +++ b/sdk/python/akernel_sdk/_service_process.py @@ -0,0 +1,191 @@ +"""Recoverable identity of an actual process inside a Compose service.""" + +from __future__ import annotations + +import json +import re +import shlex +import time +from pathlib import Path +from typing import TYPE_CHECKING, Any +from uuid import uuid4 + +from ._backends.errors import BackendOperationError, TimeoutException + +if TYPE_CHECKING: + from collections.abc import Mapping + + from ._compose import ServiceHandle + + +class ServiceProcessHandle: + """A service process, independent of its short-lived docker-exec submission. + + The in-service monitor and identity record are included in whole-sandbox + snapshots. attach() never relaunches a command. Container identity/start time + and Linux process start time are checked before signalling. Python 3 is + required inside this service; ordinary ServiceHandle.exec needs no Python. + """ + + def __init__(self, service: ServiceHandle, descriptor: Mapping[str, Any]) -> None: + self.service = service + self._descriptor = json.loads(json.dumps(descriptor, allow_nan=False)) + if ( + self._descriptor.get("schema_version") != 1 + or re.fullmatch(r"[0-9a-f]{32}", self._descriptor.get("process_id", "")) + is None + or self._descriptor.get("service") != service.name + or self._descriptor.get("project_name") + != service.project.descriptor["project_name"] + ): + raise ValueError("Invalid service process descriptor") + + @property + def descriptor(self) -> dict[str, Any]: + return json.loads(json.dumps(self._descriptor)) + + @property + def process_id(self) -> str: + return str(self._descriptor["process_id"]) + + @property + def pid(self) -> int: + return int(self._descriptor["pid"]) + + def _verify_container(self) -> None: + identity = self.service.identity() + if any( + identity[k] != self._descriptor[k] for k in ("container_id", "started_at") + ): + raise BackendOperationError( + "Service container was replaced or restarted", + code="SERVICE_PROCESS_STALE", + outcome="not_started", + ) + + def _invoke( + self, + operation: str, + argument: str, + *, + user: str | int | None = "root", + cwd: str | None = None, + envs: Mapping[str, str] | None = None, + ) -> Any: + script = Path(__file__).with_name("_service_process_guest.py").read_text() + command = ( + "for executable in /opt/akernel/harness-python/usr/local/bin/python3 " + "python3 python /usr/local/bin/python3 " + "/usr/bin/python3; do " + 'process_python=$(command -v "$executable") || continue; ' + '"$process_python" -I -S -c "import sys;sys.exit(sys.version_info[0]!=3)" ' + "> /dev/null 2>&1 || continue; " + 'exec "$process_python" -I -S -c ' + + shlex.quote(script) + + " " + + shlex.join([operation, self.process_id, argument]) + + '; done; echo "Service process handles need Python 3" >&2; exit 127' + ) + result = self.service.exec(command, cwd=cwd, envs=envs, user=user, timeout=30) + if result.exit_code != 0: + failure = BackendOperationError( + "Service process operation failed: " + result.stderr[-500:], + code="SERVICE_PROCESS_CONFIGURATION" + if result.exit_code == 127 + else "SERVICE_PROCESS_OBSERVATION", + outcome="unknown" if operation == "start" else None, + ) + failure.process_descriptor = self.descriptor # type: ignore[attr-defined] + raise failure + return json.loads(result.stdout) + + @classmethod + def start( + cls, + service: ServiceHandle, + command: str, + *, + cwd: str | None = None, + envs: Mapping[str, str] | None = None, + user: str | int | None = None, + shell: str = "/bin/bash", + ) -> ServiceProcessHandle: + if ( + not isinstance(command, str) + or not command.strip() + or not shell.startswith("/") + ): + raise ValueError("Process command and absolute shell are required") + identity = service.identity() + descriptor = { + "schema_version": 1, + "process_id": uuid4().hex, + "project_name": service.project.descriptor["project_name"], + "service": service.name, + **identity, + } + result = cls(service, descriptor) + try: + row = result._invoke( + "start", + json.dumps({"command": command, "shell": shell}), + cwd=cwd, + envs=envs, + user=user, + ) + except BaseException as failure: + # The remote process may already exist. Retain its recovery token; + # never repeat a side-effecting start after an unknown response. + failure.process_descriptor = result.descriptor # type: ignore[attr-defined] + raise + result._descriptor.update(pid=row["pid"], start_time=row["start_time"]) + return result + + @classmethod + def attach( + cls, service: ServiceHandle, descriptor: Mapping[str, Any] + ) -> ServiceProcessHandle: + result = cls(service, descriptor) + result._verify_container() + status = result._invoke("status", "") + root = status["identity"] + if "pid" in result._descriptor and ( + root["pid"] != result.pid + or root["start_time"] != result._descriptor["start_time"] + ): + raise BackendOperationError( + "Service process identity changed", code="SERVICE_PROCESS_STALE" + ) + result._descriptor.update(pid=root["pid"], start_time=root["start_time"]) + return result + + def inspect(self) -> list[dict[str, Any]]: + return self.signal("inspect") + + def signal(self, action: str) -> list[dict[str, Any]]: + if action not in {"inspect", "stop", "continue", "terminate", "kill"}: + raise ValueError("Unsupported service process action") + self._verify_container() + return self._invoke( + "signal", + json.dumps( + { + "action": action, + "pid": self.pid, + "start_time": self._descriptor["start_time"], + } + ), + ) + + def wait(self, *, timeout: float = 60) -> int: + deadline = time.monotonic() + timeout + while True: + self._verify_container() + status = self._invoke("status", "") + if status["exit_code"] is not None: + return int(status["exit_code"]) + if time.monotonic() >= deadline: + raise TimeoutException( + "Service process wait timed out", outcome="running" + ) + time.sleep(0.1) diff --git a/sdk/python/akernel_sdk/_service_process_guest.py b/sdk/python/akernel_sdk/_service_process_guest.py new file mode 100644 index 00000000..0c9c4f4b --- /dev/null +++ b/sdk/python/akernel_sdk/_service_process_guest.py @@ -0,0 +1,170 @@ +"""Dependency-free helper executed inside the selected service's PID namespace.""" + +import json +import os +import signal +import sys +import time +from pathlib import Path + + +def stat(pid): + try: + value = Path("/proc/" + str(pid) + "/stat").read_text() + fields = value[value.rindex(")") + 2 :].split() + return { + "pid": pid, + "ppid": int(fields[1]), + "state": fields[0], + "comm": value[value.index("(") + 1 : value.rindex(")")], + "start_time": fields[19], + } + except (OSError, ValueError): + return None + + +def write(path, data): + temporary = path.with_suffix(".tmp") + temporary.write_text(json.dumps(data)) + temporary.replace(path) + + +def start(root, config): + # Exclusive reservation prevents replay of an ambiguous launch. + root.mkdir(mode=0o700, exist_ok=False) + monitor = os.fork() + if monitor == 0: + os.setsid() + null = os.open("/dev/null", os.O_RDWR) + for number in (0, 1, 2): + os.dup2(null, number) + os.close(null) + pid = os.fork() + if pid == 0: + try: + for number, name in ((1, "stdout"), (2, "stderr")): + descriptor = os.open( + str(root / name), os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o600 + ) + os.dup2(descriptor, number) + os.close(descriptor) + os.execvpe( + config["shell"], + [config["shell"], "-c", config["command"]], + os.environ, + ) + except BaseException: + os._exit(127) + record = stat(pid) + if record is not None: + record["monitor_pid"] = os.getpid() + write(root / "identity.json", record) + _, status = os.waitpid(pid, 0) + code = ( + os.WEXITSTATUS(status) + if os.WIFEXITED(status) + else 128 + os.WTERMSIG(status) + ) + write(root / "exit.json", {"exit_code": code}) + os._exit(0) + deadline = time.monotonic() + 10 + while not (root / "identity.json").exists(): + if time.monotonic() >= deadline: + raise RuntimeError("process launch outcome unknown; do not resubmit") + time.sleep(0.02) + return json.loads((root / "identity.json").read_text()) + + +def inspect(root): + record = json.loads((root / "identity.json").read_text()) + current = stat(record["pid"]) + if current is None or current["state"] == "Z": + return [] + if current["start_time"] != record["start_time"]: + raise RuntimeError("process identity mismatch; refusing reused PID") + records = {} + for name in os.listdir("/proc"): + if name.isdigit(): + row = stat(int(name)) + if row is not None: + records[row["pid"]] = row + selected = {record["pid"]} + while True: + children = { + pid for pid, row in records.items() if row["ppid"] in selected + } - selected + if not children: + break + selected.update(children) + return [records[pid] for pid in sorted(selected) if pid in records] + + +def operate(root, action): + rows = inspect(root) + if action == "inspect": + return rows + signals = { + "stop": signal.SIGSTOP, + "continue": signal.SIGCONT, + "terminate": signal.SIGTERM, + "kill": signal.SIGKILL, + } + if action not in signals: + raise ValueError("unsupported process action") + for _ in range(3 if action == "stop" else 1): + parents = {row["pid"]: row["ppid"] for row in rows} + + def depth(row, parents=parents): + pid, visited = row["pid"], set() + while pid in parents and pid not in visited: + visited.add(pid) + pid = parents[pid] + return len(visited) + + # Freeze waiters before children. Thaw children before waiters so Bash + # cannot consume a pending child-stop notification as exit status 147. + # PID numeric ordering does not encode ancestry after PID reuse/wrap. + for row in sorted(rows, key=depth, reverse=action == "continue"): + current = stat(row["pid"]) + if current is not None and current["start_time"] == row["start_time"]: + try: + os.kill(row["pid"], signals[action]) + except ProcessLookupError: + pass + if action == "stop": + time.sleep(0.03) + rows = inspect(root) + return rows + + +def main(): + operation, identity, argument = sys.argv[1:4] + if len(identity) != 32 or any(c not in "0123456789abcdef" for c in identity): + raise ValueError("invalid process identity") + root = Path("/tmp/akernel-service-process-" + identity) + if operation == "start": + result = start(root, json.loads(argument)) + elif operation == "signal": + request = json.loads(argument) + expected = json.loads((root / "identity.json").read_text()) + if ( + request.get("pid") != expected["pid"] + or request.get("start_time") != expected["start_time"] + ): + raise RuntimeError("process identity record changed") + result = operate(root, request["action"]) + elif operation == "status": + result = { + "processes": inspect(root), + "identity": json.loads((root / "identity.json").read_text()), + "exit_code": None, + } + if (root / "exit.json").exists(): + result.update(json.loads((root / "exit.json").read_text())) + else: + raise ValueError("unsupported process operation") + print(json.dumps(result), flush=True) + + +if __name__ == "__main__": + main() diff --git a/sdk/python/akernel_sdk/commands.py b/sdk/python/akernel_sdk/commands.py index 70d00d7e..563fdda6 100644 --- a/sdk/python/akernel_sdk/commands.py +++ b/sdk/python/akernel_sdk/commands.py @@ -43,9 +43,7 @@ def command_id(self) -> str | None: """Stable command ID fixed at attachment, independent of later PID reuse.""" return self._command_id - def wait( - self, timeout: int | None = None, *, check: bool = False - ) -> CommandResult: + def wait(self, timeout: int | None = None, *, check: bool = False) -> CommandResult: """Wait for completion; check=True raises on nonzero or unfinished exit. The default preserves CommandResult behavior. An unfinished wait only @@ -167,6 +165,7 @@ def _checked_result(result: CommandResult, check: bool) -> CommandResult: result.error_message or "command wait timed out; the command may still run", code=result.error_code or "WAIT_TIMEOUT", + command_result=result, ) if result.exit_code != 0: raise CommandExitException(result.stdout, result.stderr, result.exit_code) diff --git a/sdk/python/akernel_sdk/sandbox.py b/sdk/python/akernel_sdk/sandbox.py index 850f210e..4a31fe6b 100644 --- a/sdk/python/akernel_sdk/sandbox.py +++ b/sdk/python/akernel_sdk/sandbox.py @@ -771,6 +771,7 @@ def create( cwd: str | None = None, port_forwardings: Sequence[int] | None = None, mounts: Sequence[Mount] | None = None, + extra_config: Mapping[str, object] | None = None, reverse_tunnel: HttpReverseTunnel | None = None, detached: bool = False, failover: bool = False, @@ -815,6 +816,7 @@ def create( cwd=cwd, port_forwardings=port_forwardings, mounts=mounts, + extra_config=extra_config, reverse_tunnel=reverse_tunnel, detached=detached, failover=failover, @@ -854,6 +856,7 @@ def create( cwd=cwd, port_forwardings=port_forwardings, mounts=mounts, + extra_config=extra_config, reverse_tunnel=reverse_tunnel, detached=detached, failover=failover, @@ -874,6 +877,7 @@ def from_snapshot( cwd: str | None = None, port_forwardings: Sequence[int] | None = None, mounts: Sequence[Mount] | None = None, + extra_config: Mapping[str, object] | None = None, reverse_tunnel: HttpReverseTunnel | None = None, detached: bool = False, failover: bool = False, @@ -888,6 +892,9 @@ def from_snapshot( Inherits captured runtime, resource geometry and process state; does not rerun Dockerfile setup or the image entrypoint. The source is unaffected. Rebind client-side tunnels, ports and mounts explicitly as needed. + ``extra_config`` is defensively copied JSON runtime configuration. + Restore the same managed-mount geometry used by the captured runtime; + the backend owns compatibility checks and does not rerun fresh setup. ``cwd`` sets a command default without creating directories. Supply ``network_policy`` explicitly to restore restrictions: the pinned ADX backend uses its default if omitted or empty. ``timeout`` is the logical @@ -939,6 +946,7 @@ def from_snapshot( command_cwd=cwd, port_forwardings=tuple(ports), mounts=tuple(mount_list), + extra_config=_normalize_extra_config(extra_config), reverse_tunnel=reverse_tunnel, detached=detached, failover=failover, diff --git a/sdk/python/docs/managed-compose.md b/sdk/python/docs/managed-compose.md new file mode 100644 index 00000000..a7dc2e1a --- /dev/null +++ b/sdk/python/docs/managed-compose.md @@ -0,0 +1,160 @@ +# Managed Compose integration + +The managed runtime runs Docker and Compose inside one outer AKernel sandbox. +Applications submit their original Dockerfiles, build contexts and Compose files. +The operator supplies build connectivity, tools, Registry authorization and shared +read-only image layers. The SDK exposes project, service and process handles. +This opt-in Linux/amd64 runsc profile is separate from the SDK's strict Dockerfile +interpreter. + +## Component responsibilities + +| Component | Responsibility | +| --- | --- | +| Application or evaluation framework | Task resources, environment variables, bind paths, verifier isolation, checkpoint timing and client-side state. Fence the source before restoring a running clone. | +| AKernel SDK | Freeze and transfer inputs, submit runtime operations, expose artifact/project/service/process handles and reconnect client tunnels. | +| Managed runtime | Run official Docker Compose and Buildx remote, publish immutable image references, prepare authorized ROFS collections and start Docker/containerd with private writable layers. | +| Agent DX | Environment creation, command/file/tunnel transport and reusable snapshot control. | +| Sandboxd and the image manager | Authorize profiles, enforce storage quotas, distribute shared ROFS content and retain live/checkpoint collection leases. | +| runsc | Capture the complete outer sandbox, including Docker, services, memory, private filesystem state and sockets. | + +BuildKit owns build-step caching. Dragonfly and the existing image manager own +final image distribution and ROFS caching. Each sandbox has a private writable +upper; the SDK does not maintain a second shared image-layer cache. + +## Operator prerequisites + +1. Configure a managed-runtime profile with an immutable tool image, a reachable + remote BuildKit endpoint, an internal Registry and authorized tenants. Build + workers need Registry push access; runtime nodes need pull access. Harbor + clients need neither Registry credentials nor Kubernetes access. +2. Enable runsc file-backed writable mounts, reflink checkpoint mode and durable + output synchronization on a compatible filesystem. On Linux 5.10, backing + files and checkpoint clones must use the same mount object. Collection leases + and checkpoint output must share a filesystem. The SDK cannot enable these + features through a snapshot option. +3. Configure Docker 28's runsc flags `--net-raw=true` and + `--allow-packet-socket-write=true`. Package Docker, Compose, Buildx, containerd, + the [image bridge](../../../builder/image-bridge/README.md) and support tools + in the operator's runtime image. User task images do not need these tools. +4. Publish the profile's storage floor through authenticated worker metadata. + See the [operator guide](../../../builder/managed-docker/README.md) for + configuration, service installation and storage-label generation. + +The SDK derives the storage floor from that worker metadata before allocation. +For example, a 4096 MiB image upper plus a 4096 MiB root reservation requires +8192 MiB total. Omitted storage uses the advertised floor; explicit requests or +limits below it fail before creation. Sandboxd checks the actual quota again. + +## Fresh creation + +Select the operator-provided profile: + +```python +from akernel_sdk import ComposeProject, DockerRuntime, Sandbox + +project = ComposeProject.prepare( + "./project", ["./project/docker-compose.yaml"], + runtime=DockerRuntime(profile="harbor-v1"), + network="bridge", output_dir="./artifacts/compose", +) +sandbox = Sandbox(**project.sandbox_options()) +try: + project.start_runtime(sandbox) + project.build(timeout=600) + project.pull(timeout=600) + project.up(wait_timeout=600) + print(project.service("main").exec("uname -s").stdout) +finally: + sandbox.kill() +``` + +Keep `output_dir` outside the input context. The SDK freezes inputs without +parsing managed YAML; official Compose resolves services after `start_runtime()`, +including native `!reset` and `!override` tags. Service handles become available +at that point. BuildKit receives the original Dockerfile and context through +Buildx remote. Normal build caches remain enabled unless `no_cache=True` is +explicitly requested. + +The runtime publishes digest-addressed images and prepares their ROFS roots. +The image bridge registers validated image aliases and committed snapshots in +inner containerd. Missing registrations fail instead of extracting replacement +lower layers into the sandbox's private upper. A partially completed startup +must not be replayed on the same project object; delete the owned sandbox and +retain its diagnostics and any cleanup errors. + +## Checkpoint and recovery + +Call `sandbox.create_snapshot()` externally. Sandboxd/runsc freezes and captures +the whole sandbox; application signals, Python and an idle model request are not +prerequisites. Persist the snapshot ID and `project.descriptor`. For a command +started with `ServiceHandle.start_command()`, persist its stable `command_id`. + +Fence the source before creating a running clone with +`Sandbox.from_snapshot(..., **project.sandbox_options())`. Preserve the upper +geometry, mounts, reverse tunnel and current network policy. Bind the restored +sandbox using `ComposeProject.attach(restored, descriptor)`, then call +`reconnect_network()` to update owned forwarding/NAT for the new outer TAP. +Reconnect captured commands with `restored.commands.connect(command_id)`. +Do not rerun builds, Compose up, Docker startup or the agent command, and do not +send SIGCONT as part of ordinary whole-sandbox restore. + +The snapshot does not include the Harbor client's memory, model proxy or +external services. Applications own consistency with those external systems. +Ordinary service exec and `start_command()` require no Python in the service. +The optional `start_process()`/`ServiceProcessHandle` API provides inner-process +inspection and signalling through a Python monitor in that service. + +A service-specific controlled TCP entry exposes an existing outer reverse tunnel +to a bridge container without rewriting HTTP, authentication headers or SSE. +After recovery, attach saved descriptors rather than creating new listeners. +See the [SDK networking API](../README.md#native-compose-networking-and-model-tunnels). + +## Retention, authorization and limits + +Managed snapshots retain node-local collection leases. The last live or retained +checkpoint reference releases unused mounts; retained snapshots therefore count +toward the node's ROFS budget. Restore validates the collection, profile and +current tenant authorization. Adding tenants does not invalidate compatible +snapshots; revoking a tenant denies restore. Changes to captured tool, bootstrap +or storage geometry require a snapshot compatibility and drain plan. + +Restore requires the retained collection on the same node and mount view. +Cross-node collection reconstruction, automatic Registry artifact GC and +replacement of a provider identity are outside this contract. Removing a local +mount does not authorize deletion of its Registry artifact. Keep referenced +runtime images and compatible complete runsc bundles available. + +Supported networking is default/custom IPv4 bridge networking with service +DNS/aliases and TCP/UDP userland port mappings. External TCP address migration, +IPv6-only egress, non-bridge drivers, remote build contexts, arbitrary +runtime-created images and full Docker content-store/export compatibility are +outside the supported scope. Scaled service process handles are rejected. + +## Client-managed runtime + +For an explicitly client-managed deployment, use +`prepare_compose_images()` and `DockerRuntime(bridge_binary=...)`, or +`ComposeProject.prepare(..., auto_images=True)`. The client discovers image +metadata and supplies mount options; node pull authorization remains separate. +These options are not needed with `DockerRuntime(profile=...)`. + +The helper must report `discover_schema: 2` through `--mode capabilities`. +Discovery confines Dockerfile, build-context and `.env` reads to the frozen +project and receives only explicitly supplied variables. External `include` / +`extends.file`, remote contexts and platform emulation are unsupported in this +client-managed path. See [the helper guide](../../../builder/image-bridge/README.md) +and [the client-managed example](../examples/compose_checkpoint.py). + +## Verification + +Run `make sdk-check`, the managed-runtime adapter unit tests and image-bridge Go +tests when changing these interfaces. Against an owned compatible endpoint, +verify unchanged task inputs, main and sidecar health, original process/memory +and filesystem recovery, and an independent verifier. Record failures and clean +up owned environments and snapshots even when an operation fails. + +The opt-in `tests/integration/test_compose_network.py` checks default/custom DNS, +network isolation, TCP/UDP mappings, controlled model HTTP/auth/SSE and existing +inner/published-port TCP sockets across two delete/restore cycles. Its endpoint +and runtime prerequisites are explicit; CI does not run it on a shared cluster. diff --git a/sdk/python/examples/compose_checkpoint.py b/sdk/python/examples/compose_checkpoint.py new file mode 100644 index 00000000..ceedaacc --- /dev/null +++ b/sdk/python/examples/compose_checkpoint.py @@ -0,0 +1,67 @@ +"""Run a Compose service process, snapshot it, and attach the original process. + +Requires an owned endpoint with the managed Compose runtime prerequisites. +Example: python sdk/python/examples/compose_checkpoint.py ./project \ + --file ./project/compose.yaml \ + --bridge /absolute/path/akernel-image-bridge --service worker \ + --command 'exec ./worker' +""" + +import argparse +import tempfile +from pathlib import Path + +from akernel_sdk import ComposeProject, DockerRuntime, Sandbox + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("directory", type=Path) + parser.add_argument("--file", action="append", required=True, type=Path) + parser.add_argument("--image", default=DockerRuntime.default_image) + parser.add_argument("--bridge", required=True, type=Path) + parser.add_argument("--service", required=True) + parser.add_argument("--command", required=True) + args = parser.parse_args() + snapshot = None + with tempfile.TemporaryDirectory(prefix="akernel-compose-example-") as output: + project = ComposeProject.prepare( + args.directory, + args.file, + runtime=DockerRuntime(bridge_binary=args.bridge, writable_mb=2048), + auto_images=True, + output_dir=Path(output), + ) + try: + with Sandbox( + image=args.image, + cpu=4000, + memory=8192, + storage_mb=30720, + failover=False, + **project.sandbox_options(), + ) as source: + project.start_runtime(source) + project.build() + project.pull() + project.up() + process = project.service(args.service).start_command(args.command) + # Sandboxd/runsc captures all processes without guest signals. + snapshot = source.create_snapshot(timeout=300) + project_state, command_id = project.descriptor, process.command_id + # Context exit acknowledges deletion before the clone continues. + with Sandbox.from_snapshot( + snapshot, failover=False, **project.sandbox_options() + ) as restored: + attached = ComposeProject.attach(restored, project_state) + attached.reconnect_network() + original = restored.commands.connect(command_id) + print("Restored original command", original.command_id) + print("Exit code:", original.wait(timeout=300)) + finally: + if snapshot is not None: + Sandbox.delete_snapshot(snapshot) + + +if __name__ == "__main__": + main() diff --git a/sdk/python/examples/managed_compose.py b/sdk/python/examples/managed_compose.py new file mode 100644 index 00000000..f8c3342e --- /dev/null +++ b/sdk/python/examples/managed_compose.py @@ -0,0 +1,31 @@ +"""Use an operator-authorized runtime profile; no client Docker configuration.""" + +import argparse + +from akernel_sdk import ComposeProject, DockerRuntime, Sandbox + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("context") + parser.add_argument("--profile", required=True) + args = parser.parse_args() + project = ComposeProject.prepare( + args.context, + [args.context + "/docker-compose.yaml"], + runtime=DockerRuntime(profile=args.profile), + network="bridge", + ) + sandbox = Sandbox(**project.sandbox_options()) + try: + project.start_runtime(sandbox) + project.build() + project.pull() + project.up() + print(project.ps()) + finally: + sandbox.kill() + + +if __name__ == "__main__": + main() diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml index 3cc916a7..4c0499c6 100644 --- a/sdk/python/pyproject.toml +++ b/sdk/python/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "akernel-sdk" -version = "0.2.0rc5" +version = "0.2.0rc6" description = "Python SDK and CLI for AKernel remote sandboxes" readme = "README.md" license = "Apache-2.0" @@ -30,6 +30,8 @@ classifiers = [ dependencies = [ "adx-environment==0.1.0rc4", "idna>=3,<4", + "httpx>=0.27,<1", + "PyYAML>=6,<7", "websockets>=10.0", "dockerfile-parse>=2.0.1", ] @@ -42,6 +44,7 @@ dev = [ "build>=1.2,<2", "mypy>=1.10,<2", "ruff>=0.11,<1", + "types-PyYAML>=6,<7", ] [project.scripts] diff --git a/sdk/python/tests/integration/test_compose_network.py b/sdk/python/tests/integration/test_compose_network.py new file mode 100644 index 00000000..fb38bbd2 --- /dev/null +++ b/sdk/python/tests/integration/test_compose_network.py @@ -0,0 +1,335 @@ +"""Opt-in managed DinD IPv4 network and original-process C/R regression. + +Requires an isolated compatible runtime (Docker 28 runsc flags, managed upper, +reflink and durable sync). Never updates a deployment. Set +AKERNEL_RUN_COMPOSE_TESTS=1, AKERNEL_TEST_DIND_IMAGE and +AKERNEL_TEST_IMAGE_BRIDGE_BINARY. The caller supplies ordinary SDK connection +variables. AKERNEL_TEST_EGRESS_HOST defaults to 1.1.1.1:80 and must be reachable. +""" + +import http.server +import json +import os +import shlex +import tempfile +import threading +import time +import unittest +from pathlib import Path + +import yaml + +from akernel_sdk import ( + ComposeProject, + DockerRuntime, + HttpReverseTunnel, + Sandbox, + ServiceProcessHandle, +) + +_ENABLED = os.environ.get("AKERNEL_RUN_COMPOSE_TESTS") == "1" +_IMAGE = os.environ.get("AKERNEL_TEST_DIND_IMAGE", "") +_BRIDGE = os.environ.get("AKERNEL_TEST_IMAGE_BRIDGE_BINARY", "") +_SSE = b'event: message_stop\ndata: {"type":"message_stop"}\n\n' +_SERVER = """ +import socket,threading +listener=socket.socket();listener.bind(('0.0.0.0',8000));listener.listen(64) +def echo(peer): + with peer: + while chunk:=peer.recv(65536):peer.sendall(chunk) +def udp(): + peer=socket.socket(socket.AF_INET,socket.SOCK_DGRAM);peer.bind(('0.0.0.0',8001)) + while True: + chunk,address=peer.recvfrom(65536);peer.sendto(chunk,address) +threading.Thread(target=udp,daemon=True).start() +while True: + peer,_=listener.accept();threading.Thread(target=echo,args=(peer,),daemon=True).start() +""" +_PROBE = """ +import socket,pathlib,time,os,json +peer=socket.create_connection(('api',8000),5);peer.settimeout(None);cookie=41 +while True: + if pathlib.Path('/tmp/mutate').exists():cookie=99 + peer.sendall(b'original');assert peer.recv(8)==b'original' + value={'pid':os.getpid(),'cookie':cookie,'local':peer.getsockname(), + 'remote':peer.getpeername(),'time':time.time()} + path=pathlib.Path('/tmp/witness.tmp');path.write_text(json.dumps(value)) + path.replace('/tmp/witness.json');time.sleep(.1) +""" + + +@unittest.skipUnless(_ENABLED and _IMAGE and _BRIDGE, "enable managed Compose tests") +class ComposeNetworkIntegrationTest(unittest.TestCase): + def setUp(self): + self.environments = [] + self.snapshots = [] + self.addCleanup(self.cleanup_remote) + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name) + self.requests = [] + requests = self.requests + + class Handler(http.server.BaseHTTPRequestHandler): + def do_POST(self): + requests.append( + (self.path, {"authorization": self.headers.get("Authorization")}) + ) + self.rfile.read(int(self.headers.get("content-length", "0"))) + if self.headers.get("x-api-key") != "trial-fixture": + self.send_error(401) + return + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Transfer-Encoding", "chunked") + self.end_headers() + for chunk in (_SSE[:13], _SSE[13:]): + self.wfile.write(f"{len(chunk):x}\r\n".encode() + chunk + b"\r\n") + self.wfile.flush() + self.wfile.write(b"0\r\n\r\n") + + def log_message(self, *_args): + pass + + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) + server.daemon_threads = True + threading.Thread(target=server.serve_forever, daemon=True).start() + self.addCleanup(server.server_close) + self.addCleanup(server.shutdown) + self.tunnel = HttpReverseTunnel(f"http://127.0.0.1:{server.server_port}") + context = self.root / "context" + context.mkdir() + services = { + name: {"image": "python:3.12-slim", "command": ["sleep", "infinity"]} + for name in ("main", "loadgen", "standard", "private") + } + services["main"].update( + networks={"default": {}, "custom": {"aliases": ["api"]}}, + ports=["127.0.0.1:18090:8000/tcp", "127.0.0.1:18091:8001/udp"], + ) + services["loadgen"]["networks"] = ["custom"] + services["private"]["networks"] = ["private"] + self.compose = context / "compose.yaml" + self.original = yaml.safe_dump( + { + "services": services, + "networks": { + "custom": {"ipam": {"config": [{"subnet": "172.28.41.0/24"}]}}, + "private": {"internal": True}, + }, + } + ) + self.compose.write_text(self.original) + self.project = ComposeProject.prepare( + context, + [self.compose], + output_dir=self.root / "prepared", + network="bridge", + auto_images=True, + runtime=DockerRuntime(bridge_binary=Path(_BRIDGE), writable_mb=2048), + ) + self.options = self.project.sandbox_options() + self.options["reverse_tunnel"] = self.tunnel + self.sandbox = Sandbox( + image=_IMAGE, + cpu=2000, + memory=4096, + idle_timeout=1200, + schedule_timeout=180, + storage_mb=8192, + **self.options, + ) + self.environments.append(self.sandbox.id) + self.project.start_runtime(self.sandbox) + self.project.up(wait_timeout=60) + for name in ("main", "private"): + self.project.service(name).start_process( + shlex.join(["python3", "-c", _SERVER]), shell="/bin/sh" + ) + # Confirm the negative isolation test has a real listening target. + self.assertEqual(self.python("private", self.echo("127.0.0.1")), "ok") + self.process = self.project.service("loadgen").start_process( + shlex.join(["python3", "-c", _PROBE]), shell="/bin/sh" + ) + self.endpoint = self.project.service("main").expose_tunnel(self.tunnel) + self.sandbox.commands.run( + shlex.join( + ["python3", "-c", _PROBE.replace("('api',8000)", "('127.0.0.1',18090)")] + ), + background=True, + ) + + def cleanup_remote(self): + for identity in self.environments: + Sandbox.delete(sandbox_id=identity) + for identity in self.snapshots: + Sandbox.delete_snapshot(identity) + + def python(self, service, script): + result = self.project.service(service).exec( + shlex.join(["python3", "-c", script]), timeout=15 + ) + self.assertEqual(result.exit_code, 0, result.stderr) + return json.loads(result.stdout) + + @staticmethod + def echo(host): + return ( + "import socket,json; " + f"s=socket.create_connection(({host!r},8000),3); " + "s.sendall(b'ok');print(json.dumps(s.recv(2).decode()))" + ) + + def witness(self): + started = time.time() + for _ in range(60): + value = self.python( + "loadgen", + "import pathlib; " + "p=pathlib.Path('/tmp/witness.json'); " + "print(p.read_text() if p.exists() else 'null')", + ) + if value and value.pop("time") > started: + return value + time.sleep(0.1) + self.fail("Existing inner TCP connection did not resume") + + def outer_witness(self): + started = time.time() + for _ in range(60): + result = self.sandbox.commands.run( + shlex.join( + [ + "python3", + "-c", + "import pathlib; p=pathlib.Path('/tmp/witness.json'); " + "print(p.read_text() if p.exists() else 'null')", + ] + ), + timeout=15, + check=True, + ) + value = json.loads(result.stdout) + if value and value.pop("time") > started: + return value + time.sleep(0.1) + self.fail("Existing published-port TCP connection did not resume") + + def check_network(self): + self.assertEqual(self.python("loadgen", self.echo("api")), "ok") + self.assertEqual(self.python("standard", self.echo("main")), "ok") + result = self.sandbox.commands.run( + shlex.join( + [ + "python3", + "-c", + """ +import socket,json +s=socket.create_connection(('127.0.0.1',18090),3);s.sendall(b'tcp');a=s.recv(3) +u=socket.socket(socket.AF_INET,socket.SOCK_DGRAM);u.settimeout(3) +u.sendto(b'udp',('127.0.0.1',18091)) +print(json.dumps([a.decode(),u.recv(3).decode()])) +""", + ] + ), + timeout=15, + check=True, + ) + self.assertEqual(json.loads(result.stdout), ["tcp", "udp"]) + for name in ("main", "private"): + endpoint = self.project.service(name).expose_tunnel(self.tunnel) + self.assertEqual( + self.python( + name, + "import urllib.request,json; " + f"r=urllib.request.Request({endpoint!r}+'/v1/messages?trace=1'," + "data=b'{}',headers={'x-api-key':'trial-fixture'," + "'Authorization':'Bearer opaque'}); " + "print(json.dumps(urllib.request.urlopen(r,timeout=5).read().decode()))", + ), + _SSE.decode(), + ) + host = os.environ.get("AKERNEL_TEST_EGRESS_HOST", "1.1.1.1") + self.assertTrue( + self.python( + "loadgen", + "import socket,json; " + f"s=socket.create_connection(({host!r},80),3);" + "print(json.dumps(bool(s.getsockname())));s.close()", + ) + ) + inspect = self.project.service("private").identity() + result = self.sandbox.commands.run( + shlex.join(["docker", "inspect", inspect["container_id"]]), check=True + ) + networks = json.loads(result.stdout)[0]["NetworkSettings"]["Networks"] + address = next(iter(networks.values()))["IPAddress"] + self.assertTrue( + self.python( + "main", + "import socket,json\ntry:\n " + f"s=socket.create_connection(({address!r},8000),1); print('false')" + "\nexcept (OSError,TimeoutError): print('true')", + ) + ) + # A different service must be rejected at TCP, even with valid model auth. + self.assertTrue( + self.python( + "loadgen", + "import socket,json,urllib.parse; " + f"u=urllib.parse.urlsplit({self.endpoint!r}); " + "s=socket.create_connection((u.hostname,u.port),3);s.settimeout(3)\n" + "try:\n s.sendall(b'GET / HTTP/1.0\\r\\n\\r\\n'); " + "print(json.dumps(s.recv(100)==b''))\n" + "except ConnectionResetError: print('true')", + ) + ) + + def test_native_network_and_two_restores(self): + self.project.service("private").expose_tunnel(self.tunnel) + identities = { + name: self.project.service(name).identity() + for name in ("main", "loadgen", "standard", "private") + } + before = self.witness() + published_before = self.outer_witness() + self.assertEqual(before["cookie"], 41) + self.check_network() + for _ in range(2): + snapshot = self.sandbox.create_snapshot() + self.snapshots.append(snapshot.snapshot_id) + self.project.service("loadgen").exec("touch /tmp/mutate") + self.assertEqual(self.witness()["cookie"], 99) + self.sandbox.kill() + self.sandbox = Sandbox.from_snapshot( + snapshot.snapshot_id, + idle_timeout=1200, + schedule_timeout=180, + **self.options, + ) + self.environments.append(self.sandbox.id) + self.project = ComposeProject.attach(self.sandbox, self.project.descriptor) + self.project.reconnect_network() + self.process = ServiceProcessHandle.attach( + self.project.service("loadgen"), self.process.descriptor + ) + self.assertEqual(self.witness(), before) + self.assertEqual(self.outer_witness(), published_before) + self.assertEqual( + {name: self.project.service(name).identity() for name in identities}, + identities, + ) + self.check_network() + self.assertEqual(self.compose.read_text(), self.original) + self.assertTrue( + all( + path == "/v1/messages?trace=1" + and headers.get("authorization") == "Bearer opaque" + for path, headers in self.requests + ), + self.requests, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/sdk/python/tests/unit/test_addresses.py b/sdk/python/tests/unit/test_addresses.py index ca2760db..f9ce79d7 100644 --- a/sdk/python/tests/unit/test_addresses.py +++ b/sdk/python/tests/unit/test_addresses.py @@ -16,11 +16,13 @@ import unittest from unittest.mock import patch +from akernel_sdk import Sandbox from akernel_sdk._addresses import ( api_endpoint_from_env, exec_endpoint_from_env, gateway_endpoint_from_env, ) +from akernel_sdk._backends import registry def endpoint_tuple(endpoint): @@ -97,10 +99,13 @@ def test_gateway_override_uses_scheme_default_port(self): ("https://gw.example.com:80", "https", 80, True), ) for address, scheme, port, tls in cases: - with self.subTest(address=address), patch.dict( - os.environ, - {"AKERNEL_GATEWAY_ADDRESS": address}, - clear=True, + with ( + self.subTest(address=address), + patch.dict( + os.environ, + {"AKERNEL_GATEWAY_ADDRESS": address}, + clear=True, + ), ): endpoint = gateway_endpoint_from_env() self.assertEqual( @@ -114,6 +119,43 @@ def test_missing_server_address_is_clear(self): with self.assertRaisesRegex(RuntimeError, "AKERNEL_SERVER_ADDRESS"): api_endpoint_from_env() + def test_unexpanded_variables_are_rejected_in_control_and_gateway_addresses(self): + for address in ( + "$AKERNEL_SERVER_ADDRESS", + "${AKERNEL_SERVER_ADDRESS}", + "https://$AKERNEL_SERVER_ADDRESS:443", + ): + for setting, resolve in ( + ("AKERNEL_SERVER_ADDRESS", api_endpoint_from_env), + ("AKERNEL_GATEWAY_ADDRESS", gateway_endpoint_from_env), + ): + with ( + self.subTest(setting=setting, address=address), + patch.dict(os.environ, {setting: address}, clear=True), + ): + with self.assertRaisesRegex(RuntimeError, "unexpanded"): + resolve() + + def test_invalid_gateway_fails_before_backend_creation(self): + with ( + patch.dict( + os.environ, + { + "AKERNEL_SERVER_ADDRESS": "https://api.example", + "AKERNEL_GATEWAY_ADDRESS": "$AKERNEL_SERVER_ADDRESS", + "AKERNEL_TOKEN": "test-token", + }, + clear=True, + ), + patch.object(registry, "_loaded_backend", None), + patch.object(registry, "_selected_backend", "adx"), + patch.object(registry, "_is_installed", return_value=True), + patch.object(registry, "new_backend") as create, + ): + with self.assertRaisesRegex(RuntimeError, "unexpanded"): + Sandbox(image="ubuntu:24.04") + create.assert_not_called() + if __name__ == "__main__": unittest.main() diff --git a/sdk/python/tests/unit/test_adx_backend.py b/sdk/python/tests/unit/test_adx_backend.py index b2ff66d3..434ceba8 100644 --- a/sdk/python/tests/unit/test_adx_backend.py +++ b/sdk/python/tests/unit/test_adx_backend.py @@ -12,11 +12,14 @@ # See the License for the specific language governing permissions and # limitations under the License. +import concurrent.futures import json import os import subprocess import tempfile +import threading import unittest +from dataclasses import replace from pathlib import Path from types import MappingProxyType, SimpleNamespace from unittest.mock import MagicMock, patch @@ -28,7 +31,7 @@ from akernel_sdk._addresses import Endpoint from akernel_sdk._backends import adx from akernel_sdk._backends.base import BackendConfig, SandboxSpec -from akernel_sdk._backends.errors import BackendOperationError +from akernel_sdk._backends.errors import BackendOperationError, TimeoutException from akernel_sdk.types import HttpReverseTunnel, Mount, NetworkPolicy, S3Config @@ -71,6 +74,238 @@ def setUp(self): token="secret", ) + def test_managed_default_reserves_floor_through_native_request_encoding(self): + key = "AKERNEL_RUNTIME_PROFILE_harbor-v1" + requests = [] + + def send(_client, request, **_kwargs): + requests.append(request) + if request.method == "GET": + return httpx.Response( + 200, + json={ + "items": [ + {"id": "node-a", "status": 0, "labels": {key: "8192"}}, + {"id": "node-b", "status": 0, "labels": {key: "12288"}}, + ] + }, + request=request, + ) + return httpx.Response( + 403, json={"message": "stop after encoding"}, request=request + ) + + with ( + patch.object(httpx.Client, "send", autospec=True, side_effect=send), + self.assertRaises(BackendOperationError), + ): + adx.AdxBackend(self.config).create( + _spec( + extra_config={"managedRuntimeProfile": "harbor-v1"}, + node_id="node-a", + ) + ) + self.assertEqual([r.method for r in requests], ["GET", "POST"]) + self.assertEqual( + str(requests[0].url), "https://api.example/api/environment/v1/resources" + ) + self.assertEqual(requests[0].headers["X-Auth"], "secret") + body = json.loads(requests[1].content) + self.assertEqual(body["storageMb"], 8192) + self.assertEqual( + body["scheduleAffinities"][0]["labelOps"], + [ + {"type": 0, "labelKey": key, "labelValues": ["8192"]}, + {"type": 0, "labelKey": "NODE_ID", "labelValues": ["node-a"]}, + ], + ) + + def test_managed_undersized_budgets_or_missing_nodes_never_create(self): + key = "AKERNEL_RUNTIME_PROFILE_harbor-v1" + for overrides in ( + {"storage_mb": 4097}, + {"storage_limit_mb": 4096}, + {"node_id": "absent"}, + ): + with self.subTest(overrides=overrides): + requests = [] + + def send(_client, request, _requests=requests, **_kwargs): + _requests.append(request) + return httpx.Response( + 200, + json={ + "items": [ + {"id": "node-a", "status": 0, "labels": {key: "8192"}} + ] + }, + request=request, + ) + + with ( + patch.object(httpx.Client, "send", autospec=True, side_effect=send), + self.assertRaises(ValueError), + ): + adx.AdxBackend(self.config).create( + _spec( + extra_config={"managedRuntimeProfile": "harbor-v1"}, + **overrides, + ) + ) + self.assertEqual([r.method for r in requests], ["GET"]) + + def test_managed_preflight_retains_authorization_errors(self): + def deny(_client, request, **_kwargs): + return httpx.Response( + 403, + json={ + "message": "denied", + "error": { + "code": "PERMISSION_DENIED", + "retry": "never", + "outcome": "not_started", + "requestId": "profile-test", + }, + }, + request=request, + ) + + with ( + patch.object(httpx.Client, "send", autospec=True, side_effect=deny), + self.assertRaises(BackendOperationError) as raised, + ): + adx.AdxBackend(self.config).create( + _spec(extra_config={"managedRuntimeProfile": "harbor-v1"}) + ) + self.assertEqual(raised.exception.code, "PERMISSION_DENIED") + self.assertEqual(raised.exception.request_id, "profile-test") + + def test_concurrent_managed_creates_share_geometry_without_merging_specs(self): + backend = adx.AdxBackend(self.config) + barrier = threading.Barrier(16) + calls = [] + key = "AKERNEL_RUNTIME_PROFILE_harbor-v1" + + def send(_client, request, **_kwargs): + calls.append(request) + return httpx.Response( + 200, + json={ + "items": [ + {"id": "a", "status": 0, "labels": {key: "8192"}}, + {"id": "b", "status": 0, "labels": {key: "12288"}}, + ] + }, + request=request, + ) + + def resolve(index): + barrier.wait(timeout=5) + return backend._managed_storage( + _spec( + extra_config={"managedRuntimeProfile": "harbor-v1"}, + node_id="a" if index % 2 == 0 else "b", + ) + )[0].storage_mb + + with patch.object(httpx.Client, "send", autospec=True, side_effect=send): + with concurrent.futures.ThreadPoolExecutor(max_workers=16) as pool: + result = list(pool.map(resolve, range(16))) + self.assertEqual(result, [8192, 12288] * 8) + self.assertEqual(len(calls), 1) + + def test_managed_probe_expires_and_does_not_serve_stale_authorization(self): + backend = adx.AdxBackend(self.config) + spec = _spec(extra_config={"managedRuntimeProfile": "harbor-v1"}) + key = "AKERNEL_RUNTIME_PROFILE_harbor-v1" + clock = [0.0] + replies = [8192, None, 16384] + + def send(_client, request, **_kwargs): + floor = replies.pop(0) + if floor is None: + return httpx.Response(403, json={"message": "revoked"}, request=request) + return httpx.Response( + 200, + json={"items": [{"id": "a", "status": 0, "labels": {key: str(floor)}}]}, + request=request, + ) + + with ( + patch.object(httpx.Client, "send", autospec=True, side_effect=send), + patch.object(adx.time, "monotonic", side_effect=lambda: clock[0]), + ): + self.assertEqual(backend._managed_storage(spec)[0].storage_mb, 8192) + clock[0] = 0.5 + self.assertEqual(backend._managed_storage(spec)[0].storage_mb, 8192) + clock[0] = 1.0 + with self.assertRaises(BackendOperationError): + backend._managed_storage(spec) + self.assertEqual(backend._managed_storage(spec)[0].storage_mb, 16384) + self.assertEqual(replies, []) + + def test_managed_probe_isolated_by_authenticated_backend(self): + key = "AKERNEL_RUNTIME_PROFILE_harbor-v1" + tokens = [] + + def send(_client, request, **_kwargs): + token = request.headers["X-Auth"] + tokens.append(token) + floor = 8192 if token == "secret" else 12288 + return httpx.Response( + 200, + json={"items": [{"id": "a", "status": 0, "labels": {key: str(floor)}}]}, + request=request, + ) + + spec = _spec(extra_config={"managedRuntimeProfile": "harbor-v1"}) + with patch.object(httpx.Client, "send", autospec=True, side_effect=send): + first = adx.AdxBackend(self.config)._managed_storage(spec)[0] + second = adx.AdxBackend( + replace(self.config, token="other") + )._managed_storage(spec)[0] + self.assertEqual((first.storage_mb, second.storage_mb), (8192, 12288)) + self.assertEqual(tokens, ["secret", "other"]) + + def test_managed_probe_releases_followers_when_leader_is_interrupted(self): + backend = adx.AdxBackend(self.config) + entered = threading.Event() + release = threading.Event() + interruption = KeyboardInterrupt("cancel resource read") + + def send(_client, _request, **_kwargs): + entered.set() + self.assertTrue(release.wait(timeout=5)) + raise interruption + + with patch.object(httpx.Client, "send", autospec=True, side_effect=send): + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool: + leader = pool.submit(backend._managed_resource_nodes) + self.assertTrue(entered.wait(timeout=5)) + follower = pool.submit(backend._managed_resource_nodes) + release.set() + for result in [leader, follower]: + with self.assertRaises(KeyboardInterrupt): + result.result(timeout=5) + self.assertIsNone(backend._managed_resource_pending) + + def test_managed_probe_drops_inherited_process_state_before_locking(self): + backend = adx.AdxBackend(self.config) + backend._managed_resource_pid = -1 + backend._managed_resource_lock.acquire() + with patch.object( + httpx.Client, + "send", + autospec=True, + return_value=httpx.Response( + 200, + json={"items": []}, + request=httpx.Request("GET", "https://api.example"), + ), + ): + self.assertEqual(backend._managed_resource_nodes(), []) + self.assertEqual(backend._managed_resource_pid, os.getpid()) + def _failed_create_calls(self, stage, ownership): calls = [] @@ -83,24 +318,33 @@ def send(_client, request, **_kwargs): if action: if (stage == "cwd" and action == "file.make_dir") or stage == "route": return httpx.Response( - 403, json={"message": "initialization denied", "error": { - "code": "PERMISSION_DENIED", "retry": "never", - "outcome": "not_started", "requestId": "test", - }}, request=request, + 403, + json={ + "message": "initialization denied", + "error": { + "code": "PERMISSION_DENIED", + "retry": "never", + "outcome": "not_started", + "requestId": "test", + }, + }, + request=request, ) return httpx.Response(200, json={"processes": []}, request=request) payload = {"environmentId": "default-shared", "status": "running"} if ownership is not None: payload["newlyCreated"] = ownership return httpx.Response( - 200, headers={"content-type": "text/event-stream"}, + 200, + headers={"content-type": "text/event-stream"}, content="event: final\ndata: " + json.dumps(payload) + "\n\n", request=request, ) tunnel = ( HttpReverseTunnel(target="http://127.0.0.1:9000") - if stage == "tunnel" else None + if stage == "tunnel" + else None ) with ( patch.object(httpx.Client, "send", autospec=True, side_effect=send), @@ -108,11 +352,16 @@ def send(_client, request, **_kwargs): ): client.return_value.start.return_value = False with self.assertRaises(BackendOperationError): - adx.AdxBackend(self.config).create(_spec( - name="shared", detached=True, cpu=2000, memory=4096, - command_cwd="/proc/akernel-review" if stage == "cwd" else None, - reverse_tunnel=tunnel, - )) + adx.AdxBackend(self.config).create( + _spec( + name="shared", + detached=True, + cpu=2000, + memory=4096, + command_cwd="/proc/akernel-review" if stage == "cwd" else None, + reverse_tunnel=tunnel, + ) + ) return calls def test_failed_native_and_backend_initialization_respect_create_ownership(self): @@ -133,10 +382,13 @@ def deny(_client, request, **_kwargs): return httpx.Response( 403, json={ - "code": 403, "message": "host mount is not authorized", + "code": 403, + "message": "host mount is not authorized", "error": { - "code": "PERMISSION_DENIED", "retry": "never", - "outcome": "not_started", "requestId": "host-path-test", + "code": "PERMISSION_DENIED", + "retry": "never", + "outcome": "not_started", + "requestId": "host-path-test", }, }, request=request, @@ -175,7 +427,8 @@ def test_create_converts_mixed_mount_sources_without_changing_permissions(self): Mount(target="/inputs", host_path="/apps/inputs"), Mount(target="/tools", image_url="tools:v1"), Mount( - target="/weights", type="erofs", + target="/weights", + type="erofs", s3_config=S3Config("https://s3.example", "data", "weights.img"), ), ) @@ -189,9 +442,15 @@ def test_create_converts_mixed_mount_sources_without_changing_permissions(self): def test_create_preserves_public_error_context_and_cause(self): error = EnvironmentHTTPError( - 504, {}, "CPU cannot fit", code="DEADLINE_EXCEEDED", - retry="never", outcome="not_started", request_id="create-test", - operation_id="op-test", environment_id="default-test", + 504, + {}, + "CPU cannot fit", + code="DEADLINE_EXCEEDED", + retry="never", + outcome="not_started", + request_id="create-test", + operation_id="op-test", + environment_id="default-test", ) error.scheduling_failure = {"rejections": {"insufficient_cpu": 3}} with ( @@ -202,15 +461,24 @@ def test_create_preserves_public_error_context_and_cause(self): public = raised.exception self.assertIs(public.__cause__, error) for field in ( - "code", "retry", "outcome", "request_id", "operation_id", - "environment_id", "status_code", "scheduling_failure", + "code", + "retry", + "outcome", + "request_id", + "operation_id", + "environment_id", + "status_code", + "scheduling_failure", ): self.assertEqual(getattr(public, field), getattr(error, field), field) def test_unknown_create_retains_same_operation_guidance(self): error = adx.adx_environment.EnvironmentError( - "response lost", code="OUTCOME_UNKNOWN", retry="same_operation", - outcome="unknown", request_id="create-unknown", + "response lost", + code="OUTCOME_UNKNOWN", + retry="same_operation", + outcome="unknown", + request_id="create-unknown", environment_id="default-unknown", ) with ( @@ -223,6 +491,42 @@ def test_unknown_create_retains_same_operation_guidance(self): self.assertIsNone(raised.exception.status_code) self.assertIsNone(raised.exception.scheduling_failure) + def test_create_retains_first_native_failure_and_final_retry_guidance(self): + first = EnvironmentHTTPError( + 503, + {}, + "initial service failure", + code="SERVICE_UNAVAILABLE", + retry="same_operation", + outcome="unknown", + request_id="create-first", + ) + final = EnvironmentHTTPError( + 504, + {}, + "deadline expired during ownership lookup", + code="DEADLINE_EXCEEDED", + retry="same_operation", + outcome="unknown", + request_id="create-final", + environment_id="default-pending", + ) + final.first_error = first + with ( + patch.object(adx, "_OwnedSandbox", side_effect=final), + self.assertRaises(BackendOperationError) as raised, + ): + adx.AdxBackend(self.config).create(_spec()) + public = raised.exception + self.assertIs(public.first_error, first) + self.assertIs(public.__cause__, final) + self.assertEqual(public.code, "DEADLINE_EXCEEDED") + self.assertEqual(public.request_id, "create-final") + self.assertEqual(public.environment_id, "default-pending") + self.assertEqual(public.retry, "same_operation") + self.assertEqual(public.outcome, "unknown") + self.assertIs(adx._convert_error("cleanup", public).first_error, first) + def test_unstructured_error_does_not_invent_execution_outcome(self): error = adx._convert_error("create sandbox", RuntimeError("local failure")) self.assertIsNone(error.code) @@ -230,6 +534,7 @@ def test_unstructured_error_does_not_invent_execution_outcome(self): self.assertIsNone(error.outcome) self.assertIsNone(error.request_id) self.assertIsNone(error.environment_id) + self.assertIsNone(error.first_error) self.assertEqual( str(BackendOperationError("existing caller")), "existing caller" ) @@ -410,6 +715,51 @@ def test_wait_preserves_running_result_and_error_metadata(self): ) self.assertEqual(driver.wait(12, timeout=10).exit_code, 0) + def test_foreground_timeout_retains_partial_output_without_using_it_as_reason(self): + native = MagicMock() + native.run.return_value = SimpleNamespace( + stdout="partial stdout", + stderr="ordinary build progress, no diagnostic", + exit_code=None, + status=adx.adx_environment.CommandStatus.TIMED_OUT, + error_code="COMMAND_TIMEOUT", + error_message=None, + ) + driver = adx._CommandsDriver(native) + with self.assertRaisesRegex( + TimeoutException, "timed out after 600 seconds" + ) as caught: + driver.run("native build", envs=None, cwd=None, timeout=600) + error = caught.exception + self.assertEqual(error.code, "COMMAND_TIMEOUT") + self.assertIsNone(error.retry) + self.assertIsNone(error.outcome) + self.assertNotIn("ordinary build progress", str(error)) + self.assertIsNotNone(error.command_result) + self.assertEqual(error.command_result.stdout, "partial stdout") + self.assertEqual( + error.command_result.stderr, "ordinary build progress, no diagnostic" + ) + self.assertIsNone(error.command_result.exit_code) + + def test_foreground_wait_timeout_is_not_reported_as_a_terminal_exit(self): + native = MagicMock() + native.run.return_value = SimpleNamespace( + stdout="partial", + stderr="progress", + exit_code=None, + status=adx.adx_environment.CommandStatus.RUNNING, + error_code="WAIT_TIMEOUT", + error_message="observation expired", + ) + driver = adx._CommandsDriver(native) + with self.assertRaises(TimeoutException) as caught: + driver.run("native build", envs=None, cwd=None, timeout=600) + self.assertEqual(caught.exception.code, "WAIT_TIMEOUT") + self.assertIsNone(caught.exception.outcome) + self.assertIsNotNone(caught.exception.command_result) + self.assertEqual(caught.exception.command_result.status, "RUNNING") + def test_reload_waits_for_new_data_route_without_repeating_reload(self): backend = adx.AdxBackend(self.config) native = MagicMock(id="default-worker") diff --git a/sdk/python/tests/unit/test_command_compat.py b/sdk/python/tests/unit/test_command_compat.py index f7a27366..f7a71d01 100644 --- a/sdk/python/tests/unit/test_command_compat.py +++ b/sdk/python/tests/unit/test_command_compat.py @@ -13,8 +13,23 @@ class CommandCompatibilityTest(unittest.TestCase): - - + def test_checked_incomplete_result_retains_partial_output_and_status(self): + driver = MagicMock() + result = CommandResult( + "partial output", + "progress", + None, + status="RUNNING", + error_code="WAIT_TIMEOUT", + error_message="still running", + ) + driver.run.return_value = result + with self.assertRaises(TimeoutException) as raised: + Commands(driver).run("work", timeout=20, check=True) + self.assertIs(raised.exception.command_result, result) + self.assertEqual(raised.exception.code, "WAIT_TIMEOUT") + self.assertIsNone(raised.exception.outcome) + driver.run.assert_called_once() def test_nonzero_exit_remains_compatible_unless_explicitly_checked(self): driver = MagicMock() diff --git a/sdk/python/tests/unit/test_compose.py b/sdk/python/tests/unit/test_compose.py new file mode 100644 index 00000000..8e8b4b2c --- /dev/null +++ b/sdk/python/tests/unit/test_compose.py @@ -0,0 +1,337 @@ +"""Compose phases, immutable inputs, failure diagnostics and restore attachment.""" + +import json +import tempfile +import unittest +from pathlib import Path +from unittest.mock import MagicMock, patch + +from akernel_sdk import ComposeOperationError, ComposeProject, DockerRuntime +from akernel_sdk._backends.errors import BackendOperationError +from akernel_sdk.types import CommandResult + + +class ComposeTest(unittest.TestCase): + def setUp(self): + temp = tempfile.TemporaryDirectory() + self.addCleanup(temp.cleanup) + self.root = Path(temp.name) + self.context = self.root / "source" + self.context.mkdir() + (self.context / "Dockerfile").write_text("FROM scratch\n") + self.file = self.context / "docker-compose.yaml" + self.file.write_text( + "services:\n api:\n build: .\n load:\n image: alpine:3.20\n" + ) + self.sandbox = MagicMock(id="source") + self.sandbox.commands.run.return_value = CommandResult( + stdout="", stderr="", exit_code=0 + ) + + def prepare(self, **kwargs): + kwargs.setdefault("network", "host") + return ComposeProject.prepare( + self.context, [self.file], output_dir=self.root / "prepared", **kwargs + ) + + def test_service_background_command_needs_no_guest_monitor(self): + project = ComposeProject.attach(self.sandbox, self.prepare().descriptor) + handle = MagicMock(pid=37, command_id="original-command") + self.sandbox.commands.run.return_value = handle + result = project.service("api").start_command( + "exec native-worker 'quoted arg'", + cwd="/app", + user="worker", + envs={"VALUE": "a b"}, + shell="/bin/bash", + ) + self.assertIs(result, handle) + self.sandbox.commands.run.assert_called_once() + call = self.sandbox.commands.run.call_args + self.assertTrue(call.kwargs["background"]) + self.assertIn("exec -T -w /app -u worker", call.args[0]) + self.assertIn("api /bin/bash -c", call.args[0]) + self.assertNotIn("python", call.args[0]) + self.sandbox.files.write.assert_not_called() + + def test_service_background_submission_does_not_retry_unknown_outcome(self): + project = ComposeProject.attach(self.sandbox, self.prepare().descriptor) + failure = BackendOperationError( + "lost reply", outcome="unknown", request_id="accepted" + ) + self.sandbox.commands.run.side_effect = failure + with self.assertRaises(BackendOperationError) as raised: + project.service("api").start_command("worker") + self.assertIs(raised.exception, failure) + self.sandbox.commands.run.assert_called_once() + + def test_frozen_inputs_drive_both_discovery_and_remote_upload(self): + binary = self.root / "bridge" + binary.write_bytes(b"fixture") + images = MagicMock() + images.mounts.return_value = [] + with patch( + "akernel_sdk._compose.prepare_compose_images", return_value=images + ) as discover: + project = self.prepare( + auto_images=True, runtime=DockerRuntime(bridge_binary=binary) + ) + frozen = discover.call_args.args[0] + self.assertEqual((frozen / "Dockerfile").read_text(), "FROM scratch\n") + (self.context / "Dockerfile").write_text("FROM changed\n") + self.file.write_text("services: {different: {image: wrong}}") + with patch.object(DockerRuntime, "start"): + project.start_runtime(self.sandbox) + uploaded = Path(self.sandbox.files.copy_from_local.call_args.args[0]) + self.assertEqual((uploaded / "Dockerfile").read_text(), "FROM scratch\n") + staged = "".join(c.args[1] for c in self.sandbox.files.write.call_args_list) + self.assertIn("load:", staged) + self.assertNotIn("different:", staged) + self.assertNotIn("harbor", project.command(["ps"])) + + def test_unknown_build_outcome_is_not_replayed(self): + project = self.prepare() + with patch.object(DockerRuntime, "start"): + project.start_runtime(self.sandbox) + self.sandbox.commands.run.reset_mock() + failure = BackendOperationError( + "lost reply", outcome="unknown", request_id="same-request" + ) + self.sandbox.commands.run.side_effect = failure + with self.assertRaises(BackendOperationError) as raised: + project.build() + self.assertIs(raised.exception, failure) + self.assertEqual(self.sandbox.commands.run.call_count, 1) + + def test_build_and_up_are_separate_policy_boundaries(self): + project = self.prepare(buildkit=False) + with patch.object(DockerRuntime, "start"): + project.start_runtime(self.sandbox) + self.sandbox.commands.run.reset_mock() + project.build(no_cache=True) + commands = [c.args[0] for c in self.sandbox.commands.run.call_args_list] + self.assertEqual(len(commands), 1) + self.assertTrue(commands[0].startswith("DOCKER_BUILDKIT=0 ")) + self.assertIn("build --no-cache", commands[0]) + self.assertNotIn(" up ", commands[0]) + project.up() + self.assertIn("up -d --wait", self.sandbox.commands.run.call_args.args[0]) + + def test_automatic_images_verify_pinned_inputs_and_never_pull_them_again(self): + binary = self.root / "bridge" + binary.write_bytes(b"fixture") + images = MagicMock() + images.mounts.return_value = [] + original = ( + "services:\n api:\n build: .\n load:\n image: " + "registry.example/load:release@sha256:" + "a" * 64 + "\n" + ) + self.file.write_text(original) + with patch("akernel_sdk._compose.prepare_compose_images", return_value=images): + project = self.prepare( + auto_images=True, runtime=DockerRuntime(bridge_binary=binary) + ) + attached = ComposeProject.attach(self.sandbox, project.descriptor) + attached.pull() + self.assertIn("start.py --verify", self.sandbox.commands.run.call_args.args[0]) + self.sandbox.commands.run.reset_mock() + attached.build(no_cache=True) + attached.up() + commands = [c.args[0] for c in self.sandbox.commands.run.call_args_list] + self.assertIn("build --no-cache", commands[0]) + self.assertIn("start.py --verify", commands[1]) + self.assertIn("--pull never", commands[2]) + self.assertNotIn("--no-build", commands[2]) + self.assertFalse(any(" pull " in c for c in commands)) + self.assertEqual(self.file.read_text(), original) + self.assertEqual((self.root / "prepared/source-0.yaml").read_text(), original) + + def test_automatic_image_verification_failure_is_not_hidden_or_retried(self): + project = self.prepare() + descriptor = project.descriptor + descriptor["automatic_images"] = True + attached = ComposeProject.attach(self.sandbox, descriptor) + self.sandbox.commands.run.return_value = CommandResult( + stdout="", stderr="missing ROFS snapshot", exit_code=1 + ) + for phase in ("pull", "up"): + with self.subTest(phase=phase): + self.sandbox.commands.run.reset_mock() + with self.assertRaises(ComposeOperationError) as raised: + getattr(attached, phase)() + self.assertEqual(raised.exception.phase, phase) + self.assertEqual(raised.exception.stderr, "missing ROFS snapshot") + self.assertEqual(self.sandbox.commands.run.call_count, 1) + + def test_unmanaged_pull_keeps_native_compose_behavior(self): + project = ComposeProject.attach(self.sandbox, self.prepare().descriptor) + project.pull() + self.assertTrue( + self.sandbox.commands.run.call_args.args[0].endswith( + "pull --ignore-buildable" + ) + ) + + def test_missing_exit_status_is_an_unknown_operation_outcome(self): + failure = ComposeOperationError( + "build", CommandResult(stdout="partial", stderr="", exit_code=None) + ) + self.assertEqual(failure.outcome, "unknown") + + def test_zero_timeout_image_verification_waits_without_foreground_timeout(self): + project = self.prepare() + descriptor = project.descriptor + descriptor["automatic_images"] = True + project = ComposeProject.attach(self.sandbox, descriptor) + handle = self.sandbox.commands.run.return_value = MagicMock( + command_id="verify-id" + ) + handle.wait.side_effect = [ + CommandResult(stdout="", stderr="", exit_code=None), + CommandResult(stdout="verified", stderr="", exit_code=0), + ] + project.pull(timeout=0) + self.assertIs(self.sandbox.commands.run.call_args.kwargs["background"], True) + handle.kill.assert_not_called() + + def test_failed_cancel_keeps_original_timeout_and_command_identity(self): + from threading import Event + + from akernel_sdk import TimeoutException + + project = ComposeProject.attach(self.sandbox, self.prepare().descriptor) + handle = self.sandbox.commands.run.return_value = MagicMock( + command_id="owned-command" + ) + cleanup = RuntimeError("kill response lost") + handle.kill.side_effect = cleanup + cancellation = Event() + cancellation.set() + with self.assertRaises(TimeoutException) as raised: + project.run(["ps"], timeout=0, cancellation=cancellation) + self.assertEqual(raised.exception.command_id, "owned-command") + self.assertEqual(raised.exception.cleanup_errors, (cleanup,)) + + def test_failed_up_retains_output_and_collects_logs_without_restarting(self): + project = self.prepare() + with patch.object(DockerRuntime, "start"): + project.start_runtime(self.sandbox) + self.sandbox.commands.run.reset_mock() + self.sandbox.commands.run.side_effect = [ + CommandResult( + stdout="service partial", stderr="health failed", exit_code=7 + ), + CommandResult(stdout="diagnostic", stderr="", exit_code=0), + ] + with self.assertRaises(ComposeOperationError) as raised: + project.up() + self.assertEqual(raised.exception.phase, "up") + self.assertEqual(raised.exception.exit_code, 7) + self.assertEqual(raised.exception.stderr, "health failed") + self.assertEqual(self.sandbox.commands.run.call_count, 2) + self.assertEqual( + (self.root / "prepared/services.log").read_text(), "diagnostic" + ) + + def test_attach_rebinds_only_and_retains_managed_upper_geometry(self): + project = self.prepare(runtime=DockerRuntime(writable_mb=2048)) + with patch.object(DockerRuntime, "start"): + project.start_runtime(self.sandbox) + other = MagicMock(id="restored") + attached = ComposeProject.attach(other, project.descriptor) + other.commands.run.assert_not_called() + other.files.copy_from_local.assert_not_called() + self.assertEqual(attached.sandbox_options(), project.sandbox_options()) + with self.assertRaises(RuntimeError): + attached.start_runtime(other) + attached.service("load").exec("echo restored", user="user", envs={"A": "x y"}) + self.assertIn("exec -T -u user -e", other.commands.run.call_args.args[0]) + self.assertIn("load /bin/sh -c", other.commands.run.call_args.args[0]) + + def test_explicit_network_fails_before_remote_mutations(self): + self.file.write_text("services: {api: {networks: [private]}}") + with self.assertRaisesRegex(ValueError, "explicit networks"): + self.prepare() + self.sandbox.commands.run.assert_not_called() + + def test_service_identity_rejects_scaled_service(self): + project = self.prepare() + attached = ComposeProject.attach(self.sandbox, project.descriptor) + self.sandbox.commands.run.return_value = CommandResult( + stdout="one\ntwo\n", stderr="", exit_code=0 + ) + with self.assertRaisesRegex(BackendOperationError, "exactly one"): + attached.service("api").identity() + + def test_prebuilt_images_do_not_add_undeclared_services_or_builds(self): + images = MagicMock() + images.mounts.return_value = [] + project = self.prepare( + runtime=DockerRuntime(images=images), + prebuilt_images={"api": "registry/api:seed", "absent": "unused"}, + ) + other = ComposeProject.attach(self.sandbox, project.descriptor) + other.build() + self.sandbox.commands.run.assert_not_called() + other.up() + self.assertIn( + "--no-build --pull never", self.sandbox.commands.run.call_args.args[0] + ) + self.assertNotIn("absent", project.descriptor["services"]) + text = (self.root / "prepared/host.json").read_text() + self.assertNotIn('"build"', text) + + def test_custom_hostnames_are_preserved_in_compatibility_overlay(self): + self.file.write_text( + "services: {worker: {image: alpine, hostname: custom.example}}" + ) + self.prepare() + hosts = json.loads((self.root / "prepared/host.json").read_text())["services"] + self.assertEqual(hosts["worker"]["hostname"], "custom.example") + self.assertIn("custom.example:127.0.0.1", hosts["worker"]["extra_hosts"]) + + def test_partial_runtime_start_is_not_replayed(self): + project = self.prepare() + with patch.object( + DockerRuntime, "start", side_effect=BackendOperationError("partial") + ) as start: + with self.assertRaises(BackendOperationError): + project.start_runtime(self.sandbox) + with self.assertRaises(RuntimeError): + project.start_runtime(self.sandbox) + start.assert_called_once() + + def test_copy_cleanup_failure_does_not_replace_transfer_failure(self): + project = self.prepare() + attached = ComposeProject.attach(self.sandbox, project.descriptor) + failure = BackendOperationError("copy unavailable", request_id="transfer") + cleanup = BackendOperationError("cleanup unavailable", request_id="cleanup") + self.sandbox.files.copy_from_local.side_effect = failure + self.sandbox.files.remove.side_effect = cleanup + with self.assertRaises(BackendOperationError) as raised: + attached.service("api").upload("local", "/target") + self.assertIs(raised.exception, failure) + self.assertEqual(failure.cleanup_errors, (cleanup,)) + + def test_zero_timeout_waits_without_deadline_and_cancellation_keeps_identity(self): + from threading import Event + + from akernel_sdk import TimeoutException + + project = self.prepare() + attached = ComposeProject.attach(self.sandbox, project.descriptor) + handle = self.sandbox.commands.run.return_value = MagicMock( + command_id="stable-command" + ) + handle.wait.side_effect = [ + CommandResult(stdout="", stderr="", exit_code=None), + CommandResult(stdout="finished", stderr="", exit_code=0), + ] + self.assertEqual(attached.run(["ps"], timeout=0).stdout, "finished") + self.assertTrue(self.sandbox.commands.run.call_args.kwargs["background"]) + cancellation = Event() + cancellation.set() + with self.assertRaises(TimeoutException) as raised: + attached.run(["ps"], timeout=0, cancellation=cancellation) + self.assertEqual(raised.exception.command_id, "stable-command") + handle.kill.assert_called_once() diff --git a/sdk/python/tests/unit/test_compose_network.py b/sdk/python/tests/unit/test_compose_network.py new file mode 100644 index 00000000..16bee731 --- /dev/null +++ b/sdk/python/tests/unit/test_compose_network.py @@ -0,0 +1,285 @@ +"""Container-accessible tunnel ownership, restore rebinding and byte transport.""" + +import json +import socket +import tempfile +import threading +import unittest +from pathlib import Path +from unittest.mock import MagicMock, patch + +from akernel_sdk import ComposeProject, HttpReverseTunnel +from akernel_sdk import _docker_network_guest as guest +from akernel_sdk._backends.errors import BackendOperationError +from akernel_sdk._compose import ServiceHandle +from akernel_sdk.types import CommandResult + + +class ComposeNetworkTest(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + root = Path(temporary.name) + source = root / "source" + source.mkdir() + self.original = ( + "services: {worker: {image: test, networks: [private]}}\n" + "networks: {private: {internal: true}}\n" + ) + compose = source / "compose.yaml" + compose.write_text(self.original) + self.project = ComposeProject.prepare( + source, [compose], network="bridge", output_dir=root / "out" + ) + self.sandbox = MagicMock() + self.project._sandbox = self.sandbox + self.service = self.project.service("worker") + self.identity = {"container_id": "full-id", "started_at": "original-start"} + identity = patch.object( + ServiceHandle, "identity", return_value=self.identity.copy() + ) + self.mock_identity = identity.start() + self.addCleanup(identity.stop) + self.tunnel = HttpReverseTunnel("http://127.0.0.1:12345") + self.inspect = { + "HostConfig": {"NetworkMode": "project_private"}, + "NetworkSettings": { + "Networks": { + "project_private": { + "Gateway": "172.28.41.1", + "IPAddress": "172.28.41.2", + } + } + }, + } + self.sandbox.commands.run.side_effect = [ + CommandResult(stdout=json.dumps([self.inspect]), stderr="", exit_code=0), + CommandResult( + stdout='{"url":"http://172.28.41.1:19000"}', stderr="", exit_code=0 + ), + ] + + def test_native_prepare_has_no_network_overlay(self): + self.assertEqual(self.project._contents, [self.original]) + self.assertEqual(self.project.descriptor["network"], "bridge") + self.assertNotIn("127.0.0.1", "".join(self.project._contents)) + + def test_gateway_endpoint_is_peer_restricted_and_restore_never_relaunches(self): + url = self.service.expose_tunnel(self.tunnel) + self.assertEqual(url, "http://172.28.41.1:19000") + command = self.sandbox.commands.run.call_args.args[0] + self.assertIn("172.28.41.2", command) + descriptor = self.project.descriptor + self.assertNotIn(self.tunnel.target, json.dumps(descriptor)) + restored = MagicMock() + attached = ComposeProject.attach(restored, descriptor) + self.assertEqual(attached.service("worker").expose_tunnel(self.tunnel), url) + restored.commands.run.assert_not_called() + attached.reconnect_network() + self.assertIn("reconnect", restored.commands.run.call_args.args[0]) + self.assertNotIn("dockerd", restored.commands.run.call_args.args[0]) + self.assertEqual(restored.commands.run.call_count, 1) + + def test_missing_default_gateway_uses_internal_bridge_ipam(self): + self.inspect["NetworkSettings"]["Networks"]["project_private"]["Gateway"] = "" + self.sandbox.commands.run.side_effect = [ + CommandResult(stdout=json.dumps([self.inspect]), stderr="", exit_code=0), + CommandResult( + stdout=json.dumps( + [ + { + "IPAM": { + "Config": [ + { + "Subnet": "172.28.41.0/24", + "Gateway": "172.28.41.1", + } + ] + } + } + ] + ), + stderr="", + exit_code=0, + ), + CommandResult( + stdout='{"url":"http://172.28.41.1:19000"}', stderr="", exit_code=0 + ), + ] + self.assertEqual( + self.service.expose_tunnel(self.tunnel), "http://172.28.41.1:19000" + ) + + def test_host_mode_retains_loopback_without_starting_a_proxy(self): + self.inspect["HostConfig"]["NetworkMode"] = "host" + self.sandbox.commands.run.side_effect = [ + CommandResult(stdout=json.dumps([self.inspect]), stderr="", exit_code=0) + ] + self.assertEqual(self.service.expose_tunnel(self.tunnel), self.tunnel.url) + self.assertEqual(self.sandbox.commands.run.call_count, 1) + attached = ComposeProject.attach(MagicMock(), self.project.descriptor) + self.assertEqual( + attached.service("worker").expose_tunnel(self.tunnel), self.tunnel.url + ) + + def test_native_service_network_namespace_uses_owners_bridge(self): + shared = { + "HostConfig": {"NetworkMode": "container:api-id"}, + "NetworkSettings": {"Networks": {}}, + } + self.sandbox.commands.run.side_effect = [ + CommandResult(stdout=json.dumps([shared]), stderr="", exit_code=0), + CommandResult(stdout=json.dumps([self.inspect]), stderr="", exit_code=0), + CommandResult( + stdout='{"url":"http://172.28.41.1:19000"}', stderr="", exit_code=0 + ), + ] + self.assertEqual( + self.service.expose_tunnel(self.tunnel), "http://172.28.41.1:19000" + ) + self.assertIn( + "docker inspect api-id", self.sandbox.commands.run.call_args_list[1].args[0] + ) + self.assertIn("172.28.41.2", self.sandbox.commands.run.call_args.args[0]) + + def test_cyclic_namespace_owner_does_not_launch_relay(self): + shared = {"HostConfig": {"NetworkMode": "container:api-id"}} + self.sandbox.commands.run.side_effect = [ + CommandResult(stdout=json.dumps([shared]), stderr="", exit_code=0), + CommandResult(stdout=json.dumps([shared]), stderr="", exit_code=0), + ] + with self.assertRaisesRegex(ValueError, "Cyclic"): + self.service.expose_tunnel(self.tunnel) + self.assertEqual(self.sandbox.commands.run.call_count, 2) + + def test_runtime_binding_is_pure_and_preserves_restore_tunnel_fence(self): + fresh = ComposeProject.bind_runtime(self.sandbox, self.project.descriptor) + self.sandbox.commands.run.assert_not_called() + url = fresh.service("worker").expose_tunnel(self.tunnel) + restored = MagicMock() + attached = ComposeProject.attach(restored, fresh.descriptor) + self.assertEqual(attached.service("worker").expose_tunnel(self.tunnel), url) + restored.commands.run.assert_not_called() + + def test_stale_container_and_unknown_launch_are_never_replayed(self): + self.service.expose_tunnel(self.tunnel) + self.mock_identity.return_value = { + "container_id": "different", + "started_at": "new", + } + with self.assertRaisesRegex(BackendOperationError, "replaced"): + self.service.expose_tunnel(self.tunnel) + self.assertEqual(self.sandbox.commands.run.call_count, 2) + self.mock_identity.return_value = self.identity.copy() + record = self.project._descriptor["tunnel_endpoints"]["worker:8766"] + record.pop("url") + with self.assertRaises(BackendOperationError) as failure: + self.service.expose_tunnel(self.tunnel) + self.assertEqual(failure.exception.outcome, "unknown") + self.assertEqual(self.sandbox.commands.run.call_count, 2) + + def test_no_network_or_untrusted_listener_fails_closed(self): + self.inspect["NetworkSettings"]["Networks"] = {} + self.sandbox.commands.run.side_effect = [ + CommandResult(stdout=json.dumps([self.inspect]), stderr="", exit_code=0) + ] + with self.assertRaisesRegex(ValueError, "IPv4"): + self.service.expose_tunnel(self.tunnel) + self.assertEqual(self.sandbox.commands.run.call_count, 1) + + def test_snapshot_attachment_cannot_accidentally_bootstrap_a_missing_relay(self): + restored = MagicMock() + attached = ComposeProject.attach(restored, self.project.descriptor) + with self.assertRaisesRegex(RuntimeError, "existing tunnels"): + attached.service("worker").expose_tunnel(self.tunnel) + restored.commands.run.assert_not_called() + + +class RelayTransportTest(unittest.TestCase): + def test_http_auth_chunked_sse_and_half_close_are_preserved_byte_for_byte(self): + payload = ( + b"POST /v1/messages?trace=1 HTTP/1.1\r\n" + b"Authorization: Bearer opaque\r\nx-api-key: key\r\n\r\n" + b"x" * 200000 + ) + response = ( + b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\n" + b"Transfer-Encoding: chunked\r\n\r\n" + b"data: unchanged\n\n" * 10000 + ) + upstream = socket.socket() + upstream.bind(("127.0.0.1", 0)) + upstream.listen(1) + received = [] + + def server(): + peer, _ = upstream.accept() + with peer: + chunks = [] + while chunk := peer.recv(65536): + chunks.append(chunk) + received.append(b"".join(chunks)) + peer.sendall(response) + peer.shutdown(socket.SHUT_WR) + + thread = threading.Thread(target=server, daemon=True) + thread.start() + application, relay = socket.socketpair() + forwarding = threading.Thread( + target=guest.forward, args=(relay, upstream.getsockname()), daemon=True + ) + forwarding.start() + try: + application.settimeout(5) + application.sendall(payload) + application.shutdown(socket.SHUT_WR) + result = [] + while chunk := application.recv(65536): + result.append(chunk) + self.assertEqual(b"".join(result), response) + self.assertEqual(received, [payload]) + finally: + application.close() + upstream.close() + forwarding.join(timeout=5) + thread.join(timeout=5) + self.assertFalse(forwarding.is_alive()) + self.assertFalse(thread.is_alive()) + + +class EgressRebindTest(unittest.TestCase): + def test_new_tap_replaces_only_owned_chains_without_flushing_conntrack(self): + with tempfile.TemporaryDirectory() as directory: + state = Path(directory) / "network.json" + state.write_text( + json.dumps( + { + "uplink": "tap.old", + "source": "10.88.0.2", + "iptables": "/sbin/iptables-legacy", + } + ) + ) + with ( + patch.object(guest, "STATE", state), + patch.object( + guest.shutil, "which", side_effect=lambda name: "/sbin/" + name + ), + patch.object( + guest.subprocess, + "check_output", + return_value="1.1.1.1 via 10.88.0.1 dev tap.new src 10.88.0.3", + ), + patch.object(guest.subprocess, "run") as run, + ): + record = guest.configure(reconnect=True) + self.assertEqual(record["source"], "10.88.0.3") + run.assert_called_once() + self.assertIn("--noflush", run.call_args.args[0]) + rules = run.call_args.kwargs["input"] + self.assertIn("AKERNEL_DIND_NAT -o tap.new", rules) + self.assertNotIn("tap.old", rules) + self.assertNotIn("10.88.0.2", rules) + self.assertNotIn("DOCKER", rules) + self.assertEqual(json.loads(state.read_text()), record) + run.reset_mock() + guest.configure(reconnect=True) + run.assert_not_called() diff --git a/sdk/python/tests/unit/test_docker_runtime.py b/sdk/python/tests/unit/test_docker_runtime.py new file mode 100644 index 00000000..55ccd54d --- /dev/null +++ b/sdk/python/tests/unit/test_docker_runtime.py @@ -0,0 +1,396 @@ +"""Shared lower integrity, private upper restore config and one-time startup.""" + +import hashlib +import json +import subprocess +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock, patch + +import httpx + +from akernel_sdk import DockerRuntime, ImageMountPlan, Sandbox +from akernel_sdk import _image_resolver as resolver +from akernel_sdk import sandbox as public +from akernel_sdk._docker_runtime import _PREPARE_RUNTIME +from akernel_sdk._image_bridge_guest import verify_image_bridge + + +def digest(data): + return "sha256:" + hashlib.sha256(data).hexdigest() + + +class DockerRuntimeTest(unittest.TestCase): + def setUp(self): + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + self.directory = Path(directory.name) + + def plan(self): + blobs = [] + + def metadata(kind, value, media_type): + payload = json.dumps(value).encode() + name = kind + ".json" + (self.directory / name).write_bytes(payload) + blobs.append( + { + "file": name, + "descriptor": { + "digest": digest(payload), + "size": len(payload), + "mediaType": media_type, + }, + } + ) + return blobs[-1]["descriptor"] + + config = metadata( + "config", + { + "architecture": "amd64", + "os": "linux", + "rootfs": {"type": "layers", "diff_ids": ["sha256:" + "a" * 64]}, + }, + "application/vnd.oci.image.config.v1+json", + ) + metadata( + "manifest", + { + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "config": config, + "layers": [ + { + "digest": "sha256:" + "b" * 64, + "size": 123, + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + } + ], + }, + "application/vnd.oci.image.manifest.v1+json", + ) + target = blobs[1]["descriptor"] + reference = "registry.example/test@" + target["digest"] + image = { + "reference": reference, + "root": "/opt/akernel-images/" + target["digest"][7:], + "target": target, + "config": blobs[0]["descriptor"], + "blobs": blobs, + "chain_id": "sha256:" + "a" * 64, + } + path = self.directory / "plan.json" + path.write_text( + json.dumps({"images": [image], "services": {"main": reference}}) + ) + return path + + def test_old_helper_is_rejected_before_project_data_is_dispatched(self): + response = SimpleNamespace(returncode=0, stdout='{"references":[]}', stderr="") + with patch.object(resolver.subprocess, "run", return_value=response) as run: + with self.assertRaisesRegex(ValueError, "schema 2"): + resolver.prepare_compose_images( + self.directory, + [], + output_dir=self.directory / "output", + bridge_binary=self.directory / "bridge", + ) + self.assertEqual(run.call_count, 1) + self.assertEqual(run.call_args.args[0][-1], "capabilities") + self.assertNotIn("input", run.call_args.kwargs) + + def test_discovery_does_not_implicitly_inherit_client_credentials(self): + response = SimpleNamespace( + returncode=0, stdout='{"references":[],"discover_schema":2}', stderr="" + ) + with ( + patch.dict("os.environ", {"AKERNEL_TOKEN": "host-only-fixture"}), + patch.object(resolver.subprocess, "run", return_value=response) as run, + ): + plan = resolver.prepare_compose_images( + self.directory, + [], + output_dir=self.directory / "output", + bridge_binary=self.directory / "bridge", + ) + self.assertTrue(plan.automatic) + self.assertTrue(all(call.kwargs["env"] == {} for call in run.call_args_list)) + + def test_integrity_and_path_escape_rejected_before_sandbox_creation(self): + for defect in ("corrupt", "escape", "root"): + with self.subTest(defect=defect): + path = self.plan() + plan = json.loads(path.read_text()) + if defect == "corrupt": + (self.directory / "config.json").write_bytes(b"changed") + elif defect == "escape": + plan["images"][0]["blobs"][0]["file"] = "../private" + else: + plan["images"][0]["root"] += "different" + path.write_text(json.dumps(plan)) + with self.assertRaises(ValueError): + ImageMountPlan.load(path) + + def test_restore_options_retain_lower_and_upper_without_startup(self): + runtime = DockerRuntime( + images=ImageMountPlan.load(self.plan()), writable_mb=2048 + ) + backend = MagicMock() + backend.from_snapshot.return_value.id = "restored" + with patch.object(public, "load_backend", return_value=backend): + for factory in (Sandbox.from_snapshot, Sandbox.create): + options = runtime.sandbox_options() + sb = factory("snapshot", **options) + spec = backend.from_snapshot.call_args.args[0] + reference = runtime.images.images[0]["reference"] + self.assertEqual(spec.mounts[0].image_url, reference) + upper = options["extra_config"]["fileBackedTmpfsMounts"][0] + self.assertEqual( + spec.extra_config["fileBackedTmpfsMounts"][0]["sizeMB"], 2048 + ) + upper["sizeMB"] = 999 + self.assertEqual( + spec.extra_config["fileBackedTmpfsMounts"][0]["sizeMB"], 2048 + ) + backend.create.assert_not_called() + backend.from_snapshot.return_value.commands.run.assert_not_called() + sb.kill() + + def test_start_checks_return_codes_and_refuses_existing_daemon(self): + sandbox = MagicMock() + sandbox.commands.run.side_effect = RuntimeError("existing daemon") + with self.assertRaisesRegex(RuntimeError, "existing daemon"): + DockerRuntime().start(sandbox) + self.assertEqual(sandbox.commands.run.call_count, 1) + self.assertTrue(sandbox.commands.run.call_args.kwargs["check"]) + sandbox.files.copy_from_local.assert_not_called() + + def test_public_runtime_bootstrap_installs_missing_tools_before_bridge(self): + sandbox = MagicMock() + runtime = DockerRuntime() + with patch.object(DockerRuntime, "_bridge", return_value=[]) as bridge: + # Refusing an existing daemon is the first operation. A failed package + # preparation on a fresh sandbox must not start any image services. + sandbox.commands.run.side_effect = [ + SimpleNamespace(), + RuntimeError("bootstrap failed"), + ] + with self.assertRaisesRegex(RuntimeError, "bootstrap failed"): + runtime.start(sandbox) + bridge.assert_not_called() + self.assertIn("apk add --no-cache", sandbox.commands.run.call_args.args[0]) + self.assertTrue(sandbox.commands.run.call_args.kwargs["check"]) + + def test_bootstrap_reuses_ready_images_and_handles_public_alpine(self): + binaries = self.directory / "bin" + binaries.mkdir() + record = self.directory / "packages" + + def executable(name, body="exit 0"): + path = binaries / name + path.write_text("#!/bin/sh\n" + body + "\n") + path.chmod(0o755) + + for name in ("docker", "dockerd", "containerd", "ctr"): + executable(name) + executable("apk", 'printf "%s\\n" "$*" > "$RECORD"') + environment = {"PATH": str(binaries), "RECORD": str(record)} + result = subprocess.run( + ["/bin/sh", "-c", _PREPARE_RUNTIME], + env=environment, + capture_output=True, + text=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual( + record.read_text().strip(), + "add --no-cache bash python3 coreutils util-linux shadow tar xz", + ) + record.unlink() + for name in ( + "bash", + "python3", + "findmnt", + "mount", + "tar", + "xz", + "realpath", + "usermod", + ): + executable(name) + result = subprocess.run( + ["/bin/sh", "-c", _PREPARE_RUNTIME], + env=environment, + capture_output=True, + text=True, + check=False, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertFalse(record.exists()) + (binaries / "dockerd").unlink() + result = subprocess.run( + ["/bin/sh", "-c", _PREPARE_RUNTIME], + env=environment, + capture_output=True, + text=True, + check=False, + ) + self.assertEqual(result.returncode, 127) + self.assertIn("missing dockerd", result.stderr) + self.assertFalse(record.exists()) + + def test_bridge_upload_is_sdk_owned_and_precedes_dockerd(self): + sandbox = MagicMock() + sandbox.commands.run.return_value = SimpleNamespace( + stdout='["--config-file", "/bridge/docker.json"]', stderr="" + ) + binary = self.directory / "bridge" + binary.write_bytes(b"fixture") + runtime = DockerRuntime( + images=ImageMountPlan.load(self.plan()), bridge_binary=binary + ) + runtime.start(sandbox, network="host") + commands = [c.args[0] for c in sandbox.commands.run.call_args_list] + start = next(i for i, c in enumerate(commands) if "start.py --root" in c) + docker = next(i for i, c in enumerate(commands) if "dockerd --config-file" in c) + self.assertLess(start, docker) + uploads = [c.args[0] for c in sandbox.files.copy_from_local.call_args_list] + self.assertTrue( + any(p.endswith("akernel_sdk/_image_bridge_guest.py") for p in uploads) + ) + self.assertFalse(any("harbor" in p for p in uploads)) + + def test_registry_auth_pins_metadata_and_never_downloads_layers(self): + config = json.dumps( + { + "os": "linux", + "architecture": "amd64", + "rootfs": {"diff_ids": ["sha256:" + "a" * 64]}, + } + ).encode() + manifest = json.dumps( + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "config": { + "mediaType": "application/vnd.oci.image.config.v1+json", + "digest": digest(config), + "size": len(config), + }, + "layers": [{"digest": "sha256:" + "b" * 64}], + } + ).encode() + calls = [] + + def handle(request): + calls.append(request) + if request.url.host == "auth.example": + return httpx.Response(200, json={"token": "test-only"}) + if request.headers.get("Authorization") != "Bearer test-only": + return httpx.Response( + 401, + headers={ + "www-authenticate": 'Bearer realm="https://auth.example/token",' + 'scope="repository:library/test:pull"' + }, + ) + if "/manifests/" in request.url.path: + return httpx.Response( + 200, + content=manifest, + headers={"docker-content-digest": digest(manifest)}, + ) + if request.url.path.endswith(digest(config)): + return httpx.Response(200, content=config) + raise AssertionError("must never request a layer") + + with patch.dict("os.environ", {"DOCKER_CONFIG": str(self.directory)}): + with httpx.Client(transport=httpx.MockTransport(handle)) as client: + image = resolver.RegistryMetadata(client, self.directory).prepare( + "docker.io/library/test:latest" + ) + self.assertEqual( + image["reference"], "docker.io/library/test@" + digest(manifest) + ) + self.assertEqual(len(image["blobs"]), 2) + self.assertEqual(sum(r.url.host == "auth.example" for r in calls), 1) + + def test_prepared_image_verification_checks_alias_digest_and_committed_root(self): + path = self.plan() + data = json.loads(path.read_text()) + image = data["images"][0] + alias = "registry.example/test:tag@" + image["target"]["digest"] + data["aliases"] = {alias: image["reference"]} + path.write_text(json.dumps(data)) + calls = [] + + def run(args, **kwargs): + calls.append(args) + self.assertTrue(kwargs["check"]) + if args[0] == "ctr": + payload = json.dumps( + { + "Kind": "Committed", + "Labels": { + "containerd.io/snapshot/agent-service.image-root": image[ + "root" + ] + }, + } + ) + else: + self.assertEqual(args[:3], ["docker", "image", "inspect"]) + payload = image["target"]["digest"] + "\n" + return SimpleNamespace(stdout=payload) + + with patch("akernel_sdk._image_bridge_guest.subprocess.run", side_effect=run): + verify_image_bridge(path) + self.assertEqual(calls[0][-1], image["chain_id"]) + self.assertEqual([c[-1] for c in calls[1:]], [image["reference"], alias]) + + def test_prepared_image_verification_rejects_missing_or_changed_inputs(self): + path = self.plan() + image = json.loads(path.read_text())["images"][0] + valid = json.dumps( + { + "Kind": "Committed", + "Labels": { + "containerd.io/snapshot/agent-service.image-root": image["root"] + }, + } + ) + for results, error_type in ( + ([subprocess.CalledProcessError(1, "ctr")], subprocess.CalledProcessError), + ( + [ + SimpleNamespace(stdout=valid), + subprocess.CalledProcessError(1, "docker"), + ], + subprocess.CalledProcessError, + ), + ( + [ + SimpleNamespace(stdout=valid), + SimpleNamespace(stdout="sha256:" + "0" * 64), + ], + RuntimeError, + ), + ( + [SimpleNamespace(stdout=valid.replace("Committed", "Active"))], + RuntimeError, + ), + ( + [SimpleNamespace(stdout=valid.replace(image["root"], "/other"))], + RuntimeError, + ), + ): + with self.subTest(results=results): + with patch( + "akernel_sdk._image_bridge_guest.subprocess.run", + side_effect=results, + ): + with self.assertRaises(error_type): + verify_image_bridge(path) diff --git a/sdk/python/tests/unit/test_fork.py b/sdk/python/tests/unit/test_fork.py index 91526576..18d94573 100644 --- a/sdk/python/tests/unit/test_fork.py +++ b/sdk/python/tests/unit/test_fork.py @@ -199,6 +199,9 @@ def setUp(self): self.backend = adx.AdxBackend(config) self.spec = SnapshotRestoreSpec( snapshot_id="source-origin", + extra_config={ + "fileBackedTmpfsMounts": [{"target": "/upper", "sizeMB": 64}] + }, idle_timeout=480, schedule_timeout=35, timeout=95, @@ -303,6 +306,7 @@ def test_one_capture_partial_results_inheritance_and_snapshot_cleanup(self): self.assertEqual(restored.timeout, 95) self.assertEqual(restored.command_cwd, "/work") self.assertEqual(restored.port_forwardings, (8080,)) + self.assertEqual(restored.extra_config, self.spec.extra_config) self.assertIs(restored.reverse_tunnel, self.spec.reverse_tunnel) self.assertTrue(restored.failover) self.assertFalse(restored.detached) diff --git a/sdk/python/tests/unit/test_image_plan.py b/sdk/python/tests/unit/test_image_plan.py new file mode 100644 index 00000000..445b0bab --- /dev/null +++ b/sdk/python/tests/unit/test_image_plan.py @@ -0,0 +1,139 @@ +"""OCI metadata/chain integrity and legitimate sharing of a merged root.""" + +import hashlib +import json +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock, patch + +from akernel_sdk import DockerRuntime, ImageMountPlan +from akernel_sdk._image_bridge_guest import verify_image_bridge + + +class ImagePlanTest(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.directory = Path(temporary.name) + self.chain = "sha256:" + "a" * 64 + + def image(self, label, *, architecture="amd64"): + blobs = [] + + def blob(value, media): + payload = json.dumps(value).encode() + digest = "sha256:" + hashlib.sha256(payload).hexdigest() + name = digest.replace(":", "-") + ".json" + (self.directory / name).write_bytes(payload) + descriptor = {"digest": digest, "size": len(payload), "mediaType": media} + blobs.append({"file": name, "descriptor": descriptor}) + return descriptor + + config = blob( + { + "architecture": architecture, + "os": "linux", + "config": {"Env": ["LABEL=" + label]}, + "rootfs": {"type": "layers", "diff_ids": [self.chain]}, + }, + "application/vnd.oci.image.config.v1+json", + ) + target = blob( + { + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "config": config, + "layers": [ + { + "digest": "sha256:" + "b" * 64, + "size": 123, + "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", + } + ], + }, + "application/vnd.oci.image.manifest.v1+json", + ) + return { + "reference": "registry.example/test@" + target["digest"], + "root": "/opt/akernel-images/" + target["digest"][7:], + "target": target, + "config": config, + "chain_id": self.chain, + "blobs": blobs, + } + + def write(self, images, **kwargs): + path = self.directory / "plan.json" + path.write_text( + json.dumps( + { + "images": images, + "services": {"main": images[0]["reference"]} if images else {}, + **kwargs, + } + ) + ) + return path + + def test_same_rootfs_with_different_config_digests_is_valid(self): + first, second = self.image("first"), self.image("second") + path = self.write([first, second]) + ImageMountPlan.load(path, bases_only=True) + + def run(args, **kwargs): + if args[0] == "ctr": + label = "containerd.io/snapshot/agent-service.image-root" + return SimpleNamespace( + stdout=json.dumps( + { + "Kind": "Committed", + "Labels": { + label: first["root"] + }, + } + ) + ) + image = next(i for i in (first, second) if i["reference"] == args[-1]) + return SimpleNamespace(stdout=image["target"]["digest"]) + + with patch("akernel_sdk._image_bridge_guest.subprocess.run", side_effect=run): + verify_image_bridge(path) + + def test_claimed_chain_must_match_the_selected_manifest_config(self): + image = self.image("first") + image["chain_id"] = "sha256:" + "c" * 64 + with self.assertRaises(ValueError): + ImageMountPlan.load(self.write([image])) + + def test_config_descriptor_must_be_reachable_from_target_manifest(self): + first, second = self.image("first"), self.image("second") + first["config"] = second["config"] + first["blobs"] += second["blobs"] + with self.assertRaises(ValueError): + ImageMountPlan.load(self.write([first])) + + def test_foreign_platform_cannot_supply_a_linux_amd64_root(self): + with self.assertRaises(ValueError): + ImageMountPlan.load(self.write([self.image("arm", architecture="arm64")])) + + def test_unused_manual_images_do_not_leave_unregistered_aliases(self): + first, second = self.image("first"), self.image("second") + aliases = { + "registry.example/first:tag": first["reference"], + "registry.example/second:tag": second["reference"], + } + plan = ImageMountPlan.load(self.write([first, second], aliases=aliases)) + self.assertEqual( + plan.aliases, {"registry.example/first:tag": first["reference"]} + ) + + def test_scratch_only_preparation_retains_empty_automatic_plan_marker(self): + path = self.write([], automatic=True) + plan = ImageMountPlan.load(path, bases_only=True) + sandbox = MagicMock() + sandbox.commands.run.return_value = SimpleNamespace(stdout="[]", stderr="") + DockerRuntime(images=plan)._bridge(sandbox) + payload = json.loads(sandbox.files.write.call_args.args[1]) + self.assertIs(payload.get("automatic"), True) diff --git a/sdk/python/tests/unit/test_managed_compose.py b/sdk/python/tests/unit/test_managed_compose.py new file mode 100644 index 00000000..adf5cfdf --- /dev/null +++ b/sdk/python/tests/unit/test_managed_compose.py @@ -0,0 +1,165 @@ +"""Managed projects use runtime operations; restore never repeats startup.""" + +import json +import tempfile +import unittest +from pathlib import Path +from unittest.mock import MagicMock, patch + +from akernel_sdk import ( + ComposeProject, + DockerRuntime, + HttpReverseTunnel, + TimeoutException, +) +from akernel_sdk.types import CommandResult + + +class ManagedCompose(unittest.TestCase): + def test_managed_build_timeout_preserves_output_without_replaying_build(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "context" + source.mkdir() + definition = source / "docker-compose.yaml" + definition.write_text("services: {main: {build: .}}") + project = ComposeProject.prepare( + source, + [definition], + runtime=DockerRuntime(profile="harbor-v1"), + output_dir=root / "output", + ) + sandbox = MagicMock() + sandbox.commands.run.return_value = CommandResult('["main"]', "", 0) + with ( + patch.object(DockerRuntime, "start"), + patch("akernel_sdk._managed_build.upload_context"), + ): + project.start_runtime(sandbox) + failure = TimeoutException( + "execution deadline", + code="COMMAND_TIMEOUT", + command_result=CommandResult("partial output", "partial error", None), + ) + sandbox.commands.run.reset_mock() + sandbox.commands.run.side_effect = failure + with self.assertRaises(TimeoutException) as caught: + project.build() + self.assertIs(caught.exception, failure) + self.assertEqual( + (root / "output/build.log").read_text(), "partial outputpartial error" + ) + sandbox.commands.run.assert_called_once() + + def test_managed_project_delegates_build_and_register_and_restores_handles(self): + with tempfile.TemporaryDirectory() as temporary: + context = Path(temporary) / "context" + context.mkdir() + definition = context / "docker-compose.yaml" + definition.write_text( + "services: {main: {build: .}, api: {image: ubuntu:24.04}}\n" + ) + project = ComposeProject.prepare( + context, + [definition], + network="bridge", + runtime=DockerRuntime(profile="harbor-v1"), + ) + self.assertEqual( + project.sandbox_options()["extra_config"], + {"managedRuntimeProfile": "harbor-v1"}, + ) + self.assertEqual(project.sandbox_options()["mounts"], []) + sandbox = MagicMock() + sandbox.commands.run.return_value = CommandResult( + stdout='["main", "api"]', stderr="", exit_code=0 + ) + with ( + patch.object(DockerRuntime, "start"), + patch("akernel_sdk._managed_build.upload_context") as upload, + ): + project.start_runtime(sandbox) + upload.assert_called_once() + project.build() + command = sandbox.commands.run.call_args.args[0] + self.assertIn("/usr/lib/akernel/compose.py build", command) + self.assertNotIn("--no-cache", command) + attached = project.descriptor + attached["files"].append("/harbor/environment.build/runtime-images.yaml") + sandbox.commands.run.return_value = CommandResult( + stdout=json.dumps(attached), stderr="", exit_code=0 + ) + project.pull() + restored = MagicMock() + captured = ComposeProject.attach(restored, project.descriptor) + with self.assertRaises(RuntimeError): + captured.start_runtime(restored) + captured.reconnect_network() + self.assertEqual(restored.commands.run.call_count, 1) + self.assertEqual( + restored.commands.run.call_args.args[0], + "python3 /usr/lib/akernel/compose.py reconnect", + ) + + def test_managed_prepare_preserves_native_merge_tags_without_yaml_parsing(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "project" + source.mkdir() + base = source / "base.yaml" + override = source / "override.yaml" + base.write_text('services: {main: {image: alpine:3, ports: ["8080:80"]}}') + override.write_text( + "services: {main: {ports: !reset [], environment: !override {A: B}}}" + ) + with patch( + "akernel_sdk._compose._services", + side_effect=AssertionError("client parser used"), + ): + project = ComposeProject.prepare( + source, + [base, override], + runtime=DockerRuntime(profile="harbor-v1"), + output_dir=root / "out", + ) + self.assertEqual(project.descriptor["services"], []) + self.assertEqual( + (root / "out/source-1.yaml").read_text(), override.read_text() + ) + with self.assertRaisesRegex(RuntimeError, "start_runtime"): + project.service("main") + sandbox = MagicMock() + sandbox.commands.run.return_value = CommandResult( + stdout='["main"]', stderr="", exit_code=0 + ) + with ( + patch.object(DockerRuntime, "start"), + patch("akernel_sdk._managed_build.upload_context"), + ): + project.start_runtime(sandbox) + self.assertEqual(project.descriptor["services"], ["main"]) + self.assertIn("compose.py describe", sandbox.commands.run.call_args.args[0]) + self.assertEqual(project.service("main").name, "main") + + def test_restored_project_never_creates_replacement_tunnel(self): + descriptor = { + "schema_version": 1, + "project_name": "test", + "services": ["main"], + "directory": "/project", + "files": [], + "managed_runtime_profile": "harbor-v1", + } + sandbox = MagicMock() + project = ComposeProject.attach(sandbox, descriptor) + with ( + patch( + "akernel_sdk._compose.ServiceHandle.identity", + return_value={"container_id": "same", "started_at": "same"}, + ), + self.assertRaises(RuntimeError), + ): + project.service("main").expose_tunnel( + HttpReverseTunnel("http://127.0.0.1:12345") + ) + sandbox.commands.run.assert_not_called() diff --git a/sdk/python/tests/unit/test_managed_runtime_profile.py b/sdk/python/tests/unit/test_managed_runtime_profile.py new file mode 100644 index 00000000..0b79a5a0 --- /dev/null +++ b/sdk/python/tests/unit/test_managed_runtime_profile.py @@ -0,0 +1,220 @@ +import json +import tarfile +import tempfile +import unittest +from pathlib import Path +from unittest.mock import MagicMock, patch + +from akernel_sdk import DockerRuntime, TimeoutException, build_image +from akernel_sdk._managed_build import upload_context +from akernel_sdk.types import CommandResult + + +class ManagedProfileContract(unittest.TestCase): + def test_command_timeout_preserves_partial_build_log_and_cleanup(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "Dockerfile").write_text("FROM scratch\n") + sandbox = MagicMock(id="owned-helper") + failure = TimeoutException( + "execution deadline", + code="COMMAND_TIMEOUT", + command_result=CommandResult("stdout", "stderr", None), + ) + sandbox.commands.run.side_effect = failure + with ( + patch("akernel_sdk.sandbox.Sandbox", return_value=sandbox), + patch("akernel_sdk._managed_build.upload_context"), + self.assertRaises(TimeoutException) as caught, + ): + build_image(root, profile="harbor-v1", log_path=root / "build.log") + self.assertIs(caught.exception, failure) + self.assertEqual((root / "build.log").read_text(), "stdoutstderr") + sandbox.kill.assert_called_once() + + def test_partial_log_write_failure_cannot_replace_command_timeout(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "Dockerfile").write_text("FROM scratch\n") + sandbox = MagicMock(id="owned-helper") + failure = TimeoutException( + "execution deadline", command_result=CommandResult("partial", "", None) + ) + sandbox.commands.run.side_effect = failure + with ( + patch("akernel_sdk.sandbox.Sandbox", return_value=sandbox), + patch("akernel_sdk._managed_build.upload_context"), + patch.object(Path, "write_text", side_effect=OSError("disk full")), + self.assertRaises(TimeoutException) as caught, + ): + build_image(root, profile="harbor-v1", log_path=root / "build.log") + self.assertIs(caught.exception, failure) + if hasattr(failure, "add_note"): + self.assertTrue( + any("partial build log" in note for note in failure.__notes__) + ) + sandbox.kill.assert_called_once() + + def test_invalid_preparation_requests_fail_before_creating_a_sandbox(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "Dockerfile").write_text("FROM scratch\n") + for options in ( + {"cpu_request": 0}, + {"cpu_request": True}, + {"cpu_request": 1001}, + {"memory_request": 2049}, + {"memory_request": 1.5}, + ): + with ( + self.subTest(options=options), + patch("akernel_sdk.sandbox.Sandbox") as factory, + self.assertRaisesRegex(ValueError, "request must be an integer"), + ): + build_image(root, profile="harbor-v1", **options) + factory.assert_not_called() + + def test_preparation_requests_preserve_existing_limits(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "Dockerfile").write_text("FROM scratch\n") + sandbox = MagicMock() + sandbox.commands.run.side_effect = RuntimeError("stop after admission") + with patch("akernel_sdk.sandbox.Sandbox", return_value=sandbox) as factory: + with self.assertRaisesRegex(RuntimeError, "stop after admission"): + build_image( + root, profile="harbor-v1", cpu_request=10, memory_request=64 + ) + options = factory.call_args.kwargs + self.assertEqual((options["cpu"], options["memory"]), (10, 64)) + self.assertEqual((options["cpu_limit"], options["mem_limit"]), (1000, 2048)) + sandbox.kill.assert_called_once() + + def test_only_capability_name_is_submitted(self): + options = DockerRuntime(profile="harbor-v1").sandbox_options() + self.assertEqual( + options, {"extra_config": {"managedRuntimeProfile": "harbor-v1"}} + ) + self.assertNotIn("mounts", options) + for value in ["../node", "http://node", ""]: + with self.assertRaises(ValueError): + DockerRuntime(profile=value) + + def test_context_archive_preserves_links_modes_and_empty_directories(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "empty").mkdir() + (root / "file").write_text("data") + (root / "file").chmod(0o640) + (root / "link").symlink_to("file") + sandbox = MagicMock() + + def inspect(source, _target): + with tarfile.open(source) as archive: + self.assertTrue(archive.getmember("./link").issym()) + self.assertEqual(archive.getmember("./file").mode, 0o640) + self.assertTrue(archive.getmember("./empty").isdir()) + + sandbox.files.copy_from_local.side_effect = inspect + upload_context(sandbox, root, "/work/context") + + def test_image_build_needs_no_client_registry_or_node_configuration(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "Dockerfile").write_text("FROM scratch\n") + sandbox = MagicMock() + sandbox.id = "owned-builder" + handle = { + "reference": "private.test/task@sha256:" + "a" * 64, + "image_process": { + "Env": ["MODE=native"], + "WorkingDir": "/app", + "User": "1000", + "Cmd": ["/bin/bash"], + }, + } + + def execute(command, **_kwargs): + return CommandResult( + stdout=json.dumps(handle) + if command == "cat /work/artifact.json" + else "", + stderr="", + exit_code=0, + ) + + sandbox.commands.run.side_effect = execute + with patch("akernel_sdk.sandbox.Sandbox", return_value=sandbox) as factory: + result = build_image(root, profile="harbor-v1") + self.assertEqual( + factory.call_args.kwargs["extra_config"], + {"managedRuntimeProfile": "harbor-v1"}, + ) + self.assertNotIn("mounts", factory.call_args.kwargs) + self.assertNotIn("image", factory.call_args.kwargs) + self.assertEqual(result.envs, {"MODE": "native"}) + self.assertEqual(result.cwd, "/app") + self.assertTrue(result.has_startup_command) + sandbox.kill.assert_called_once() + + def test_effective_empty_and_nonempty_image_startup(self): + for process, expected in [ + ({}, False), + ({"Entrypoint": [], "Cmd": None}, False), + ({"Entrypoint": None, "Cmd": []}, False), + ({"Entrypoint": ["/server"], "Cmd": []}, True), + ({"Entrypoint": [], "Cmd": ["/bin/bash"]}, True), + ]: + with ( + self.subTest(process=process), + tempfile.TemporaryDirectory() as directory, + ): + root = Path(directory) + (root / "Dockerfile").write_text("FROM scratch\nENTRYPOINT []\n") + sandbox = MagicMock() + handle = { + "reference": "private.test/task@sha256:" + "a" * 64, + "image_process": process, + } + + def execute(command, _handle=handle, **_kwargs): + return CommandResult( + stdout=json.dumps(_handle) + if command == "cat /work/artifact.json" + else "", + stderr="", + exit_code=0, + ) + + sandbox.commands.run.side_effect = execute + with patch("akernel_sdk.sandbox.Sandbox", return_value=sandbox): + result = build_image(root, profile="harbor-v1") + self.assertIs(result.has_startup_command, expected) + sandbox.kill.assert_called_once() + + def test_failed_build_retains_primary_error_and_cleanup_identity(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + (root / "Dockerfile").write_text("FROM scratch\n") + sandbox = MagicMock() + sandbox.id = "owned-builder" + primary = RuntimeError("upstream build failed") + + def execute(command, **_kwargs): + if "image-build" in command: + raise primary + return CommandResult(stdout="", stderr="", exit_code=0) + + sandbox.commands.run.side_effect = execute + sandbox.kill.side_effect = RuntimeError("cleanup unavailable") + with ( + patch("akernel_sdk.sandbox.Sandbox", return_value=sandbox), + self.assertRaises(RuntimeError) as error, + ): + build_image(root, profile="harbor-v1") + self.assertIs(error.exception, primary) + self.assertTrue(getattr(error.exception, "cleanup_errors", ())) + + +if __name__ == "__main__": + unittest.main() diff --git a/sdk/python/tests/unit/test_service_process.py b/sdk/python/tests/unit/test_service_process.py new file mode 100644 index 00000000..5ca4ec14 --- /dev/null +++ b/sdk/python/tests/unit/test_service_process.py @@ -0,0 +1,217 @@ +"""Exercise the packaged service helper with real Linux processes and signals.""" + +import ctypes +import json +import os +import shlex +import shutil +import signal +import subprocess +import sys +import tempfile +import time +import unittest +from pathlib import Path +from types import SimpleNamespace + +from akernel_sdk import BackendOperationError, ServiceProcessHandle, TimeoutException +from akernel_sdk.types import CommandResult + + +class LocalService: + name = "worker" + project = SimpleNamespace(descriptor={"project_name": "sdk-test"}) + + def __init__(self): + self.current = {"container_id": "c" * 64, "started_at": "original"} + self.lose_start_reply = False + self.tokens = set() + self.identities = {} + + def identity(self): + return dict(self.current) + + def exec(self, command, **kwargs): + result = subprocess.run( + ["/bin/sh", "-c", command], + cwd=kwargs.get("cwd"), + env={**os.environ, **(kwargs.get("envs") or {})}, + text=True, + capture_output=True, + timeout=15, + ) + # Capture only this test's exclusive process directories for cleanup. + import re + + self.tokens.update(re.findall(r"\b[0-9a-f]{32}\b", command)) + for token in self.tokens: + file = Path("/tmp/akernel-service-process-" + token) / "identity.json" + if file.exists() and token not in self.identities: + self.identities[token] = json.loads(file.read_text()) + if self.lose_start_reply: + self.lose_start_reply = False + raise TimeoutException("response lost", outcome="unknown") + return CommandResult( + stdout=result.stdout, stderr=result.stderr, exit_code=result.returncode + ) + + +@unittest.skipUnless( + sys.platform.startswith("linux"), "Requires Linux /proc and subreaper" +) +class ServiceProcessTest(unittest.TestCase): + def setUp(self): + self.libc = ctypes.CDLL(None) + previous = ctypes.c_int() + self.libc.prctl(37, ctypes.byref(previous), 0, 0, 0) + self.previous_subreaper = previous.value + self.libc.prctl(36, 1, 0, 0, 0) + self.temp = tempfile.TemporaryDirectory() + self.service = LocalService() + + def tearDown(self): + for token in self.service.tokens: + root = Path("/tmp/akernel-service-process-" + token) + if not (root / "identity.json").exists(): + continue + identity = self.service.identities[token] + try: + os.kill(identity["pid"], signal.SIGKILL) + except ProcessLookupError: + pass + # The helper monitor is an adopted child, not an unrelated process. + try: + os.waitpid(identity["monitor_pid"], 0) + except ChildProcessError: + pass + shutil.rmtree(root) + self.temp.cleanup() + self.libc.prctl(36, self.previous_subreaper, 0, 0, 0) + + def start(self): + command = "exec " + sys.executable + " -c 'import time;time.sleep(60)'" + return ServiceProcessHandle.start(self.service, command, cwd=self.temp.name) + + def test_freeze_attach_and_continue_original_process(self): + handle = self.start() + pid = handle.pid + stopped = handle.signal("stop") + self.assertTrue(stopped) + self.assertTrue(all(p["state"] in ("T", "t") for p in stopped)) + attached = ServiceProcessHandle.attach(self.service, handle.descriptor) + self.assertEqual(attached.pid, pid) + self.assertEqual(attached.process_id, handle.process_id) + attached.signal("continue") + self.assertNotEqual(attached.inspect()[0]["state"], "T") + + def test_helper_ignores_task_import_hooks_and_preserves_child_environment(self): + root = Path(self.temp.name) + (root / "sitecustomize.py").write_text("print('task startup hook')\n") + (root / "json.py").write_text("raise RuntimeError('task shadows json')\n") + witness = root / "child-environment" + script = ( + "import os,pathlib,time; pathlib.Path(" + + repr(str(witness)) + + ").write_text(os.environ['APP_VALUE']); time.sleep(60)" + ) + command = "exec " + shlex.quote(sys.executable) + " -c " + shlex.quote(script) + handle = ServiceProcessHandle.start( + self.service, + command, + cwd=self.temp.name, + envs={"PYTHONPATH": self.temp.name, "APP_VALUE": "user-unchanged"}, + ) + for _ in range(100): + if witness.exists(): + break + time.sleep(0.02) + self.assertEqual(witness.read_text(), "user-unchanged") + self.assertEqual(handle.inspect()[0]["pid"], handle.pid) + + def test_container_restart_refuses_to_signal_old_pid(self): + handle = self.start() + self.service.current["started_at"] = "restarted" + with self.assertRaisesRegex(BackendOperationError, "replaced or restarted"): + handle.signal("stop") + + def test_modified_identity_record_refuses_to_signal(self): + handle = self.start() + root = Path("/tmp/akernel-service-process-" + handle.process_id) + record = json.loads((root / "identity.json").read_text()) + record["start_time"] = "0" + (root / "identity.json").write_text(json.dumps(record)) + with self.assertRaises(BackendOperationError): + handle.signal("stop") + + def test_unknown_start_recovers_by_token_without_relaunch(self): + counter = Path(self.temp.name) / "counter" + self.service.lose_start_reply = True + command = f"echo launched >> {counter}; exec sleep 60" + with self.assertRaises(TimeoutException) as raised: + ServiceProcessHandle.start(self.service, command) + attached = ServiceProcessHandle.attach( + self.service, raised.exception.process_descriptor + ) + self.assertGreater(attached.pid, 1) + self.assertEqual(counter.read_text(), "launched\n") + attached.signal("kill") + self.assertEqual(attached.wait(timeout=3), 137) + + def test_wait_reports_exit_code_and_times_out_without_killing(self): + handle = self.start() + with self.assertRaises(TimeoutException): + handle.wait(timeout=0.05) + self.assertTrue(handle.inspect()) + handle.signal("kill") + self.assertEqual(handle.wait(timeout=3), 137) + + def test_thaw_children_before_bash_waiter_does_not_report_sigstop_as_exit(self): + import time + from unittest.mock import patch + + from akernel_sdk import _service_process_guest as guest + + root = Path(self.temp.name) + ready, result = root / "ready", root / "wait-result" + command = ( + "set -m; sleep 1 & worker=$!; " + f'echo "$worker" > {ready}; wait "$worker"; echo "$?" > {result}' + ) + handle = ServiceProcessHandle.start(self.service, command) + deadline = time.monotonic() + 3 + while not ready.exists() and time.monotonic() < deadline: + time.sleep(0.01) + self.assertTrue(ready.exists()) + child = int(ready.read_text()) + child_identity = guest.stat(child) + process_root = Path("/tmp/akernel-service-process-" + handle.process_id) + kill = os.kill + + def scheduled_kill(pid, sig): + kill(pid, sig) + if sig == signal.SIGCONT and pid == handle.pid: + # Reproduce a scheduler switch to the waiting Bash immediately + # after its thaw. A still-stopped child must not yield status 147. + time.sleep(0.08) + + try: + handle.signal("stop") + with patch.object(guest.os, "kill", side_effect=scheduled_kill): + guest.operate(process_root, "continue") + self.assertEqual(handle.wait(timeout=3), 0) + self.assertEqual(result.read_text().strip(), "0") + finally: + current = guest.stat(child) + if ( + current + and child_identity + and current["start_time"] == child_identity["start_time"] + ): + try: + kill(child, signal.SIGKILL) + except ProcessLookupError: + pass + try: + os.waitpid(child, 0) + except ChildProcessError: + pass diff --git a/sdk/python/tests/unit/test_snapshot_create.py b/sdk/python/tests/unit/test_snapshot_create.py index 5ef112db..fc1ad627 100644 --- a/sdk/python/tests/unit/test_snapshot_create.py +++ b/sdk/python/tests/unit/test_snapshot_create.py @@ -63,6 +63,9 @@ def test_options_and_facades_match_legacy_restore(self): cwd="/not-created", port_forwardings=[8080], mounts=[Mount(target="/tools", image_url="tools:latest")], + extra_config={ + "fileBackedTmpfsMounts": [{"target": "/upper", "sizeMB": 64}] + }, reverse_tunnel=HttpReverseTunnel(target="http://127.0.0.1:8123"), detached=False, failover=True, @@ -105,6 +108,8 @@ def test_invalid_options_fail_before_loading_backend(self): {"snapshot_id": "saved", "port_forwardings": [8080, 8080]}, {"snapshot_id": "saved", "mounts": [object()]}, {"snapshot_id": "saved", "network_policy": {}}, + {"snapshot_id": "saved", "extra_config": []}, + {"snapshot_id": "saved", "extra_config": {"size": float("nan")}}, ): with ( self.subTest(options=options), diff --git a/sdk/python/tests/unit/test_snapshots.py b/sdk/python/tests/unit/test_snapshots.py index c970af88..7a9d6dbd 100644 --- a/sdk/python/tests/unit/test_snapshots.py +++ b/sdk/python/tests/unit/test_snapshots.py @@ -232,7 +232,14 @@ def deny(_client, request, **kwargs): patch.object(httpx.Client, "send", autospec=True, side_effect=deny), self.assertRaises(BackendOperationError) as raised, ): - Sandbox.from_snapshot("saved", cwd="/app", node_id="node-a") + Sandbox.from_snapshot( + "saved", + cwd="/app", + node_id="node-a", + extra_config={ + "fileBackedTmpfsMounts": [{"target": "/upper", "sizeMB": 64}] + }, + ) self.assertEqual(len(requests), 1) body = json.loads(requests[0].content) self.assertEqual(body["snapshotId"], "saved") @@ -247,6 +254,10 @@ def deny(_client, request, **kwargs): self.assertNotIn(key, body) self.assertEqual(body["rootfs"], {}) self.assertEqual(body["cwd"], "/app") + self.assertEqual( + body["extra_config"], + {"fileBackedTmpfsMounts": [{"target": "/upper", "sizeMB": 64}]}, + ) self.assertEqual(raised.exception.code, "PERMISSION_DENIED") self.assertEqual(raised.exception.request_id, "restore-test") self.assertEqual(raised.exception.outcome, "not_started") diff --git a/sdk/python/tests/unit/test_types.py b/sdk/python/tests/unit/test_types.py index b1c1a8ec..fc783524 100644 --- a/sdk/python/tests/unit/test_types.py +++ b/sdk/python/tests/unit/test_types.py @@ -72,6 +72,8 @@ def test_public_exports_are_minimal(self): set(akernel_sdk.__all__), { "Sandbox", + "BuiltImage", + "build_image", "Client", "SandboxQuery", "SandboxState", @@ -124,6 +126,13 @@ def test_public_exports_are_minimal(self): "DockerfileBuildError", "DockerfileParseError", "BuildInstruction", + "ComposeProject", + "ComposeOperationError", + "ServiceHandle", + "ServiceProcessHandle", + "DockerRuntime", + "ImageMountPlan", + "prepare_compose_images", }, ) for removed in ( @@ -199,9 +208,7 @@ def test_host_path_mount_defaults_to_read_only(self): ) def test_host_path_mount_requires_explicit_writable_mode(self): - mount = Mount( - target="/apps/logs", host_path="/apps/logs", read_only=False - ) + mount = Mount(target="/apps/logs", host_path="/apps/logs", read_only=False) self.assertEqual(mount.to_dict()["options"], ["rw"]) self.assertFalse(mount.to_dict()["create_if_missing"]) @@ -212,18 +219,27 @@ def test_host_path_cannot_be_combined_with_other_mount_sources(self): {"s3_config": s3}, {"image_url": "tools:v1", "s3_config": s3}, ): - with self.subTest(extra=extra), self.assertRaisesRegex( - ValueError, "exactly one" + with ( + self.subTest(extra=extra), + self.assertRaisesRegex(ValueError, "exactly one"), ): Mount(target="/logs", host_path="/apps/logs", **extra) def test_host_path_rejects_relative_traversal_and_control_paths(self): for source in ( - "", "apps/logs", "~/logs", "/apps/../logs", "/apps/./logs", - "/apps/\0logs", "/apps/\nlogs", "/apps/\x85logs", 123, + "", + "apps/logs", + "~/logs", + "/apps/../logs", + "/apps/./logs", + "/apps/\0logs", + "/apps/\nlogs", + "/apps/\x85logs", + 123, ): - with self.subTest(source=source), self.assertRaisesRegex( - ValueError, "host_path" + with ( + self.subTest(source=source), + self.assertRaisesRegex(ValueError, "host_path"), ): Mount(target="/logs", host_path=source) @@ -233,22 +249,26 @@ def test_host_path_supports_only_bind_mounts(self): def test_mount_read_only_must_be_boolean(self): for read_only in (0, 1, None, "false"): - with self.subTest(read_only=read_only), self.assertRaisesRegex( - TypeError, "read_only" + with ( + self.subTest(read_only=read_only), + self.assertRaisesRegex(TypeError, "read_only"), ): Mount(target="/logs", host_path="/apps/logs", read_only=read_only) def test_image_and_s3_mounts_remain_read_only(self): s3 = S3Config("https://s3.example.com", "data", "weights.img") for source in ({"image_url": "tools:v1"}, {"s3_config": s3}): - with self.subTest(source=source), self.assertRaisesRegex( - ValueError, "read.only" + with ( + self.subTest(source=source), + self.assertRaisesRegex(ValueError, "read.only"), ): Mount(target="/tools", read_only=False, **source) self.assertEqual( Mount("/tools", "tools:v1", None, "bind").to_dict(), { - "type": "bind", "target": "/tools", "options": ["ro"], + "type": "bind", + "target": "/tools", + "options": ["ro"], "image_url": "tools:v1", }, ) From 57087ea05c3173a57bd56f6761721a5f2bfbfeb2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9C=88=E5=8F=AF?= Date: Fri, 6 Nov 2026 17:37:52 +0800 Subject: [PATCH 4/5] feat(deploy): preserve runtime rootfs identity across upgrades MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add opt-in runtime rootfs pinning with explicit Execd verification and node configuration. Preserve writable shared-memory defaults and standalone restore settings, with chart/helper regressions and operator instructions. Signed-off-by: 月可 --- deploy/README.md | 7 ++ .../charts/core/scripts/pin-runtime-rootfs.py | 98 +++++++++++++++++++ .../charts/core/templates/node/configmap.yaml | 5 + .../charts/core/templates/node/daemonset.yaml | 43 ++++++++ .../charts/core/tests/test_default_shm.py | 59 +++++++++++ .../core/tests/test_pin_runtime_rootfs.py | 91 +++++++++++++++++ .../core/tests/test_runtime_rootfs_pin.py | 87 ++++++++++++++++ deploy/akernel/charts/core/values.yaml | 24 +++++ deploy/docs/runtime-rootfs-pinning.md | 32 ++++++ deploy/standalone/config/config.json | 12 +++ 10 files changed, 458 insertions(+) create mode 100644 deploy/akernel/charts/core/scripts/pin-runtime-rootfs.py create mode 100644 deploy/akernel/charts/core/tests/test_default_shm.py create mode 100644 deploy/akernel/charts/core/tests/test_pin_runtime_rootfs.py create mode 100644 deploy/akernel/charts/core/tests/test_runtime_rootfs_pin.py create mode 100644 deploy/docs/runtime-rootfs-pinning.md diff --git a/deploy/README.md b/deploy/README.md index aa35328c..4473ace8 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -249,6 +249,13 @@ a time and keep the others available. This also allows updates when pod anti-affinity already occupies every eligible node. A single replica retains surge-first updates. +The default standalone and Helm OCI specifications mount a private 64 MiB tmpfs +at `/dev/shm` with mode `1777`, `nosuid`, `nodev` and `noexec`. Non-root processes, +including PostgreSQL, can create POSIX shared-memory objects there. This mount +belongs to the sandbox and follows its checkpoint/restore; it is not the node's +shared-memory directory. Custom OCI specifications own their mount options. +Drain existing sandboxes and snapshots before replacing their OCI mount layout. + ### Network ACLs The bundled standalone, Helm, and Terraform sandboxd configurations enable diff --git a/deploy/akernel/charts/core/scripts/pin-runtime-rootfs.py b/deploy/akernel/charts/core/scripts/pin-runtime-rootfs.py new file mode 100644 index 00000000..f70405a6 --- /dev/null +++ b/deploy/akernel/charts/core/scripts/pin-runtime-rootfs.py @@ -0,0 +1,98 @@ +"""Install one immutable runtime rootfs; verify both old and current Execd.""" +import argparse +import hashlib +import os +from pathlib import Path +import re +import stat +import tempfile + + +def digest(path): + descriptor = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC | os.O_NONBLOCK) + with os.fdopen(descriptor, 'rb') as stream: + if not stat.S_ISREG(os.fstat(stream.fileno()).st_mode): + raise ValueError('Runtime artifact must be a regular file') + result = hashlib.sha256() + for chunk in iter(lambda: stream.read(1024 * 1024), b''): + result.update(chunk) + return result.hexdigest() + + +def sync_directory(path): + descriptor = os.open(path, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + + +def pin(root, checksum, execd_checksum, *, source, execd, stage): + if not all(re.fullmatch('[a-f0-9]{64}', value) for value in (checksum, execd_checksum)): + raise ValueError('Runtime rootfs and Execd require SHA256 values') + if digest(execd) != execd_checksum: + raise ValueError('Pinned runtime Execd differs from this platform image') + root = Path(root) + if not root.is_absolute() or root.resolve(strict=True) != root or not root.is_dir(): + raise ValueError('Runtime cache must be a canonical mounted directory') + directory = root / 'runtime-rootfs' + if stage and not directory.exists(): + directory.mkdir(mode=0o700) + sync_directory(root) + if directory.is_symlink() or not directory.is_dir(): + raise ValueError('Runtime cache directory is missing or aliased') + attributes = directory.stat() + if attributes.st_uid != os.geteuid() or attributes.st_mode & 0o022: + raise ValueError('Runtime cache must be owned and not writable by other users') + if stage and digest(source) != checksum: + raise ValueError('Runtime source SHA256 does not match the pin') + if stage: + sync_directory(root) + destination = directory / (checksum + '.img') + if destination.exists() or destination.is_symlink(): + if digest(destination) != checksum: + raise ValueError('Existing immutable runtime rootfs is corrupted') + if stage: + sync_directory(directory) + return destination + if not stage: + raise ValueError('Pinned runtime rootfs has not been staged') + temporary = None + descriptor = os.open(source, os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC | os.O_NONBLOCK) + try: + with os.fdopen(descriptor, 'rb') as input_stream: + if not stat.S_ISREG(os.fstat(input_stream.fileno()).st_mode): + raise ValueError('Runtime source must be a regular file') + with tempfile.NamedTemporaryFile(dir=directory, prefix='.stage-', delete=False) as output: + temporary = Path(output.name) + result = hashlib.sha256() + for chunk in iter(lambda: input_stream.read(1024 * 1024), b''): + result.update(chunk) + output.write(chunk) + if result.hexdigest() != checksum: + raise ValueError('Runtime source SHA256 does not match the pin') + output.flush() + os.fchmod(output.fileno(), 0o444) + os.fsync(output.fileno()) + try: + os.link(temporary, destination, follow_symlinks=False) + except FileExistsError: + if digest(destination) != checksum: + raise ValueError('Concurrent runtime rootfs installation differs') from None + sync_directory(directory) + finally: + if temporary is not None: + temporary.unlink(missing_ok=True) + return destination + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('mode', choices=['stage', 'verify']) + parser.add_argument('--root', required=True) + parser.add_argument('--sha256', required=True) + parser.add_argument('--execd-sha256', required=True) + args = parser.parse_args() + pin(args.root, args.sha256, args.execd_sha256, + source='/opt/akernel/runtime/akernel-runtime-rootfs.img', + execd='/opt/adx/current/runtime/adx-execd', stage=args.mode == 'stage') diff --git a/deploy/akernel/charts/core/templates/node/configmap.yaml b/deploy/akernel/charts/core/templates/node/configmap.yaml index fe561416..97d2730b 100644 --- a/deploy/akernel/charts/core/templates/node/configmap.yaml +++ b/deploy/akernel/charts/core/templates/node/configmap.yaml @@ -101,6 +101,11 @@ data: prepare_checkpoint_storage.py: | {{ .Files.Get "scripts/prepare_checkpoint_storage.py" | indent 4 }} +{{- if (.Values.node.runtimeRootfsPin | default dict).enabled }} + + pin-runtime-rootfs.py: | +{{ .Files.Get "scripts/pin-runtime-rootfs.py" | indent 4 }} +{{- end }} {{- if .Values.node.applicationLogs.enabled }} diff --git a/deploy/akernel/charts/core/templates/node/daemonset.yaml b/deploy/akernel/charts/core/templates/node/daemonset.yaml index 80f7cefa..ee1327f1 100644 --- a/deploy/akernel/charts/core/templates/node/daemonset.yaml +++ b/deploy/akernel/charts/core/templates/node/daemonset.yaml @@ -1,3 +1,12 @@ +{{- $runtimePin := .Values.node.runtimeRootfsPin | default dict }} +{{- if $runtimePin.enabled }} +{{- if and ($runtimePin.currentExecdSha256 | default "") (not (regexMatch "^[a-f0-9]{64}$" $runtimePin.currentExecdSha256)) }} +{{- fail "runtimeRootfsPin.currentExecdSha256 must be a separately qualified SHA256" }} +{{- end }} +{{- if or .Values.adx.e2b.enabled (contains "://" ($runtimePin.sourceImage | default "")) (not (regexMatch "^[a-z0-9][a-z0-9._:/-]*@sha256:[a-f0-9]{64}$" ($runtimePin.sourceImage | default ""))) (not (regexMatch "^[a-f0-9]{64}$" ($runtimePin.sha256 | default ""))) (not (regexMatch "^[a-f0-9]{64}$" ($runtimePin.execdSha256 | default ""))) }} +{{- fail "runtimeRootfsPin requires ADX without E2B, a digest-pinned source image, and rootfs/Execd SHA256 values" }} +{{- end }} +{{- end }} {{- if .Values.kruise.enabled }} apiVersion: apps.kruise.io/v1alpha1 {{- else }} @@ -72,6 +81,34 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} initContainers: + {{- if $runtimePin.enabled }} + {{- range $mode := list "stage" "verify" }} + - name: {{ $mode }}-runtime-rootfs + {{- if eq $mode "stage" }} + image: {{ $runtimePin.sourceImage | quote }} + {{- else }} + image: "{{ include "core.image" (dict "root" $ "image" $.Values.node.image) }}" + {{- end }} + imagePullPolicy: IfNotPresent + command: [python3, /config/pin-runtime-rootfs.py] + args: + - {{ $mode }} + - --root + - /runtime-cache + - --sha256 + - {{ $runtimePin.sha256 | quote }} + - --execd-sha256 + {{- if eq $mode "verify" }} + - {{ $runtimePin.currentExecdSha256 | default $runtimePin.execdSha256 | quote }} + {{- else }} + - {{ $runtimePin.execdSha256 | quote }} + {{- end }} + securityContext: {runAsUser: 0, runAsGroup: 0, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true} + volumeMounts: + - {name: akernel-node-config, mountPath: /config, readOnly: true} + - {name: home-disk, mountPath: /runtime-cache, readOnly: {{ eq $mode "verify" }}} + {{- end }} + {{- end }} {{- if .Values.node.config.sandboxd.enableRunc }} - name: prepare-runc-filesystems image: "{{ include "core.image" (dict "root" . "image" .Values.node.image) }}" @@ -257,6 +294,12 @@ spec: securityContext: privileged: true volumeMounts: + {{- if $runtimePin.enabled }} + - name: home-disk + mountPath: /opt/akernel/runtime/akernel-runtime-rootfs.img + subPath: {{ printf "runtime-rootfs/%s.img" $runtimePin.sha256 | quote }} + readOnly: true + {{- end }} {{- if .Values.node.applicationLogs.enabled }} - name: application-logs-disk mountPath: /app/logs diff --git a/deploy/akernel/charts/core/tests/test_default_shm.py b/deploy/akernel/charts/core/tests/test_default_shm.py new file mode 100644 index 00000000..41f1609a --- /dev/null +++ b/deploy/akernel/charts/core/tests/test_default_shm.py @@ -0,0 +1,59 @@ +"""Default OCI shared memory works for non-root database processes.""" + +import json +import subprocess +import unittest +from pathlib import Path + +import yaml + +ROOT = Path(__file__).resolve().parents[5] +CHART = ROOT / "deploy/akernel/charts/core" + + +class DefaultSharedMemoryTest(unittest.TestCase): + def assert_shared_memory(self, spec): + mounts = spec["mounts"] + shm = [mount for mount in mounts if mount["destination"] == "/dev/shm"] + self.assertEqual(len(shm), 1) + self.assertEqual(shm[0]["type"], "tmpfs") + self.assertEqual(shm[0]["source"], "shm") + self.assertTrue( + {"mode=1777", "size=65536k", "nosuid", "nodev", "noexec"}.issubset( + shm[0]["options"] + ) + ) + self.assertLess( + next(i for i, mount in enumerate(mounts) if mount["destination"] == "/dev"), + mounts.index(shm[0]), + ) + + def test_standalone_default_has_private_writable_shared_memory(self): + self.assert_shared_memory( + json.loads((ROOT / "deploy/standalone/config/config.json").read_text()) + ) + + def test_rendered_helm_default_has_private_writable_shared_memory(self): + rendered = subprocess.check_output( + [ + "helm", + "template", + "akernel", + str(CHART), + "--set", + "adx.tls.existingSecret=adx-test-tls", + ], + text=True, + ) + config = next( + resource + for resource in yaml.safe_load_all(rendered) + if resource + and resource["kind"] == "ConfigMap" + and resource["metadata"]["name"] == "akernel-node-config" + ) + self.assert_shared_memory(json.loads(config["data"]["config.json"])) + + +if __name__ == "__main__": + unittest.main() diff --git a/deploy/akernel/charts/core/tests/test_pin_runtime_rootfs.py b/deploy/akernel/charts/core/tests/test_pin_runtime_rootfs.py new file mode 100644 index 00000000..9aa7a0cb --- /dev/null +++ b/deploy/akernel/charts/core/tests/test_pin_runtime_rootfs.py @@ -0,0 +1,91 @@ +"""Real filesystem checks for immutable runtime image installation.""" +import hashlib +import importlib.util +import os +from pathlib import Path +import tempfile +import unittest + +SCRIPT = Path(__file__).resolve().parents[1] / 'scripts/pin-runtime-rootfs.py' +SPEC = importlib.util.spec_from_file_location('pin_runtime_rootfs', SCRIPT) +MODULE = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(MODULE) + + +class PinRuntimeRootfsTest(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name).resolve() + self.source = self.root / 'source.img' + self.source.write_bytes(b'qualified runtime bytes') + self.executable = self.root / 'adx-execd' + self.executable.write_bytes(b'unchanged static executable') + self.checksum = hashlib.sha256(self.source.read_bytes()).hexdigest() + self.execd_checksum = hashlib.sha256(self.executable.read_bytes()).hexdigest() + + def run_pin(self, stage=True): + return MODULE.pin(self.root, self.checksum, self.execd_checksum, + source=self.source, execd=self.executable, stage=stage) + + def test_stage_reuse_and_verify_keep_same_immutable_inode(self): + cached = self.run_pin() + inode = cached.stat().st_ino + self.assertEqual(cached.read_bytes(), self.source.read_bytes()) + self.assertEqual(cached.stat().st_mode & 0o777, 0o444) + self.run_pin() + self.run_pin(stage=False) + self.assertEqual(cached.stat().st_ino, inode) + self.assertFalse(list(cached.parent.glob('.stage-*'))) + + def test_changed_current_execd_is_rejected_even_with_good_cache(self): + cached = self.run_pin() + self.executable.write_bytes(b'new guest ABI') + with self.assertRaisesRegex(ValueError, 'Execd differs'): + self.run_pin(stage=False) + self.assertEqual(hashlib.sha256(cached.read_bytes()).hexdigest(), self.checksum) + + def test_corrupt_cache_is_rejected_without_replacing_it(self): + cached = self.run_pin() + cached.chmod(0o600) + cached.write_bytes(b'corrupt') + inode = cached.stat().st_ino + with self.assertRaisesRegex(ValueError, 'corrupted'): + self.run_pin() + self.assertEqual(cached.stat().st_ino, inode) + self.assertEqual(cached.read_bytes(), b'corrupt') + + def test_source_mismatch_publishes_no_file(self): + self.source.write_bytes(b'wrong runtime') + with self.assertRaisesRegex(ValueError, 'source SHA256'): + self.run_pin() + self.assertFalse(list((self.root / 'runtime-rootfs').iterdir())) + + def test_source_and_cache_symlinks_are_rejected(self): + original = self.source.with_suffix('.original') + self.source.rename(original) + self.source.symlink_to(original) + with self.assertRaises(OSError): + self.run_pin() + self.source.unlink() + original.rename(self.source) + cached = self.run_pin() + cached.unlink() + cached.symlink_to(self.source) + with self.assertRaises(OSError): + self.run_pin(stage=False) + + def test_missing_staged_file_cannot_be_rebuilt_by_verify(self): + with self.assertRaisesRegex(ValueError, 'directory is missing'): + self.run_pin(stage=False) + self.assertFalse((self.root / 'runtime-rootfs').exists()) + + def test_fifo_is_rejected_without_blocking(self): + self.source.unlink() + os.mkfifo(self.source) + with self.assertRaisesRegex(ValueError, 'regular file'): + self.run_pin() + + +if __name__ == '__main__': + unittest.main() diff --git a/deploy/akernel/charts/core/tests/test_runtime_rootfs_pin.py b/deploy/akernel/charts/core/tests/test_runtime_rootfs_pin.py new file mode 100644 index 00000000..d52bfa9d --- /dev/null +++ b/deploy/akernel/charts/core/tests/test_runtime_rootfs_pin.py @@ -0,0 +1,87 @@ +"""The runtime rootfs must remain immutable across control-plane upgrades.""" +import subprocess +import unittest +from pathlib import Path + +import yaml + +CHART = Path(__file__).resolve().parents[1] +SOURCE = 'example.test/platform@sha256:' + 'a' * 64 + + +def render(**changes): + values = {'node.runtimeRootfsPin.enabled': 'true', + 'node.runtimeRootfsPin.sourceImage': SOURCE, + 'node.runtimeRootfsPin.sha256': 'b' * 64, + 'node.runtimeRootfsPin.execdSha256': 'c' * 64} + values.update(changes) + command = ['helm', 'template', 'test', str(CHART), '--set', + 'adx.tls.existingSecret=test-tls'] + for key, value in values.items(): + command.extend(['--set' if key.endswith('.enabled') else '--set-string', key + '=' + value]) + return subprocess.run(command, text=True, capture_output=True, check=False) + + +class RuntimeRootfsPinTest(unittest.TestCase): + def test_pin_stages_old_artifact_but_verifies_current_execd(self): + result = render() + self.assertEqual(result.returncode, 0, result.stderr) + objects = [item for item in yaml.safe_load_all(result.stdout) if item] + node = next(item for item in objects if item['kind'] == 'DaemonSet') + spec = node['spec']['template']['spec'] + initializers = {item['name']: item for item in spec['initContainers']} + self.assertIn('stage-runtime-rootfs', initializers) + self.assertEqual(initializers['stage-runtime-rootfs']['image'], SOURCE) + main = next(item for item in spec['containers'] if item['name'] == 'akernel-node') + self.assertEqual(initializers['verify-runtime-rootfs']['image'], main['image']) + mount = next(item for item in main['volumeMounts'] if + item['mountPath'] == '/opt/akernel/runtime/akernel-runtime-rootfs.img') + self.assertTrue(mount['readOnly']) + self.assertEqual(mount['name'], 'home-disk') + self.assertEqual(mount['subPath'], 'runtime-rootfs/' + 'b' * 64 + '.img') + self.assertIn('c' * 64, initializers['verify-runtime-rootfs']['args']) + config = next(item for item in objects if item['kind'] == 'ConfigMap' and + item['metadata']['name'] == 'akernel-node-config') + self.assertIn('pin-runtime-rootfs.py', config['data']) + + def test_rejects_unpinned_source_and_invalid_hashes(self): + for changes in ({'node.runtimeRootfsPin.sourceImage': 'example.test/platform:latest'}, + {'node.runtimeRootfsPin.sha256': '../image'}, + {'node.runtimeRootfsPin.execdSha256': ''}): + with self.subTest(changes=changes): + self.assertNotEqual(render(**changes).returncode, 0) + + def test_separately_qualified_current_execd_does_not_change_source_or_guest(self): + result = render(**{'node.runtimeRootfsPin.currentExecdSha256': 'd' * 64}) + self.assertEqual(result.returncode, 0, result.stderr) + objects = [item for item in yaml.safe_load_all(result.stdout) if item] + spec = next(item for item in objects if item['kind'] == 'DaemonSet')['spec']['template']['spec'] + initializers = {item['name']: item for item in spec['initContainers']} + self.assertIn('c' * 64, initializers['stage-runtime-rootfs']['args']) + self.assertNotIn('d' * 64, initializers['stage-runtime-rootfs']['args']) + self.assertIn('d' * 64, initializers['verify-runtime-rootfs']['args']) + self.assertNotIn('c' * 64, initializers['verify-runtime-rootfs']['args']) + main = next(item for item in spec['containers'] if item['name'] == 'akernel-node') + mount = next(item for item in main['volumeMounts'] if item['mountPath'] == '/opt/akernel/runtime/akernel-runtime-rootfs.img') + self.assertEqual(mount['subPath'], 'runtime-rootfs/' + 'b' * 64 + '.img') + self.assertTrue(mount['readOnly']) + for invalid in ['../image', 'd' * 63, 'D' * 64]: + with self.subTest(invalid=invalid): + self.assertNotEqual(render(**{'node.runtimeRootfsPin.currentExecdSha256': invalid}).returncode, 0) + + def test_disabled_pin_leaves_worker_runtime_mounts_unchanged(self): + result = render(**{'node.runtimeRootfsPin.enabled': 'false'}) + self.assertEqual(result.returncode, 0, result.stderr) + objects = [item for item in yaml.safe_load_all(result.stdout) if item] + spec = next(item for item in objects if item['kind'] == 'DaemonSet')['spec']['template']['spec'] + self.assertFalse(any(item['name'].endswith('-runtime-rootfs') for item in spec['initContainers'])) + main = next(item for item in spec['containers'] if item['name'] == 'akernel-node') + self.assertFalse(any(item['mountPath'] == '/opt/akernel/runtime/akernel-runtime-rootfs.img' + for item in main['volumeMounts'])) + + def test_envd_profile_requires_separate_qualification(self): + self.assertNotEqual(render(**{'adx.e2b.enabled': 'true'}).returncode, 0) + + +if __name__ == '__main__': + unittest.main() diff --git a/deploy/akernel/charts/core/values.yaml b/deploy/akernel/charts/core/values.yaml index 3b1b2b9f..156d83ab 100644 --- a/deploy/akernel/charts/core/values.yaml +++ b/deploy/akernel/charts/core/values.yaml @@ -174,6 +174,18 @@ node: # Operator-managed preparation; see deploy/docs/node-local-directories.md # and deploy/akernel/examples/node-local-directories.yaml. Lists replace on merge. initContainers: [] + # Keep guest runtime bytes fixed while updating control/worker executables. + # The source image must be qualified, digest-pinned and contain the standard + # runtime rootfs and static Execd paths. Changing this pin requires separate + # snapshot compatibility qualification; cached files are not auto-collected. + runtimeRootfsPin: + enabled: false + sourceImage: "" + sha256: "" + execdSha256: "" + # Defaults to execdSha256. A different carrier executable requires explicit + # old-guest/new-platform C/R qualification; the pinned guest stays unchanged. + currentExecdSha256: "" directoryHelper: # Package the script for the init-container example without invoking it. # Disabled by default to preserve the node rollout checksum. @@ -412,6 +424,18 @@ node: "type": "tmpfs", "source": "tmpfs" }, + { + "destination": "/dev/shm", + "type": "tmpfs", + "source": "shm", + "options": [ + "nosuid", + "nodev", + "noexec", + "mode=1777", + "size=65536k" + ] + }, { "destination": "/sys", "type": "sysfs", diff --git a/deploy/docs/runtime-rootfs-pinning.md b/deploy/docs/runtime-rootfs-pinning.md new file mode 100644 index 00000000..e4bdd665 --- /dev/null +++ b/deploy/docs/runtime-rootfs-pinning.md @@ -0,0 +1,32 @@ +# Pin the guest runtime rootfs during platform upgrades + +Native runsc snapshots refer to the read-only EROFS runtime mounted at `/__adx`. Keeping the same path or Execd version is insufficient: rebuilding an EROFS image can change its UUID, build time and superblock. Restore must receive the original filesystem bytes. Runsc's superblock verification remains enabled. + +Use `node.runtimeRootfsPin` to upgrade control-plane or worker executables while retaining a qualified guest runtime. Harbor configuration does not change. This setting applies to the Execd-only Kubernetes profile; E2B is rejected because its additional envd compatibility requirements are not qualified here. + +Before enabling the pin, qualify a digest-pinned source image containing Python 3, `/opt/akernel/runtime/akernel-runtime-rootfs.img` and `/opt/adx/current/runtime/adx-execd`. Record the runtime file's SHA256 and the static Execd SHA256 from that image, and confirm that the source runtime contains that executable. The previously deployed, qualified all-in-one image can supply these artifacts. By default the new platform must contain the same static Execd bytes. + +When a published platform bundle changes its packaged Execd, an operator may explicitly set `node.runtimeRootfsPin.currentExecdSha256` to the separately verified carrier executable's SHA256 after qualifying that exact new platform with the old guest runtime and pre-upgrade snapshots. The source initializer still verifies `execdSha256`; only the new-platform initializer uses `currentExecdSha256`. Empty values preserve the original equal-hash check. This does not replace the guest executable, alter the cached rootfs or bypass runsc filesystem checks. Different hashes alone are not evidence of protocol compatibility: retain an explicit compatibility record and the original artifacts. + +The rootfs pin preserves filesystem identity only. It does not provide cross-version runsc checkpoint compatibility: runsc requires its checkpoint version to match the restoring binary. Retain the original complete runsc bundle and guest-rootfs tuple for old snapshots, or explicitly drain that snapshot generation before upgrading the active runtime. Do not bypass the runtime version check. + +The enabled worker pool must use the same architecture and qualified runtime tuple. This single pin does not configure heterogeneous runtime generations or automatically select an older runtime for a legacy snapshot. + +```bash +helm upgrade akernel deploy/akernel/charts/core \ + -f /path/to/operator-values.yaml \ + --set node.runtimeRootfsPin.enabled=true \ + --set-string node.runtimeRootfsPin.sourceImage="$RUNTIME_SOURCE_IMAGE" \ + --set-string node.runtimeRootfsPin.sha256="$RUNTIME_ROOTFS_SHA256" \ + --set-string node.runtimeRootfsPin.execdSha256="$RUNTIME_EXECD_SHA256" +``` + +`RUNTIME_SOURCE_IMAGE` must include an immutable `@sha256:` reference. Preserve this tuple in operator deployment values through subsequent control-plane upgrades. Prepare and review the rendered change, drain execution according to the deployment's shutdown procedure, and qualify a snapshot captured before the upgrade against the new workers before resuming submissions. + +The `stage-runtime-rootfs` initializer runs only the checked-in installer in the source image; it does not start old AKernel services. It verifies the source Execd and runtime image, then installs the image once under the existing node data disk at `runtime-rootfs/.img`. Publication is atomic and durable; an existing corrupted file is rejected rather than replaced. The `verify-runtime-rootfs` initializer uses the new worker image and checks its Execd and the cached runtime. The main worker mounts that file read-only at the existing runtime path. Snapshot restore therefore sees the same EROFS identity without copying the immutable runtime into every checkpoint. + +Cache files survive worker Pod replacement and remain separate from the image-manager directory. They are not automatically garbage-collected. Retain the source image and cached files while running environments or snapshots depend on them. A newly created node must be able to pull the source image. + +Changing guest runtime contents, the Execd executable, or this pin requires a separate compatibility and retention plan. This option does not migrate arbitrary old snapshots, reconstruct missing original images, make writable host mounts rollbackable, or qualify cross-node checkpoint transfer. Keep older required runtime artifacts available; do not bypass filesystem verification to load a snapshot with different bytes. + +The chart feature is disabled by default, preserving existing deployment behavior. Operators using snapshots across platform upgrades should explicitly maintain a qualified runtime pin rather than relying on an incidental Docker build-cache hit. diff --git a/deploy/standalone/config/config.json b/deploy/standalone/config/config.json index 25458c9e..faa72c97 100644 --- a/deploy/standalone/config/config.json +++ b/deploy/standalone/config/config.json @@ -211,6 +211,18 @@ "type": "tmpfs", "source": "tmpfs" }, + { + "destination": "/dev/shm", + "type": "tmpfs", + "source": "shm", + "options": [ + "nosuid", + "nodev", + "noexec", + "mode=1777", + "size=65536k" + ] + }, { "destination": "/sys", "type": "sysfs", From 1657ead0f483701c353ef785d08c099e6ccc51ba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E6=9C=88=E5=8F=AF?= Date: Fri, 6 Nov 2026 17:37:52 +0800 Subject: [PATCH 5/5] ci: validate managed runtime adapters and packaged SDK helpers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run runtime adapter and bridge checks, validate packaged SDK helpers, and record the managed runtime maintenance and qualification contracts. Keep native plan validation distinct from live runtime acceptance. Signed-off-by: 月可 --- .../actions/python-distributions/action.yml | 6 +++ .github/workflows/ci.yml | 28 ++++++++++ AGENTS.md | 54 +++++++++++++++++++ 3 files changed, 88 insertions(+) diff --git a/.github/actions/python-distributions/action.yml b/.github/actions/python-distributions/action.yml index c2104458..fa3878cf 100644 --- a/.github/actions/python-distributions/action.yml +++ b/.github/actions/python-distributions/action.yml @@ -60,6 +60,12 @@ runs: import adx_environment import akernel_sdk + from akernel_sdk import ComposeProject, DockerRuntime, ServiceHandle, ServiceProcessHandle + from importlib.resources import files + + assert all(callable(value) for value in (ComposeProject, DockerRuntime, ServiceHandle, ServiceProcessHandle)) + for helper in ("_image_bridge_guest.py", "_docker_network_guest.py", "_service_process_guest.py"): + assert files("akernel_sdk").joinpath(helper).read_text() assert version("akernel-sdk") == os.environ["EXPECTED_VERSION"] assert any( diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 58ff27a6..1b793ca8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,6 +15,34 @@ concurrency: cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: + image-bridge: + name: Managed Docker image bridge + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Check out repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - name: Set up runtime adapter tests + uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 + with: + python-version: "3.12" + - name: Test managed runtime adapters + run: python -m unittest discover -s builder/managed-docker -p 'test_*.py' + - name: Set up Go + uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6 + with: + go-version: '1.25.5' + cache-dependency-path: builder/image-bridge/go.sum + - name: Test and build the host/guest helper + working-directory: builder/image-bridge + env: + CGO_ENABLED: '0' + run: | + test -z "$(gofmt -l .)" + go vet ./... + go test ./... + GOOS=linux GOARCH=amd64 go build -trimpath -o "$RUNNER_TEMP/akernel-image-bridge" . + sdk-unit-tests: name: Python SDK unit tests (${{ matrix.python-version }}) runs-on: ubuntu-latest diff --git a/AGENTS.md b/AGENTS.md index baad7624..b7b468d4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -196,6 +196,13 @@ covers the archive, not a bare runsc binary. Keep CI's final-image payload checks synchronized; do not mix helper versions or fetch them at runtime. The sandboxd submodule supplies the source compiled by `make build`. +For native Kubernetes snapshots spanning control/worker upgrades, preserve the +qualified EROFS runtime bytes independently of the platform image. The opt-in +`node.runtimeRootfsPin` stages a digest-pinned source runtime on the existing +node data disk and verifies the source/current static Execd hashes before a +read-only mount. Keep the pin unchanged during control-only upgrades and retain +referenced cache files; do not relax runsc superblock checks or claim arbitrary +guest-runtime migration. See `deploy/docs/runtime-rootfs-pinning.md`. Helm and standalone enable `runsc.background_restore` and `checkpoint_sync.sync_output`; runtime files and private writable layers remain on node-local storage. AKernel's `builder/distill-fs-versions.env` fixes the distill-fs release URL @@ -963,3 +970,50 @@ ADX profile, applying patches only to present roles. Resolve profile/role selectors with ADX's single-pass environment/default/escape rules while preserving source expressions for the authoritative loader. Severity queries consume Collector-normalized log metadata and plain bodies. + + +## Managed Docker runtime + +`build_image()` accepts optional preparation CPU/memory scheduling requests while retaining its existing 1000 millicore/2048 MiB limits. Validate requests before creation and retain cleanup on all submission outcomes. Reservation overrides are explicit load-test inputs, not additional runtime capacity; ordinary callers keep the existing defaults. + +Managed profile storage metadata is derived by `builder/managed-docker/profile_metadata.py` from the same Sandboxd profile and advertised through ordinary ADXlet node labels. The ADX backend reserves the profile floor and constrains placement before creation; node-side authorization and storage checks remain authoritative. Managed YAML must reach official Compose unchanged, including native merge tags. Service handles become available after runtime service discovery. Keep authorization allowlists outside the snapshot compatibility fingerprint, while checking current authorization on every restore. + +Coalesce managed resource probes within the authenticated backend and bound +successful advisory views to one second. Do not cache failures or share views +across backend credentials; discard inherited process state before locking. +This does not limit concurrent creates or replace fresh server authorization. + +Foreground command deadlines raise TimeoutException with an explicit timeout +label and available partial command_result. Preserve its native error code and +retry/outcome guidance; application stderr is output, not the timeout reason. +Build/Compose logs retain that partial output without replaying commands or +replacing the primary error when local log persistence fails. + +Publish native builds under fresh invocation tags. Reconcile only the known +post-publication Buildx history error, using official Docker inspection and +verified immutable manifests for every planned output. Preserve the original +error and fail on partial publication; do not replay an unknown build outcome. +The exact post-publication internal deadline additionally requires every fresh +target's completed push acknowledgement to match its inspected manifest digest; +explicit attestations require native completion. Exit 100 alone proves neither +history failure nor publication. Default optional metadata provenance to Docker's +supported disabled mode while preserving operator overrides; do not claim this +repairs the BuildKit server's history or storage paths. + +The experimental remote-build path in `builder/managed-docker` delegates Dockerfile and Compose parsing/builds to the official Docker CLI, Compose, and Buildx remote driver. Connect to an operator-provided BuildKit service; its native Nydus exporter publishes to the configured internal Distribution Registry. Single-image builds use the broker's metadata-only `/describe` result and leave final rootfs acquisition to the final sandbox; never mount a task image on a disposable build helper just to read its process defaults. Compose image preparation uses sandboxd's existing image manager and Dragonfly/ROFS cache, with a private, operator-authorized image collection. Keep the broker's explicit owner and collection-flock protocol synchronized with Sandboxd's node-local live/checkpoint leases; missing/copied lease markers fail closed, and unknown legacy collections require an operator drain. Neither local mount release nor an expired capability authorizes Registry artifact deletion. Cross-node reconstruction, per-tenant Registry authorization and artifact GC remain separate release requirements. Do not add Dockerfile instructions to the legacy SDK interpreter for this path. `ComposeProject.bind_runtime()` accepts a fresh runtime-prepared opaque descriptor without remote startup; checkpoint recovery must continue using pure `attach()` and captured tunnel descriptors. + +Client-managed DockerRuntime owns image discovery/OCI metadata validation, ROFS mount plans and guest daemon startup. With an opt-in platform-managed profile, the node supplies the pinned tool/bootstrap image, authorized collection, credentials and startup parameters; the SDK transfers task files and invokes runtime operations. Sources live in `builder/image-bridge`, `builder/dind`, and `sdk/python/akernel_sdk/_docker_runtime.py`. Callers supply original project files and keep service orchestration in their adapter. Snapshot restore must rebind mounts, runtime extra_config and local tunnels without rerunning startup. Reuse the existing sandboxd image provider; do not add a parallel layer cache or modify deployment/runtime pins for client refactoring. See the SDK and builder guides for managed-upper prerequisites and supported scope. + +Managed builds export native targets without adding a Python helper layer or changing task PATH. Whole-sandbox C/R is independent of guest interpreters and application quiescence. `ServiceHandle.start_command()` runs native Compose exec as a reconnectable sandbox command, with an outer Docker CLI PID and stable command ID; restore reattaches without replay or signals. Keep the explicit Python-based inner `start_process()` API optional. + +`BuiltImage.has_startup_command` reflects the effective OCI ENTRYPOINT/CMD +returned by the runtime. Consumers that inherit image startup must disable +inheritance for empty commands; such images remain valid interactive roots. + +Compose project lifecycle and Docker service/process operations are SDK extensions in `_compose.py` and `_service_process.py`; the dependency-free `_service_process_guest.py` runs in the selected service PID namespace. Keep project descriptors JSON-compatible and restore attachment free of fresh-boot operations. Preserve explicit build/network-policy/up boundaries, unknown-outcome identities, source fencing, and container/process start-time checks before signals. Callers retain task/harness semantics. Run the real-process helper unit tests with owned-process cleanup and the complete SDK quality gate when changing these contracts. This feature does not add Docker semantics to Agent DX or change runtime pins. + +Automatic Compose image preparation covers image-only services as well as Dockerfile bases. Verify pinned Docker references and committed ROFS snapshots instead of running native pulls, including during up. Keep digest-reference regression coverage and fail closed on missing registrations; never relax bridge extraction protection or populate the private upper with replacement lower layers. + +Managed discovery requires helper schema 2 and probes capabilities before sending project input. Confine client Dockerfile/build-context and `.env` reads to the frozen project; never inherit unspecified host environment variables. External include/extends files remain unsupported in client-managed discovery. Validate manifest/config/chain relationships and allow legitimate same-chain roots with different target digests. Keep the empty automatic-plan marker for scratch-only builds. Unknown command exits and failed cancellation retain their identities and cleanup errors without replay. + +Managed Compose defaults preserve native IPv4 bridge definitions. Keep forwarding/NAT and per-service tunnel exposure in the SDK/runtime, with daemon-only legacy iptables selection for Docker DNS. Restore attachment stays pure; explicit reconnect_network refreshes only owned chains for the new outer TAP before the harness resumes. Do not claim external TCP address migration. Keep the opt-in Compose network integration regression, including existing inner and published-port sockets, synchronized with these APIs. Docker 28 operator flags are prerequisites; client work must not change production or runtime bundle pins.