benchmarking: add the nighthawk-egress test type

benchmarking/automation/Dockerfile

@@ -64,7 +64,10 @@ ENV PATH=/opt/automation/venv/bin:$PATH
 
 COPY orchestrator.py util.py controlplane.py /opt/automation/
 COPY testtypes/ /opt/automation/testtypes/
-COPY manifests/runner-job.yaml.tmpl manifests/nighthawk-ingress-runner-job.yaml.tmpl /opt/automation/manifests/
+COPY manifests/runner-job.yaml.tmpl \
+     manifests/nighthawk-ingress-runner-job.yaml.tmpl \
+     manifests/nighthawk-egress-runner-job.yaml.tmpl \
+     /opt/automation/manifests/
 
 ENV PYTHONUNBUFFERED=1
 

benchmarking/automation/README.md

@@ -4,9 +4,12 @@ Scheduled, repeatable benchmark runs of a substrate branch. A CronJob on an
 **orchestration cluster** drives the full build/deploy/run/teardown cycle
 against a separate **test cluster**, once per entry in [tests.yaml](tests.yaml).
 Each run uploads results to GCS via `benchmarking/locust/runner.py`
-(`type: locust`) or `benchmarking/nighthawk-ingress/runner.py` (`type: nighthawk-ingress`,
+(`type: locust`), `benchmarking/nighthawk-ingress/runner.py` (`type: nighthawk-ingress`,
 the router capacity benchmark — see
-[benchmarking/nighthawk-ingress/README.md](../nighthawk-ingress/README.md)).
+[benchmarking/nighthawk-ingress/README.md](../nighthawk-ingress/README.md)), or
+`benchmarking/nighthawk-egress/runner.py` (`type: nighthawk-egress`, the egress
+gateway capacity benchmark, see
+[benchmarking/nighthawk-egress/README.md](../nighthawk-egress/README.md)).
 To run the cold resume tests from your workstation instead, see
 [ateapi-cold-resume.md](ateapi-cold-resume.md).
 
@@ -18,8 +21,9 @@ To run the cold resume tests from your workstation instead, see
    `gcloud container clusters get-credentials` for the test cluster.
 3. Shallow-clones `--repo` at `--branch`, captures the commit hash.
 4. `docker build && docker push` builds the runner image for each test type
-   in use, tagged with the commit hash: `${KO_DOCKER_REPO}/locust-test:<commit>`
-   and/or `${KO_DOCKER_REPO}/nighthawk-ingress-test:<commit>`.
+   in use, tagged with the commit hash: `${KO_DOCKER_REPO}/locust-test:<commit>`,
+   `${KO_DOCKER_REPO}/nighthawk-ingress-test:<commit>` and/or
+   `${KO_DOCKER_REPO}/nighthawk-egress-test:<commit>`.
 5. `go run ./cmd/ate-setup deploy ate-system` (with `ATE_CREDENTIAL_PROVIDER`
    in `.ate-dev-env.sh`, or `--credential-provider` in the test's
    `ateArgs`; passes `--sandbox-classes=<class>` so `deploy ate-system`
@@ -40,10 +44,17 @@ To run the cold resume tests from your workstation instead, see
    requests=limits and Envoy's thread count to its own `envoyCpu`, then
    waits for the rollout before deploying workloads. The teardown after
    each test redeploys substrate, so the pin never outlives its run.
+   For a `nighthawk-egress` test `benchmarking/workloads/deploy.sh --deploy`
+   does not run (`USES_SHARED_WORKLOADS = False`). The orchestrator pins
+   `deployment/atenet-egress` to 1 replica with the cpu requests=limits of
+   `envoy` and `ext-proc` and Envoy's thread count at `envoyCpu`, runs
+   `hack/setup-e2e-egress-tls-kind.sh`, and deploys `nighthawk-server` and
+   the `nighthawk/tester` actor. The teardown after each test deletes them.
 6. For each test in `tests.yaml`, or only those named by `--test-name`:
    - Submits a Job using the just-built image for the test's type
      (`runner-job.yaml.tmpl` for locust,
-     `nighthawk-ingress-runner-job.yaml.tmpl` for nighthawk-ingress).
+     `nighthawk-ingress-runner-job.yaml.tmpl` for nighthawk-ingress,
+     `nighthawk-egress-runner-job.yaml.tmpl` for nighthawk-egress).
      A locust Job also gets `--metadata` with the target cluster's name,
      location and project, merged with the `BENCHMARK_METADATA` environment
      variable (a JSON object) if set. A `--metadata` in the test's own
@@ -56,7 +67,8 @@ To run the cold resume tests from your workstation instead, see
 
 Each entry in `tests.yaml` may set `sandboxClass: gvisor | microvm` (default
 `gvisor`). This controls both `spec.sandboxClasses[0].name` on the benchmark WorkerPool
-and its `workerImage` (`ateworker-gvisor` vs `ateworker-microvm`).
+and its `workerImage` (`ateworker-gvisor` vs `ateworker-microvm`). `nighthawk-egress`
+deploys its own gVisor rig and accepts only `gvisor`.
 
 For `microvm` tests the target cluster must have KVM-capable nodes. The
 orchestrator uploads the microVM assets, so its Workload Identity principal

benchmarking/nighthawk-egress/traffic-driver-invoker.yaml → benchmarking/automation/manifests/nighthawk-egress-runner-job.yaml.tmpl

@@ -12,8 +12,26 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+# Template rendered per test by orchestrator.py for `type: nighthawk-egress`
+# entries. Shares the `benchmarking` namespace + `benchmark-runner`
+# ServiceAccount with runner-job.yaml.tmpl so the same GKE Workload Identity
+# binding grants GCS write access.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: benchmarking
+  labels:
+    ate.dev/api-access: "true"
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: benchmark-runner
+  namespace: benchmarking
+---
+# Config of the `invoker` sidecar in the Job below, which runs as benchmark-runner:
 # In-cluster Envoy proxy that listens for plaintext HTTP/1.1 and HTTP/2
-# (h2c/gRPC) on :19980, injects `ate-target-actor: nighthawk/tester` on every
+# (h2c/gRPC) on 127.0.0.1:19980, injects `ate-target-actor: nighthawk/tester` on every
 # request, and forwards upstream over mTLS to
 # atenet-router.ate-system.svc.cluster.local:443 using a SPIFFE client
 # certificate issued by podidentity.podcert.ate.dev/identity.
@@ -25,7 +43,7 @@ apiVersion: v1
 kind: ConfigMap
 metadata:
   name: traffic-driver-invoker
-  namespace: nighthawk
+  namespace: benchmarking
 data:
   envoy.yaml: |
     # SDS requires a node id and cluster.
@@ -38,8 +56,7 @@ data:
       - name: listener_http_19980
         address:
           socket_address:
-            address: "::"
-            ipv4_compat: true
+            address: 127.0.0.1
             port_value: 19980
         listener_filters:
         - name: envoy.filters.listener.http_inspector
@@ -129,35 +146,44 @@ data:
     - "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret
       name: servicedns_validation_context
       validation_context:
-        trusted_ca: { filename: /run/servicedns.podcert.ate.dev/trust-bundle.pem }
-        watched_directory: { path: /run/servicedns.podcert.ate.dev }
+        trusted_ca: { filename: /run/servicedns-ca/ca.crt }
+        watched_directory: { path: /run/servicedns-ca }
 ---
-apiVersion: apps/v1
-kind: Deployment
+apiVersion: batch/v1
+kind: Job
 metadata:
-  name: traffic-driver-invoker
-  namespace: nighthawk
-  labels:
-    app: traffic-driver-invoker
+  name: ${JOB_NAME}
+  namespace: benchmarking
 spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: traffic-driver-invoker
+  backoffLimit: 0
+  ttlSecondsAfterFinished: 86400
   template:
     metadata:
       labels:
-        app: traffic-driver-invoker
+        app: substrate-benchmark-runner
+        benchmark: nighthawk-egress
+        test-name: "${NAME}"
+        tag: "${TAG}"
     spec:
+      serviceAccountName: benchmark-runner
+      restartPolicy: Never
+      tolerations:
+      - key: ate.dev/benchmarking
+        operator: Equal
+        value: "true"
+        effect: NoSchedule
       securityContext:
         runAsUser: 65532
         runAsGroup: 65532
         runAsNonRoot: true
         fsGroup: 65532
-      containers:
-      - name: envoy
+        seccompProfile:
+          type: RuntimeDefault
+      initContainers:
+      - name: invoker
         # Same envoyproxy/envoy digest as cmd/dataplane/envoy/Dockerfile.
         image: envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43
+        restartPolicy: Always
         command:
         - /usr/local/bin/envoy
         args:
@@ -171,26 +197,54 @@ spec:
           capabilities:
             drop:
             - ALL
-        ports:
-        - name: http
-          containerPort: 19980
-          protocol: TCP
-        readinessProbe:
-          tcpSocket:
-            port: http
-          periodSeconds: 5
+        # No probe: the kubelet cannot reach the loopback listener. runner.py
+        # waits for the port.
         volumeMounts:
-        - name: config
+        - name: invoker-config
           mountPath: /etc/envoy
           readOnly: true
         - name: podidentity
           mountPath: /run/podidentity.podcert.ate.dev
           readOnly: true
         - name: servicedns-ca
-          mountPath: /run/servicedns.podcert.ate.dev
+          mountPath: /run/servicedns-ca
+          readOnly: true
+      containers:
+      - name: runner
+        image: ${IMAGE}
+        imagePullPolicy: IfNotPresent
+        command:
+        - "python3"
+        - "/app/runner.py"
+        securityContext:
+          allowPrivilegeEscalation: false
+          capabilities:
+            drop:
+            - ALL
+        args:
+        - "--name"
+        - "${NAME}"
+        - "--tag"
+        - "${TAG}"
+        - "--dest"
+        - "${DEST}"
+        - "--envoy-cpu"
+        - "${ENVOY_CPU}"
+        - "--mode"
+        - "${MODE}"
+        volumeMounts:
+        - name: servicedns-ca
+          mountPath: /run/servicedns-ca
           readOnly: true
+        resources:
+          requests:
+            cpu: "1"
+            memory: "1Gi"
+          limits:
+            cpu: "2"
+            memory: "2Gi"
       volumes:
-      - name: config
+      - name: invoker-config
         configMap:
           name: traffic-driver-invoker
       - name: podidentity
@@ -208,20 +262,4 @@ spec:
               labelSelector:
                 matchLabels:
                   podcert.ate.dev/canarying: live
-              path: trust-bundle.pem
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: traffic-driver-invoker
-  namespace: nighthawk
-spec:
-  type: ClusterIP
-  ipFamilyPolicy: PreferDualStack
-  selector:
-    app: traffic-driver-invoker
-  ports:
-  - name: http
-    port: 19980
-    targetPort: http
-    protocol: TCP
+              path: ca.crt

benchmarking/automation/orchestrator.py

@@ -26,13 +26,14 @@
         the runner image build [hook: build_image] are cached per
         target cluster.
      b. Sweep leftovers, deploy substrate, let the type shape the
-        cluster [hook: pre_test], deploy workloads (+ microvm deps when
-        the sandbox class needs them).
+        cluster [hook: pre_test], deploy the shared workloads when the
+        type uses them [USES_SHARED_WORKLOADS] (+ microvm deps when the
+        sandbox class needs them).
      c. Render the type's Job template [hooks: job_tmpl, job_subs],
         submit it, wait, tail logs, delete the Job.
-     d. Tear substrate + workloads down again so tests don't pollute
-        each other, then drop the env file so the next test can't
-        inherit this one's cluster.
+     d. Tear substrate + workloads down again [hook:
+        teardown_workloads] so tests don't pollute each other, then drop
+        the env file so the next test can't inherit this one's cluster.
   4. Exit non-zero if any test didn't complete.
 """
 
@@ -486,8 +487,12 @@ def deploy_workloads(
     run(cmd)
 
 
-def teardown_workloads() -> None:
-    run_no_check(["benchmarking/workloads/deploy.sh", "--delete"])
+def teardown_test_workloads(test_types: Iterable[str]) -> None:
+    types = list(test_types)
+    if any(TYPES[t].USES_SHARED_WORKLOADS for t in types):
+        run_no_check(["benchmarking/workloads/deploy.sh", "--delete"])
+    for t in types:
+        TYPES[t].teardown_workloads()
 
 
 def run_test(
@@ -644,7 +649,7 @@ def main() -> None:
             # state behind) would otherwise leak its substrate + workloads
             # into this run. All teardowns use --ignore-not-found, so
             # this is cheap on a clean cluster.
-            teardown_workloads()
+            teardown_test_workloads(TYPES)
             teardown_substrate()
 
             sandbox_class = test.get("sandboxClass", "gvisor")
@@ -654,15 +659,16 @@ def main() -> None:
             try:
                 deploy_substrate(substrate_args(test))
                 TYPES[ttype].pre_test(test)
-                deploy_workloads(
-                    test.get("workerCount", 1),
-                    sandbox_class,
-                    test.get("actorMemory", ""),
-                    test.get("workerWaitTimeout", ""),
-                    test.get("workerMemory", ""),
-                    isolate_workload_args(test),
-                    test.get("actorCPU", ""),
-                )
+                if TYPES[ttype].USES_SHARED_WORKLOADS:
+                    deploy_workloads(
+                        test.get("workerCount", 1),
+                        sandbox_class,
+                        test.get("actorMemory", ""),
+                        test.get("workerWaitTimeout", ""),
+                        test.get("workerMemory", ""),
+                        isolate_workload_args(test),
+                        test.get("actorCPU", ""),
+                    )
                 try:
                     status = run_test(
                         test,
@@ -682,7 +688,7 @@ def main() -> None:
             finally:
                 # Always tear down, even if deploy or run failed, so the
                 # next test (and the next CronJob fire) starts clean.
-                teardown_workloads()
+                teardown_test_workloads([ttype])
                 teardown_substrate()
             duration = time.time() - start_time
             results.append((test["name"], status, duration, failure_msg))