Repository navigation
benchmarking/automation: add nighthawk-egress capacity suites - #3630
Closed
Yuan Gao (ygao-g) wants to merge 13 commits into
Closed
Yuan Gao (ygao-g) wants to merge 13 commits into
Yuan Gao (ygao-g) wants to merge 13 commits into
Conversation
Yuan Gao (ygao-g)
marked this pull request as draft
November 4, 2026 18:58
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-egress-3-automation
branch
from
November 4, 2026 20:50
9fe23ea to
e340d50
Compare
This was referenced Nov 4, 2026
Open
Collaborator
Author
Reviewer notes (Stage 1 stack guide)Stack position: Stacks on #3629 ( Commit-by-commit checks (
Local check: hack/verify/benchmarking-python.sh |
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-egress-3-automation
branch
2 times, most recently
from
November 5, 2026 22:14
7227f00 to
ac6b856
Compare
…f nighthawk-ingress Move nighthawk-ingress/output.py to nighthawk_common unchanged so nighthawk-egress can reuse it. The ingress Dockerfile copies the package into the image and the ingress tests put benchmarking/ on sys.path. Signed-off-by: Yuan Gao <ypgao@google.com>
… out of nighthawk-ingress Move nighthawk-ingress/spec.py to nighthawk_common unchanged and point its importers at the new path. actors.py imported it without using it. Signed-off-by: Yuan Gao <ypgao@google.com>
…ut of nighthawk_common build_session_spec in nighthawk_common now takes a finished traffic template. The router SNI, the certificate paths and build_spec_dict, which assembles that template, move back to nighthawk-ingress/spec.py. Signed-off-by: Yuan Gao <ypgao@google.com>
benchmarking/workloads/deploy.sh built kubectl-ate inline, and the nighthawk-egress deploy script was about to carry a copy. Move the function into benchmarking/lib/kubectl-ate.sh for both to source. Signed-off-by: Yuan Gao <ypgao@google.com>
…ration With a 2 vCPU actor sandbox and no requests on the worker pod or nighthawk-server, nothing kept the load generator and the origin server from saturating before atenet-egress, so a run could measure them instead of the gateway. Give the sandbox 16 vCPUs, request 12 vCPUs for the worker pod and 6 for nighthawk-server, run the server with --concurrency 8, and move the example nighthawk_client run from one worker to sixteen. Signed-off-by: Yuan Gao <ypgao@google.com>
deploy-traffic-driver.sh required kubectl-ate on PATH, so a fresh environment needed a manual go install step first. Source build_kubectl_ate from benchmarking/lib/kubectl-ate.sh instead, and say so in the README prerequisites in place of the install step. Signed-off-by: Yuan Gao <ypgao@google.com>
… per stage Add the queue_to_connect and sequencer.blocking statistics and the latency pstdev to each stage row, make actors optional in capacity_summary, accept Path for descriptor paths, and export the helpers nighthawk-egress uses. Compute achieved-rps and send-rate from the exact stage duration, because Nighthawk truncates execution_duration to whole seconds. Signed-off-by: Yuan Gao <ypgao@google.com>
…nter parser Define the egress-specific failure predicates and Nighthawk template for the adaptive load search, targeting the atenet-egress gateways. Add the prometheus output parser for the egress metrics. Signed-off-by: Yuan Gao <ypgao@google.com>
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-egress-3-automation
branch
from
November 5, 2026 22:54
ac6b856 to
44d6b61
Compare
…and run-dev.sh Add runner.py, spec.py and output.py glue, cluster_setup.py, the runner image and run-dev.sh, which drives one adaptive capacity run end to end and scrapes atenet-egress counters before and after it through the atenet-egress-metrics Service. Name nighthawk-server exactly in egress-policy.yaml, with and without the cluster.local suffix, because the actor resolv.conf sets ndots:5 and atunnel sees the suffixed name. Signed-off-by: Yuan Gao <ypgao@google.com>
…ters, and results Signed-off-by: Yuan Gao <ypgao@google.com>
Add testtypes/nighthawk_egress.py for nighthawk-egress benchmarks, implementing the TestType protocol hooks to configure atenet-egress CPU limits, deploy the in-cluster test rig, and render runner Job manifests. Signed-off-by: Yuan Gao <ypgao@google.com>
…dule dependencies Signed-off-by: Yuan Gao <ypgao@google.com>
…rator unit tests Signed-off-by: Yuan Gao <ypgao@google.com>
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-egress-3-automation
branch
from
November 6, 2026 06:59
44d6b61 to
9160d73
Compare
This was referenced Nov 6, 2026
Collaborator
Author
|
Superseded by #3708, which carries the test type, suites and orchestrator changes from this branch together with the runner. |
shankgan
pushed a commit
to shankgan/substrate
that referenced
this pull request
Nov 13, 2026
…to nighthawk_common (agent-substrate#3628) Part of agent-substrate#3624 (Stage 1, item 2): share the adaptive load session spec and output helpers between `nighthawk-ingress` and the coming `nighthawk-egress` runner. ### What changes Pure relocation, no behavior change. `nighthawk-ingress/output.py` and `spec.py` move to `benchmarking/nighthawk_common/`. The router-specific parts of the spec builder (router SNI, certificate paths, `build_spec_dict`) stay in `nighthawk-ingress/spec.py`, which now calls the generic `build_session_spec`. Callers and tests import `nighthawk_common` directly; the ingress Dockerfile copies the package into the image. ### Review guide 1. `output.py` rename (GitHub shows 100%), two import lines, the Dockerfile `COPY`, two `sys.path` lines in the ingress tests. 2. `spec.py` rename at 100%, importers repointed, an unused import dropped from `actors.py`. 3. The split: read `nighthawk_common/spec.py` as deletions and `nighthawk-ingress/spec.py` as the same lines moving back. ### Testing `hack/verify/benchmarking-python.sh`: the ingress suite passes at each commit, and fails without the new `sys.path` lines, so the import path is exercised. Independent of agent-substrate#3627. agent-substrate#3629 and agent-substrate#3630 stack on both. AI use: AI-assisted for the diff, the commit split and the test runs; I reviewed the diff. - [x] Tests pass - [ ] Docs updated (none needed) --------- Signed-off-by: Yuan Gao <ypgao@google.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3624 (Stage 1, item 4): run the egress capacity benchmark from
benchmarking/automation.Stacks on #3628, #3627 and #3629, in that order; review the last three commits.
What changes
testtypes/nighthawk_egress.py: the test type. Validates the numeric and duration knobs up front, shapesatenet-egressCPU, deploysnighthawk-serverandnighthawk/testerinpre_test, tears them down after.orchestrator.py:deploy_substratetakes extra ingress SPIFFE ids, sotraffic-driver-invokeris allowed only for egress suites; teardown asks each test type for its workloads instead of always running the shareddeploy.sh --delete.tests.yaml: four suites,egress_gatewaycap_{http,mitm,passthrough}_1cpuandegress_gatewaycap_http_2cpu.Review guide
testtypesregistry.tests.yaml, Dockerfile, and the orchestrator unit tests.Testing
hack/verify/benchmarking-python.shgreen (automation 69, egress 24, ingress 13).AI use: AI-assisted (Claude Code) for the diff, the commit split and the test runs; I reviewed the diff.