Repository navigation
benchmarking/nighthawk-egress: add egress spec and counter parser - #3707
Draft
Yuan Gao (ygao-g) wants to merge 7 commits into
Draft
Yuan Gao (ygao-g) wants to merge 7 commits into
Yuan Gao (ygao-g) wants to merge 7 commits into
Conversation
…f nighthawk-ingress Move nighthawk-ingress/output.py to nighthawk_common unchanged so nighthawk-egress can reuse it. The ingress Dockerfile copies the package into the image and the ingress tests put benchmarking/ on sys.path. Signed-off-by: Yuan Gao <ypgao@google.com>
… out of nighthawk-ingress Move nighthawk-ingress/spec.py to nighthawk_common unchanged and point its importers at the new path. actors.py imported it without using it. Signed-off-by: Yuan Gao <ypgao@google.com>
…ut of nighthawk_common build_session_spec in nighthawk_common now takes a finished traffic template. The router SNI, the certificate paths and build_spec_dict, which assembles that template, move back to nighthawk-ingress/spec.py. Signed-off-by: Yuan Gao <ypgao@google.com>
This was referenced Nov 6, 2026
Closed
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-egress-b1-lib
branch
2 times, most recently
from
November 6, 2026 17:51
188b5d1 to
8647a6d
Compare
This was referenced Nov 6, 2026
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-egress-b1-lib
branch
4 times, most recently
from
November 6, 2026 21:53
28cfbc4 to
ba3e550
Compare
Nighthawk divides achieved-rps and send-rate by execution_duration truncated to whole seconds, so a stage that overran its nominal length by a fraction of a second read high, by up to 1/N for an N-second stage. stage_rows() now computes both from the exact duration and keeps Nighthawk's values under the *_cpp keys. Stage verdicts come from Nighthawk's threshold scores and do not change. The nighthawk-ingress README notes how far runs recorded before this change read high. Each stage row also records the queue_to_connect and sequencer.blocking statistics and the latency pstdev, so a client-side stall can be told apart from gateway latency. The tests for the new behavior live in benchmarking/nighthawk_common/tests; the ingress suite is unchanged. Signed-off-by: Yuan Gao <ypgao@google.com>
Rename _builtin_key, _global_result and _counters to builtin_key, global_result and counters, because nighthawk-egress/output.py imports them in the next commit. capacity_summary() takes actors as an optional argument and emits it only when given, since the egress benchmark has no actor fleet. build_session_spec() treats a tail_latency_slo_ms of 0 as disabled, as the ingress README documents. Signed-off-by: Yuan Gao <ypgao@google.com>
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-egress-b1-lib
branch
from
November 9, 2026 23:26
ba3e550 to
602805c
Compare
Collaborator
Author
|
Stack: #3628, then #3711, then this PR. Review the last three commits. The diff of this PR alone is at https://github.com/ygao-g/substrate/pull/86/files
The counter snapshots under |
…nter parser Define the egress-specific failure predicates and Nighthawk template for the adaptive load search, targeting the atenet-egress gateways. Add the prometheus output parser for the egress metrics. Signed-off-by: Yuan Gao <ypgao@google.com>
benchmarking/nighthawk-egress/tests covers spec.py and output.py. Add the suite to benchmarking-python.sh so it runs with the other benchmarking suites. Signed-off-by: Yuan Gao <ypgao@google.com>
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-egress-b1-lib
branch
from
November 11, 2026 22:42
602805c to
2f1e219
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The egress gateway has a manual Nighthawk setup, but no benchmark that searches for its capacity. Such a benchmark needs a load definition for each egress mode:
http,mitmandpassthrough. It also needs proof that the load went through the gateway path of that mode. A rate that is measured on another path is not a capacity of the mode.A new library, with tests, provides both parts. The first part builds the adaptive Nighthawk session for each mode, with the TLS settings of that mode. The second part reads the gateway counters from before and after a run, and adds the differences and four checks to the capacity summary. The checks are
ext_proc_invariant_ok,mode_exclusivity_ok,forward_proxy_overflow_okandgateway_counter_reset. Nothing calls the library until the runner is added. In the shared Nighthawk library, atail_latency_slo_msof 0 now disables the latency threshold, as the nighthawk-ingress README states.Part of #3624
AI use: AI-assisted for the diff, the split and the test runs; I reviewed the diff.