Repository navigation
benchmarking/nighthawk_common: compute stage rates from exact duration - #3711
Open
Yuan Gao (ygao-g) wants to merge 4 commits into
Open
Yuan Gao (ygao-g) wants to merge 4 commits into
Yuan Gao (ygao-g) wants to merge 4 commits into
Conversation
…f nighthawk-ingress Move nighthawk-ingress/output.py to nighthawk_common unchanged so nighthawk-egress can reuse it. The ingress Dockerfile copies the package into the image and the ingress tests put benchmarking/ on sys.path. Signed-off-by: Yuan Gao <ypgao@google.com>
… out of nighthawk-ingress Move nighthawk-ingress/spec.py to nighthawk_common unchanged and point its importers at the new path. actors.py imported it without using it. Signed-off-by: Yuan Gao <ypgao@google.com>
…ut of nighthawk_common build_session_spec in nighthawk_common now takes a finished traffic template. The router SNI, the certificate paths and build_spec_dict, which assembles that template, move back to nighthawk-ingress/spec.py. Signed-off-by: Yuan Gao <ypgao@google.com>
This was referenced Nov 6, 2026
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-ingress-rates
branch
3 times, most recently
from
November 6, 2026 19:09
3f71a82 to
24847c9
Compare
Yuan Gao (ygao-g)
requested review from
Bowei Du (bowei) and
Chuang Wang (chuangw6)
November 6, 2026 19:26
Yuan Gao (ygao-g)
marked this pull request as ready for review
November 6, 2026 21:19
Nighthawk divides achieved-rps and send-rate by execution_duration truncated to whole seconds, so a stage that overran its nominal length by a fraction of a second read high, by up to 1/N for an N-second stage. stage_rows() now computes both from the exact duration and keeps Nighthawk's values under the *_cpp keys. Stage verdicts come from Nighthawk's threshold scores and do not change. The nighthawk-ingress README notes how far runs recorded before this change read high. Each stage row also records the queue_to_connect and sequencer.blocking statistics and the latency pstdev, so a client-side stall can be told apart from gateway latency. The tests for the new behavior live in benchmarking/nighthawk_common/tests; the ingress suite is unchanged. Signed-off-by: Yuan Gao <ypgao@google.com>
Yuan Gao (ygao-g)
force-pushed
the
nighthawk-ingress-rates
branch
from
November 9, 2026 23:26
24847c9 to
af0a3db
Compare
Collaborator
Author
|
Stack: this PR is on top of #3628. Review the last commit only. The diff of this PR alone is at https://github.com/ygao-g/substrate/pull/88/files
Not done yet: an ingress run at this head that compares |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nighthawk divides
achieved-rpsandsend-rateby a stage duration that it rounds down to whole seconds. When a stage ran a fraction of a second longer than its nominal length, both values were too high. The error was up to 10% on a 10 s stage.slo_max_rpscomes fromachieved_rps, so the reported capacity was too high as well.The benchmark output now computes
achieved_rpsandsend_ratefrom the exact stage duration. Nighthawk's own values stay instats.jsonlunder the*_cppkeys. The stage verdicts are still Nighthawk's and do not change. The nighthawk-ingress README states how much too high the results from before this change can be. Each stage row also records the connection wait time, the sequencer blocking time and the standard deviation of the latency. With these values, a reader can tell a stall in the load client from latency in the system under test.Part of #3624
AI use: AI-assisted for the diff, the split and the test runs; I reviewed the diff.