Repository navigation
docs/install: openchami: fix the pre-boot cloud-init probe on OpenCHAMI 0.1.7 - #15
Closed
middelkoopt wants to merge 8 commits into
Closed
middelkoopt wants to merge 8 commits into
middelkoopt wants to merge 8 commits into
Conversation
The compute nodes intermittently come up with no cloud-init data at all: cloud-init: Getting data from DataSourceNoCloudNet failed cloud-init: finished ... Datasource DataSourceNone cc_final_message.py[WARNING]: Used fallback datasource Without a datasource the node gets neither the root SSH key nor the munge key, keeps its default hostname, and never becomes reachable, so the boot wait that follows can never succeed. Two timers race here. cloud-init's DataSourceNoCloud fetches the seed with util.read_seeded(seedfrom, timeout=None), which defaults to retries=10 and sec_between=1, so the node gives up after roughly 20 seconds; this is not tunable from the kernel command line. The OpenCHAMI cloud-init server, on the other side, answers /meta-data by looking the requesting IP up in an in-memory map (SMDClient.IDfromIP) that a 60 second ticker refreshes, with no refresh on a miss. The recipe powers the nodes on 90 milliseconds after ochami discover static, so whether the node's 20 second window contains a cache tick is luck. Add a bounded wait, after discovery and before the power-on, for the server to resolve every compute node. The probe is the request the node itself will send; X-Forwarded-For carries the node's address because haproxy appends its own entry and the server reads the first one. The check has to run after node discovery, so move the pre-boot verification chapter below it. The three checks it already carried do not care either way. Signed-off-by: Adrian Reber <adrian@lisas.de>
The wait for sshd was an unbounded until loop, so a boot that failed hung the recipe instead of failing it. In CI that means the job runs to the six hour default and has to be cancelled by hand; one such job logged "Waiting for compute nodes..." 204 times before someone noticed. Bound it with a new node_boot_timeout setting, 600 seconds by default, and say where to look when it gives up. A node that has no sshd by then has not provisioned, and nothing below this point can report anything useful about it. The bound starts where the provision_wait sleep ends; the two are separate on purpose. provision_wait is the blind sleep for a recipe that has nothing to poll, which is why the CI harness sets it to 0 for a provisioner that does poll, and a timeout taken from it would then expire on the first iteration. This only converts the hang into a failure. The cause of the failure it was hiding is addressed separately. Signed-off-by: Adrian Reber <adrian@lisas.de>
When the compute nodes fail to provision, the job log carries their serial consoles and nothing else. The consoles show what the node saw: with the OpenCHAMI provisioner, a node that gets no cloud-init data logs Getting data from DataSourceNoCloudNet failed and that is the whole story. Whether the head node answered 422, 404 or nothing at all, and whether its services were even running, is not recorded anywhere, so a failure on the head node cannot be told apart from one on the node. Dump the state of the head node on both failure paths, the recipe exiting non-zero and a node never answering SSH: the openchami.target dependency tree, any failed units, the containers, and the tails of the cloud-init, smd, bss and haproxy logs. Also ask the cloud-init server the same question each node asks, with X-Forwarded-For set to the node's address the way haproxy passes it on, so the status code names the node the server cannot resolve. This replaces the bare journalctl dump on the recipe path, which is kept as the first section. Signed-off-by: Adrian Reber <adrian@lisas.de>
The integration test job had no timeout-minutes, so it inherited the six hour default. It provisions VMs and waits on them, which is exactly the shape of job that hangs rather than fails. One run sat for six hours on 2026-11-02 before being cancelled by hand. The job normally takes 10 to 16 minutes, so cap it at 60. The tmate step is the other half of the same problem. It fires on any failure and then holds the runner until someone connects and exits, so a two minute package mirror hiccup, which is what that run actually hit, occupies a runner for the rest of the day. Keep it available, but only when it was asked for: add the ci-debug label to the pull request and re-run the job. Bound the session too, so an unattended one still ends. Signed-off-by: Adrian Reber <adrian@lisas.de>
cloud-init status --wait blocks for as long as cloud-init takes and has no limit of its own, so the step that waits for it could still hang the recipe after the wait for sshd above was bounded. Bound it on the compute node with a new cloud_init_timeout setting, 600 seconds by default. Only the timeout is fatal. timeout(1) reports 124 and nothing else does, while cloud-init status also reports a run that finished with recoverable errors, which any cc_* module warning produces. A cluster whose nodes are up and usable should not stop here for that, so the remote command tests for 124 rather than passing cloud-init's own status back through pdsh -S. $? is escaped because it has to be expanded by the compute node, while cloud_init_timeout has to be expanded by the head node. Signed-off-by: Adrian Reber <adrian@lisas.de>
OpenCHAMI 0.1.7's haproxy sets X-Forwarded-For to the client's own address (set-header, where 0.1.6 appended with option forwardfor), so the pre-boot probe's X-Forwarded-For never reached the cloud-init server. The server looked up the head node and answered 422 on every try, and the recipe stopped after 90 s. Send the probe to the cloud-init-server container's own address instead, where the header is read as sent. The CI diagnostics asked the same way and always showed 422; they ask the server directly now too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Timothy Middelkoop <tmiddelkoop@internet2.edu>
The Debug via SSH (tmate) step was gated on a ci-debug pull request label, but a label added to turn it on for a re-run never takes effect: a re-run keeps the first run's event payload, so github.event never shows it. Drive it from a workflow_dispatch input instead: re-run the workflow by hand with "Open an interactive SSH session" ticked. That is action-tmate's documented pattern and needs no label or gh api call. Signed-off-by: Timothy Middelkoop <tmiddelkoop@internet2.edu>
Give the cloud-init server probes a 5 second limit (curl -m 5), in the pre-boot check and in the CI diagnostics, so a server that accepts the connection and never answers cannot hang the recipe again. Fail the cloud-init wait when cloud-init itself failed (exit 1) as well as on the timeout (124), and let only a finished run (0) or one with recoverable errors (2) through. Start the compute group check with "Next": the new node check is now the final one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Timothy Middelkoop <tmiddelkoop@internet2.edu>
adrianreber
force-pushed
the
2026-11-03-ci-openchami-boot-reliability
branch
from
November 6, 2026 07:04
3b4e580 to
49ebfab
Compare
Owner
|
merged in the original PR in ohpc |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes openhpc#2851's OpenCHAMI integration failure after the rebase onto openhpc#2852. openhpc#2852 moved the recipe to OpenCHAMI 0.1.7, whose haproxy sets
X-Forwarded-Forto the client's own address (set-header) where 0.1.6 appended to it (option forwardfor). The probe's header never reaches the cloud-init server, the server looks up the head, and every try gets a 422.curl -m 5on both, and the cloud-init wait now also fails when cloud-init itself failed (1); recoverable errors (2) still pass.workflow_dispatchinput instead of theci-debuglabel, since a label added for a re-run never reaches the re-run's event.Tested on qemu, Rocky 10 aarch64 on an M3 Mac: 3b4e580 fails in pre-boot-verify as in CI, and this branch passes hands-off to L3, with root on each compute still getting its cloud-init data through haproxy and
nobodyrefused. The fork's CI on this branch (plus two test-only trigger commits) passes all 19 jobs, including AlmaLinux 10 openchami Slurm Integration. Not tested: bare metal.@adrianreber take as much or as little of this into openhpc#2851 as you want.
🤖 Generated with Claude Code