Make CLI and Ansible release tests configurable

[osmman]

Why

Release checks assume a fixed CLI inventory and require an Ansible collection regardless of which artifacts a release ships. RHTAS 1.5 replaces tuftool with tufcli and no longer publishes the model-transparency stack, Ansible collection, or client-server image.

What

• Run shared CLI archive and executable checks inside each product suite, using cliStack.images YAML inventories. Include RHTAS 1.5 and Model Transparency defaults, plus a RHTAS 1.4 override.
• Add rhtas.ansible.enabled to disable Ansible for the current 1.5 defaults and allow any future release to restore it. Older-stream examples explicitly enable it. Partial overrides retain the flag and image lists.
• Preserve legacy client-server comparisons and skip that suite for RHTAS 1.5 and later.
• Fail invalid CLI configuration and missing configured artifacts. Enabled Ansible checks continue to require the collection image.

Validated CLI archives from 1.5.0 and 1.4.3, Model Transparency defaults, the Ansible 1.5 skip, legacy collection loading, future re-enablement, configuration tests, acceptance-package compilation, and the CI-pinned linter.

Breaking changes

The standalone test/acceptance/cli_stack package is removed. Release CI must select the owning product suite and supply stream-specific inventories through TEST_CONFIG. Releases shipping Ansible must set rhtas.ansible.enabled: true to override the current defaults. The README documents the migration; the releases repository is unchanged.

[qodo-for-securesign]

PR Summary by Qodo

Configure CLI stack tests by product and release stream

🐞 Bug fix 🧪 Tests ⚙️ Configuration changes 📝 Documentation 🕐 40+ Minutes

AI Description

• Move CLI archive checks into product suites with independently configurable inventories.
• Define RHTAS 1.5 defaults, 1.4 overrides, and empty inventories for older streams.
• Validate configurations and executables while skipping removed RHTAS 1.5 client-server coverage.

Diagram

graph TD
  CONFIG["TEST_CONFIG YAML"] --> LOADER["Config Loader"] --> SUITES["Product Suites"] --> TESTS["Shared CLI Tests"] --> SNAPSHOT["Release Snapshot"] --> IMAGES["CLI Images"] --> VERIFY["Archive Validation"]
  DEFAULTS["Embedded Defaults"] --> LOADER

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Central version-indexed inventory registry

• ➕ Keeps all release inventories in one source
• ➕ Could select inventories automatically from VERSION
• ➖ Couples product release schedules to shared test code
• ➖ Requires code changes whenever published inventory changes
• ➖ Makes explicit CI configuration less visible

2. Retain a standalone CLI suite

• ➕ Preserves existing CI package selection
• ➕ Keeps CLI checks isolated from other acceptance suites
• ➖ Continues treating unrelated products as one inventory
• ➖ Requires additional routing logic for product and stream selection
• ➖ Weakens ownership of republished CLI artifacts

Recommendation: Keep the PR's product-owned, YAML-configured approach. It aligns inventories with the suites responsible for publishing them, supports explicit stream overrides and empty inventories, and avoids hard-coding version policy into shared Go code.

Files changed (18) +607 / -10

Enhancement (2) +162 / -0

config.goLoad and validate product CLI inventories +101/-0

Load and validate product CLI inventories

• Introduces strict YAML decoding, product-specific inventory replacement, explicit disable support, and archive path and platform validation.

test/support/cli/config.go

helper.goProvide shared configurable CLI acceptance checks +61/-0

Provide shared configurable CLI acceptance checks

• Registers per-product image checks, handles invalid or disabled inventories, extracts configured archives, and verifies executable formats.

test/support/cli/helper.go

Bug fix (2) +13 / -2

client_server_test.goSkip removed client-server checks for RHTAS 1.5 +5/-0

Skip removed client-server checks for RHTAS 1.5

• Skips legacy client-server comparisons for RHTAS 1.5 and later while retaining earlier-release coverage.

test/acceptance/rhtas/client_server_test.go

config.goRecognize CLI stack suite configuration +8/-2

Recognize CLI stack suite configuration

• Supports unwrapped RHTAS cliStack sections and rejects non-mapping product configurations instead of silently ignoring them.

test/support/config/config.go

Refactor (1) +12 / -7

executable.goMove executable validation into shared CLI support +12/-7

Move executable validation into shared CLI support

• Repackages binary format validation for reuse by product suites and centralizes common architecture constants.

test/support/cli/executable.go

Tests (6) +197 / -0

cli_stack_images_test.goRegister shared CLI checks for Model Transparency +5/-0

Register shared CLI checks for Model Transparency

• Adds shared CLI stack test registration using the Model Transparency product identity and embedded defaults.

test/acceptance/model_transparency/cli_stack_images_test.go

mt_acceptance_suite_test.goEmbed Model Transparency suite defaults +6/-0

Embed Model Transparency suite defaults

• Defines the product key and embeds defaults.yaml for shared configurable acceptance checks.

test/acceptance/model_transparency/mt_acceptance_suite_test.go

cli_stack_images_test.goRegister shared CLI checks for Policy Controller +5/-0

Register shared CLI checks for Policy Controller

• Adds shared CLI stack test registration using the Policy Controller product configuration.

test/acceptance/policy_controller/cli_stack_images_test.go

cli_stack_images_test.goRegister configurable RHTAS CLI checks +5/-0

Register configurable RHTAS CLI checks

• Runs the shared CLI image and executable checks inside the RHTAS acceptance suite.

test/acceptance/rhtas/cli_stack_images_test.go

config_test.goTest CLI configuration and release inventories +143/-0

Test CLI configuration and release inventories

• Covers defaults, wrapped and legacy overrides, disabled inventories, invalid fields, path validation, supported platforms, and expected release contents.

test/support/cli/config_test.go

executable_test.goTest executable format validation +33/-0

Test executable format validation

• Verifies the current test binary succeeds and mismatched architectures, unsupported systems, and malformed executables fail.

test/support/cli/executable_test.go

Documentation (1) +53 / -0

README.mdDocument product-owned CLI stack configuration +53/-0

Document product-owned CLI stack configuration

• Explains inventory structure, override and disable semantics, supported archive validation, example commands, and release CI migration requirements.

README.md

Other (6) +170 / -1

defaults.yamlDisable Model Transparency CLI checks by default +2/-0

Disable Model Transparency CLI checks by default

• Defines an explicit empty default CLI inventory that release-specific configuration can replace.

test/acceptance/model_transparency/defaults.yaml

defaults.yamlDisable Policy Controller CLI checks by default +3/-0

Disable Policy Controller CLI checks by default

• Adds an explicit empty CLI inventory while preserving existing FBC defaults.

test/acceptance/policy_controller/defaults.yaml

defaults.yamlDefine the RHTAS 1.5 CLI inventory +74/-1

Define the RHTAS 1.5 CLI inventory

• Adds the default RHTAS 1.5 inventory covering 56 archives across seven CLI images, including tufcli instead of tuftool.

test/acceptance/rhtas/defaults.yaml

testconfig-1.2.2.yamlDisable CLI checks for RHTAS 1.2 +3/-0

Disable CLI checks for RHTAS 1.2

• Adds an explicit empty inventory so 1.2 releases do not inherit current CLI defaults.

testdata/testconfig-1.2.2.yaml

testconfig-1.3.2.yamlDisable CLI checks for RHTAS 1.3 +3/-0

Disable CLI checks for RHTAS 1.3

• Adds an explicit empty inventory so 1.3 releases do not inherit current CLI defaults.

testdata/testconfig-1.3.2.yaml

testconfig-1.4.yamlDefine the legacy RHTAS 1.4 CLI inventory +85/-0

Define the legacy RHTAS 1.4 CLI inventory

• Adds a stream-specific inventory retaining tuftool and Model Transparency archives alongside the other 1.4 CLI images.

testdata/testconfig-1.4.yaml

[qodo-for-securesign]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (2) 📜 Skill insights (0)

1. Older releases use wrong inventory 🔗 Cross-repo conflict ≡ Correctness

Description

resolveTestConfig discards a documented top-level cliStack whenever it finds any wrapped product
mapping, while GetCLIStackConfig retains the embedded RHTAS 1.5 defaults unless the resulting
configuration contains an explicit rhtas.cliStack override. Mixed configurations and existing
pre-1.5 release configurations therefore check for tufcli-cli-stack-image instead of their
stream-specific archives, including the 1.4.3 snapshot that publishes tuftool-cli-stack-image and
model-transparency.

Code

test/acceptance/rhtas/defaults.yaml[R130-132]

+    - imageKey: tufcli-cli-stack-image
+      binaries:
+        - {path: /binaries/tufcli_linux_amd64.tar.gz, os: linux, arch: amd64}

Relevance

●●● Strong

Resolver returns prematurely for mixed configurations, contradicting documented unwrapped cliStack
behavior and failing its own regression test.

PR-#52

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The resolver explicitly skips cliStack while collecting wrapped product mappings and returns as
soon as any mapping is found, making its later handling of the unwrapped RHTAS section unreachable
for mixed files even though the README documents that form as applying to RHTAS independently of
other product sections. The overlay logic then replaces the embedded 1.5 inventory only when it
receives rhtas.cliStack; the releases workflow supplies the adjacent complete 1.4.3 configuration
without such a section, while that release snapshot contains tuftool and model-transparency rather
than the default tufcli image.

test/acceptance/rhtas/defaults.yaml[130-138]
test/support/cli/config.go[41-54]
testdata/testconfig-1.4.yaml[57-85]
test/support/config/config.go[39-58]
README.md[84-86]
External repo: securesign/releases, .github/workflows/structural.yml [180-219]
External repo: securesign/releases, rhtas/rhtas-operator/1.4.3/stable/testingResources/structural-tests.yaml [1-84]
External repo: securesign/releases, rhtas/rhtas-operator/1.4.3/stable/snapshot.json [44-54]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Existing pre-1.5 release configurations that omit `rhtas.cliStack` inherit the incompatible embedded 1.5 inventory, and a documented top-level `cliStack` cannot correct it when the same `TEST_CONFIG` contains any product-wrapped section because the resolver discards the unwrapped RHTAS configuration. This prevents combined and stream-specific release configurations from supplying the intended RHTAS CLI inventory.

## Fix Focus Areas
- test/acceptance/rhtas/defaults.yaml[130-138]
- test/support/cli/config.go[41-54]
- testdata/testconfig-1.4.yaml[1-85]
- rhtas/rhtas-operator/1.4.3/stable/testingResources/structural-tests.yaml[1-84]
- test/support/config/config.go[39-58]
- test/support/cli/config_test.go[20-80]

## Recommended Fix
Update configuration resolution so that a top-level `cliStack` section is merged into the `rhtas` product map before parsed product sections are returned, while preserving an explicitly wrapped `rhtas.cliStack` as authoritative when both forms are present. Add a regression test with a wrapped non-RHTAS product and an unwrapped `cliStack`. Before enabling these tests in release CI, also add stream-specific `rhtas.cliStack` inventories to each affected releases-repository `testingResources/structural-tests.yaml`, using the 1.4 override for 1.4 releases and explicit empty inventories where checks should remain disabled; alternatively, make structural tests select compatible embedded inventories from `VERSION` when no release override exists.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. CLI snapshots run all suites 🔗 Cross-repo conflict ☼ Reliability

Description

Deleting the standalone test/acceptance/cli_stack package leaves the releases workflow’s
CLI_STACK package search with no match. When a dedicated CLI-stack snapshot triggers that path,
the workflow falls back to ./test/..., applying every product suite to a CLI-only snapshot and
causing unrelated acceptance failures.

Code

test/acceptance/cli_stack/cli_stack_acceptance_suite_test.go[L12-15]

-func TestAcceptance(t *testing.T) {
-	format.MaxLength = 0
-
-	RegisterFailHandler(Fail)

Relevance

●●● Strong

Explicit release-CI migration is required; deleting the package leaves the documented CLI_STACK
selection fallback unsafe.

PR-#52

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR explicitly removes the standalone suite and documents that releases CI must be migrated. The
pinned releases workflow still searches for a package containing cli_stack; because the deleted
package no longer appears in go list, its empty selection activates the workflow’s all-tests
fallback for the dedicated CLI-only snapshots.

README.md[109-114]
External repo: securesign/releases, .github/workflows/structural.yml [169-179]
External repo: securesign/releases, .github/workflows/structural.yml [268-299]
External repo: securesign/releases, rhtas/cli-stack/1.4.1/tech-preview/snapshot.json [1-15]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The releases workflow still selects the deleted standalone CLI-stack package and falls back to all tests when no package matches.

## Fix Focus Areas
- test/acceptance/cli_stack/cli_stack_acceptance_suite_test.go[12-15]
- README.md[109-114]
- .github/workflows/structural.yml[268-289]

## Recommended Fix
Coordinate this merge with a releases-repository workflow update that maps CLI-stack releases to the owning product suite and runs only its CLI-stack checks. Ensure the selected snapshot and `TEST_CONFIG` contain the complete product inventory required by that suite, and remove the obsolete standalone-package selection.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Cross-repo context — repo relationships

  Explored: repo: securesign/releases (sha: e5fe10fb)

  Explored: repo: securesign/model-transparency (sha: d2e7c6d6)

  Explored: repo: securesign/model-transparency-go (sha: 4a057328)

  Explored: repo: securesign/gitsign (sha: 9b580f08)

  Explored: repo: securesign/cosign (sha: d493674f)

  Explored: repo: securesign/trillian (sha: f0514f44)

  Explored: repo: securesign/rekor (sha: df4a220a)

  Explored: repo: securesign/timestamp-authority (sha: 3736f2d6)

Review mode: 🧠 Deep: This is a broad, behavior-changing configuration and test refactor spanning multiple suites, parsing/validation paths, release overrides, archive checks, and version-dependent behavior, creating many independent opportunities for subtle defects.

Tip of the day

💡 Did you know, you can route each action level your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗