e2e-ish: generate e2e tests from e2e-ish tests. by klihub · Pull Request #849 · containers/nri-plugins

klihub · 2026-10-14T19:35:55Z

Add support for reverse-generating e2e tests from e2e-ish unit tests, for those tests which are specifically defined as such, by using the new NewE2EScript(). If E2EISH_EMIT=1 is set, corresponding e2e tests will be generated for such e2e-ish unit tests on succesful completion.

This can be used (in some cases) to first iron out all implementation details using e2e-ish unit tests, and once everything is working generate a corresponding e2e test for final and subsequent future verification.

Copilot

🟡 Changes recommended

Assertion translation can change comparison semantics, and failed generated tests can leave namespace workloads that prevent reruns.

4 open issues

6 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Top-level null values can discard configuration overrides during emission, making generated tests differ from their unit-test sources.

1 open issue

Preserve overrides across blank or null values documents · New

4 resolved since last review

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Empty-namespace deletion can produce generated e2e behavior that differs from the passing unit test.

1 open issue

Align empty-namespace deletion semantics · New

1 resolved since last review

Preserve overrides across blank or null values documents

_{Review effort: Balanced}

revTr is the inverse of translatePy. It gives the e2e verify expression of a require assertion of a Script, like len(cpus["pod0c0"]) == 1 for require.Len(t, s.CPUs("pod0c0"), 1). A test reverses every assertion of the generated tests and translates it forward again. readTopology reads topology.var.json to determine the widths of names to generate. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>

Add NewE2EScript which can be used to define an e2eish unit test which gets translated into a corresponding e2e test on success, if E2EISH_EMIT is set. The resulting test is written to the named target, for instance n4c16/test50-name. The defined test's steps record their e2e commands as they run, with the line of the test calling them. The source of the test gives the order of the statements, and its assertions are translated back to python. Anything else is refused. An e2e test dir with the same files is left alone. One which differs is an error, unless E2EISH_EMIT_FORCE=1 is set. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>

emit_tests.sh writes the e2e tests of an e2eish tests of a policy made with NewE2EScript. TestE2eishExample is an example of such a test, the source of n4c16/test99-e2eish-example. It is skipped unless E2EISH_EXAMPLES=1 is set. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>

run_tests.sh -o writes the output of a run into a directory of its own, with no VM directory in the path, and the generator took the topology from that VM. Such a log named a topology which does not exist and the generation stopped. Match the machine its launches asked for against the topology.var.json of each topology instead. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>

Copilot

🟡 Changes recommended

Unquoted metadata keys can change generated pod labels or annotations, or prevent pod creation.

1 open issue

Quote YAML metadata keys in generated templates · New

1 resolved since last review

Align empty-namespace deletion semantics

_{Review effort: Balanced}

klihub · 2026-10-23T07:11:51Z

+		add(fmt.Sprintf("LABEL%d", i), k+": "+strconv.Quote(spec.Labels[k]))
+	}
+	for i, k := range slices.Sorted(maps.Keys(spec.Annotations)) {
+		add(fmt.Sprintf("ANN%d", i), k+": "+strconv.Quote(spec.Annotations[k]))


@bart0sh @askervin I really don't think that we'd ever want to allow/handle annotations or labels like 'true: false' anywhere and especially not in the test framework. I'd just rather ignore this.

bart0sh

/lgtm

askervin

LGTM

klihub added this pull request to stack #848 October 14, 2026 19:35

klihub changed the title ~~stacked/klihub/e2eish/reverse gen~~ e2e-ish: generate e2e tests from e2e-ish tests. Oct 14, 2026

klihub marked this pull request as ready for review October 14, 2026 19:46

klihub requested review from askervin, bart0sh and fmuyassarov October 14, 2026 19:46

askervin force-pushed the stacked/klihub/e2eish/reverse-gen branch from 459388f to 4a32c0c Compare October 15, 2026 06:14

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from 4a32c0c to c62b51a Compare October 15, 2026 07:05

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch 4 times, most recently from 7ad471c to ad95f0c Compare October 16, 2026 17:44

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from ad95f0c to 5c5c32b Compare October 19, 2026 15:15

bart0sh force-pushed the stacked/klihub/e2eish/reverse-gen branch from 5c5c32b to bb5b053 Compare October 19, 2026 15:25

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch 2 times, most recently from b76ec6a to c4b0b7d Compare October 19, 2026 19:23

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from c4b0b7d to 2526624 Compare October 20, 2026 07:55

klihub removed this pull request from stack #848 October 20, 2026 08:38

klihub added this pull request to stack #867 October 20, 2026 08:39

bart0sh force-pushed the stacked/klihub/e2eish/reverse-gen branch from 2526624 to 2c0d28b Compare October 20, 2026 09:11

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from 2c0d28b to 664aeea Compare October 20, 2026 11:26

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from 664aeea to 8fdafea Compare October 20, 2026 11:33

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from 8fdafea to 212f6dd Compare October 20, 2026 19:04

klihub requested a balanced review from Copilot October 21, 2026 15:10

Copilot started reviewing on behalf of klihub October 21, 2026 15:10

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread pkg/testutils/e2eish/e2e.go

Comment thread pkg/testutils/e2eish/pyrev.go

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from 61d4be0 to 04fdce7 Compare October 21, 2026 17:12

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from 04fdce7 to 50a64b5 Compare October 21, 2026 19:31

klihub requested a balanced review from Copilot October 21, 2026 19:37

Copilot started reviewing on behalf of klihub October 21, 2026 19:37

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Comment thread pkg/testutils/e2eish/e2e.go

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch 2 times, most recently from fd3d43f to ffb705d Compare October 22, 2026 07:32

klihub requested a balanced review from Copilot October 22, 2026 07:43

Copilot started reviewing on behalf of klihub October 22, 2026 07:44

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Comment thread pkg/testutils/e2eish/e2e.go

Base automatically changed from stacked/klihub/e2eish/go-verified-tests to main October 22, 2026 10:57

askervin force-pushed the stacked/klihub/e2eish/reverse-gen branch from ffb705d to 6cacc34 Compare October 22, 2026 10:57

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from 6cacc34 to b5adc1f Compare October 22, 2026 11:12

klihub added 3 commits October 22, 2026 14:58

klihub force-pushed the stacked/klihub/e2eish/reverse-gen branch from b5adc1f to c4a53bd Compare October 22, 2026 12:13

klihub requested a balanced review from Copilot October 22, 2026 12:14

Copilot started reviewing on behalf of klihub October 22, 2026 12:14

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

bart0sh approved these changes Oct 23, 2026

View reviewed changes

askervin approved these changes Oct 23, 2026

View reviewed changes

askervin merged commit d242f8b into main Oct 23, 2026
15 checks passed

Navigation Menu

Repository navigation

e2e-ish: generate e2e tests from e2e-ish tests. - #849

e2e-ish: generate e2e tests from e2e-ish tests.#849
askervin merged 4 commits into
mainfrom
stacked/klihub/e2eish/reverse-gen

klihub commented Oct 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

klihub Oct 23, 2026

Uh oh!

bart0sh Oct 23, 2026

Uh oh!

bart0sh left a comment

Uh oh!

askervin left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

klihub commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

klihub Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

bart0sh Oct 23, 2026

Choose a reason for hiding this comment

Uh oh!

bart0sh left a comment

Choose a reason for hiding this comment

Uh oh!

askervin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

klihub commented Oct 14, 2026 •

edited

Loading