Repository navigation
chore: proposal - opt-in end-to-end test control surface for local stacks and pull request previews - #9441
chore: proposal - opt-in end-to-end test control surface for local stacks and pull request previews#9441ThomasRooney wants to merge 5 commits into
Conversation
|
There was a problem hiding this comment.
5 issues found across 8 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="server/cmd/gram/deps_e2e.go">
<violation number="1" location="server/cmd/gram/deps_e2e.go:78">
P2: Custom agent: **Flag Security Vulnerabilities**
The reseed replaces the local organization, but its only authorization token is a checked-in constant; a caller who can reach the default `:8080` listener can retrieve it and reset that data. Require a per-developer secret or restrict this listener to loopback.</violation>
<violation number="2" location="server/cmd/gram/deps_e2e.go:89">
P1: `Run` releases the shared seed lock before `RunLocalFixtures` starts. A concurrent seed can overwrite the tenant between these phases, leaving mixed or incomplete fixtures; hold one lock across both operations.</violation>
</file>
<file name="dev-idp/internal/modes/mockworkos/workos.go">
<violation number="1" location="dev-idp/internal/modes/mockworkos/workos.go:72">
P3: `defaultUserMode` repeats `oauth21.Mode`. Export a single constant from `defaultuser` and reference it from both packages, so the slot name cannot drift between the bootstrap and the oauth2-1 resolver.</violation>
<violation number="2" location="dev-idp/internal/modes/mockworkos/workos.go:157">
P1: Local `GRAM_E2E_USER_EMAIL` overrides and case-only email differences can leave the magic-auth user without the default-org membership. Bootstrap the configured identity using one canonical email value instead of comparing it to `git config user.email`.</violation>
<violation number="3" location="dev-idp/internal/modes/mockworkos/workos.go:159">
P2: A failed bootstrap is only logged, and the sign-in still succeeds without the memberships this block exists to provide. Return a 500 as the upsert failure below does, so the test fails visibly.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| magicAuthLifetime = 10 * time.Minute | ||
| // defaultUserMode is the mode whose currentUser the default-user bootstrap | ||
| // records (oauth21.Mode; that package is not imported here). | ||
| defaultUserMode = "oauth2-1" |
There was a problem hiding this comment.
P3: defaultUserMode repeats oauth21.Mode. Export a single constant from defaultuser and reference it from both packages, so the slot name cannot drift between the bootstrap and the oauth2-1 resolver.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At dev-idp/internal/modes/mockworkos/workos.go, line 72:
<comment>`defaultUserMode` repeats `oauth21.Mode`. Export a single constant from `defaultuser` and reference it from both packages, so the slot name cannot drift between the bootstrap and the oauth2-1 resolver.</comment>
<file context>
@@ -67,6 +67,9 @@ const invitationLifetime = 30 * 24 * time.Hour
magicAuthLifetime = 10 * time.Minute
+ // defaultUserMode is the mode whose currentUser the default-user bootstrap
+ // records (oauth21.Mode; that package is not imported here).
+ defaultUserMode = "oauth2-1"
)
</file context>
There was a problem hiding this comment.
2 issues found across 8 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="server/internal/e2econtrol/e2econtrol.go">
<violation number="1" location="server/internal/e2econtrol/e2econtrol.go:266">
P2: The local HTTP configuration leaves the E2E session unusable because this cookie is always marked `Secure`. Set `Secure` from the dashboard URL scheme so HTTP loopback stacks can send the cookie.</violation>
</file>
<file name="server/cmd/gram/deps_e2e.go">
<violation number="1" location="server/cmd/gram/deps_e2e.go:26">
P2: Custom agent: **Flag Security Vulnerabilities**
The `serverFlags` registration accepts the control token as `--e2e-control-token`, exposing it in process arguments and potentially shell history. Read this secret only from `GRAM_E2E_CONTROL_TOKEN` instead of accepting a CLI value.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| Value: sessionID, | ||
| MaxAge: constants.SessionCookieMaxAgeSeconds, | ||
| Path: "/", | ||
| Secure: true, |
There was a problem hiding this comment.
P2: The local HTTP configuration leaves the E2E session unusable because this cookie is always marked Secure. Set Secure from the dashboard URL scheme so HTTP loopback stacks can send the cookie.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At server/internal/e2econtrol/e2econtrol.go, line 266:
<comment>The local HTTP configuration leaves the E2E session unusable because this cookie is always marked `Secure`. Set `Secure` from the dashboard URL scheme so HTTP loopback stacks can send the cookie.</comment>
<file context>
@@ -1,355 +1,291 @@
MaxAge: constants.SessionCookieMaxAgeSeconds,
Path: "/",
- Secure: s.cfg.Environment != "local",
+ Secure: true,
HttpOnly: true,
SameSite: http.SameSiteLaxMode,
</file context>
| Secure: true, | |
| Secure: s.cfg.SiteURL.Scheme == "https", |
|
|
||
| // e2eControlTokenFlag belongs to the API server only (serverFlags): no other | ||
| // command mounts the surface, so no other command accepts its token. | ||
| const e2eControlTokenFlag = "e2e-control-token" //nolint:gosec // G101: a flag name, not a credential |
There was a problem hiding this comment.
P2: Custom agent: Flag Security Vulnerabilities
The serverFlags registration accepts the control token as --e2e-control-token, exposing it in process arguments and potentially shell history. Read this secret only from GRAM_E2E_CONTROL_TOKEN instead of accepting a CLI value.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At server/cmd/gram/deps_e2e.go, line 26:
<comment>The `serverFlags` registration accepts the control token as `--e2e-control-token`, exposing it in process arguments and potentially shell history. Read this secret only from `GRAM_E2E_CONTROL_TOKEN` instead of accepting a CLI value.</comment>
<file context>
@@ -12,84 +12,101 @@ import (
- return strings.TrimSpace(string(out))
+// e2eControlTokenFlag belongs to the API server only (serverFlags): no other
+// command mounts the surface, so no other command accepts its token.
+const e2eControlTokenFlag = "e2e-control-token" //nolint:gosec // G101: a flag name, not a credential
+
+// e2eControlDeps is what the end-to-end test control surface is built from.
</file context>
There was a problem hiding this comment.
3 issues found across 8 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="server/internal/demoseed/queries/preview.sql">
<violation number="1" location="server/internal/demoseed/queries/preview.sql:20">
P2: Custom agent: **Flag Security Vulnerabilities**
This conflict guard checks only the email, so a matching global user already attached to another organization can be modified and then granted preview membership by `LocalAdoptDeveloper`. Refuse users with an existing foreign membership before updating or attaching them.</violation>
</file>
<file name="server/cmd/gram/deps_e2e.go">
<violation number="1" location="server/cmd/gram/deps_e2e.go:227">
P2: This cutoff can revoke a sign-in that starts after the reset transaction has committed but before `time.Now()` is sampled. Capture and use the reset transaction’s commit-time cutoff instead, so only sessions that started before the reset are ended.</violation>
</file>
<file name="server/internal/demoseed/preview_test.go">
<violation number="1" location="server/internal/demoseed/preview_test.go:126">
P3: This regression test does not fully verify that the existing user row remains unchanged: it would pass if the runner conflict rewrote `workos_id` or `display_name` while leaving the email intact. Assert the planted WorkOS ID and display name as well.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| ON CONFLICT (id) DO UPDATE | ||
| SET display_name = EXCLUDED.display_name, workos_id = EXCLUDED.workos_id, | ||
| admin = FALSE, deleted_at = NULL, updated_at = clock_timestamp() | ||
| WHERE users.email = EXCLUDED.email; |
There was a problem hiding this comment.
P2: Custom agent: Flag Security Vulnerabilities
This conflict guard checks only the email, so a matching global user already attached to another organization can be modified and then granted preview membership by LocalAdoptDeveloper. Refuse users with an existing foreign membership before updating or attaching them.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At server/internal/demoseed/queries/preview.sql, line 20:
<comment>This conflict guard checks only the email, so a matching global user already attached to another organization can be modified and then granted preview membership by `LocalAdoptDeveloper`. Refuse users with an existing foreign membership before updating or attaching them.</comment>
<file context>
@@ -6,13 +6,15 @@
- updated_at = clock_timestamp();
+ SET display_name = EXCLUDED.display_name, workos_id = EXCLUDED.workos_id,
+ admin = FALSE, deleted_at = NULL, updated_at = clock_timestamp()
+ WHERE users.email = EXCLUDED.email;
</file context>
| } | ||
| // Taken once the reset is committed, so a sign-in that read the | ||
| // organization while it was being rebuilt ends too. | ||
| if err := d.sessions.RaiseUserEpoch(ctx, spec.PreviewRunner().ID, time.Now()); err != nil { |
There was a problem hiding this comment.
P2: This cutoff can revoke a sign-in that starts after the reset transaction has committed but before time.Now() is sampled. Capture and use the reset transaction’s commit-time cutoff instead, so only sessions that started before the reset are ended.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At server/cmd/gram/deps_e2e.go, line 227:
<comment>This cutoff can revoke a sign-in that starts after the reset transaction has committed but before `time.Now()` is sampled. Capture and use the reset transaction’s commit-time cutoff instead, so only sessions that started before the reset are ended.</comment>
<file context>
@@ -216,14 +216,15 @@ func reseedLocalOrganization(ctx context.Context, logger *slog.Logger, d e2eCont
- if err := d.sessions.RaiseUserEpoch(ctx, spec.PreviewRunner().ID, started); err != nil {
+ // Taken once the reset is committed, so a sign-in that read the
+ // organization while it was being rebuilt ends too.
+ if err := d.sessions.RaiseUserEpoch(ctx, spec.PreviewRunner().ID, time.Now()); err != nil {
return fmt.Errorf("end the test user's earlier sessions: %w", err)
}
</file context>
|
|
||
| user, err := userrepo.New(db).GetUser(ctx, runner.ID) | ||
| require.NoError(t, err) | ||
| require.Equal(t, "someone-else@example.test", user.Email) |
There was a problem hiding this comment.
P3: This regression test does not fully verify that the existing user row remains unchanged: it would pass if the runner conflict rewrote workos_id or display_name while leaving the email intact. Assert the planted WorkOS ID and display name as well.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At server/internal/demoseed/preview_test.go, line 126:
<comment>This regression test does not fully verify that the existing user row remains unchanged: it would pass if the runner conflict rewrote `workos_id` or `display_name` while leaving the email intact. Assert the planted WorkOS ID and display name as well.</comment>
<file context>
@@ -91,6 +91,45 @@ func TestPreviewFixturesRefuseADatabaseWithoutBuiltInRoles(t *testing.T) {
+
+ user, err := userrepo.New(db).GetUser(ctx, runner.ID)
+ require.NoError(t, err)
+ require.Equal(t, "someone-else@example.test", user.Email)
+ organizations, err := orgrepo.New(db).ListOrganizationsForUser(ctx, conv.ToPGText(runner.ID))
+ require.NoError(t, err)
</file context>
Summary
server/internal/e2econtrolis four routes an end-to-end test run uses to put one test organization in a known state and to sign a browser in to it without driving an identity provider's pages.GET /rpc/e2e.tenantGram-E2E-TokenheaderPOST /rpc/e2e.reseedPOST /rpc/e2e.signInLinkGET /rpc/e2e.signIn?code=…No route takes an organization, a user or a destination from its caller, and no route calls an identity provider. All three are fixed when the server starts, by where the server is:
GRAM_ENVIRONMENTis exactlylocalGRAM_ENVIRONMENTis exactlydev, the database the server is connected to reports its own name asgram_preview_pr_<N>, and the dashboard and server URLs are one host whose first label ispr-<N>demoseed.LocalSpec)demoseed.PreviewSpec), with no identity provider organization behind itgit config user.email), a platform admin as on any local stackdemoseed.RunthenRunLocalFixtures, asmise run seeddoesdemoseed.Runwith no ClickHouse connection, thenRunPreviewFixtures: Postgres onlyWith the token set and neither column satisfied, the API server refuses to start. With no token (the default everywhere) nothing is constructed and the routes do not exist. Only the API server accepts the flag.
Other changes this needs:
demoseed.Runskips the whole ClickHouse phase, deletes included, when given no connection. Existing callers pass one and are unchanged.demoseed.PreviewSpec(n)anddemoseed.RunPreviewFixtures: the organization's role grants, the grants and settings the local organization also gets, and the test user's membership written last, in one transaction. It refuses a database without the built-in roles instead of creating them, makes nobody staff, creates no API key and writes no registry or catalog rows.sessions.Manager.Issuestores a session then writes its cookie. The invitation callback now calls it instead of its own copy of those two steps; its order and behaviour are otherwise unchanged.sessions.Manager.ConfineUser:UpdateSession, the one placeauth.switchScopesandauth.enterDemochange a session's organization, refuses to move a confined user's session. Only the preview's test user is ever confined.server/internal/e2econtrol/openapi.yamlis a small hand-written contract a test tool pins by digest. It now holds the three token routes only: the two public reads the first revision projected are gone, so a test run holds one credential and no API key.Motivation
The Reflow golden flows depend on two shared flows that only work against a stack booted in the same job: the suite's root checks the seed with direct SQL, and the shared sign-in clicks through the dev-idp. Going through the product instead (reseed through a route, sign in through a one-time link) lets the same flows run against the build a pull request actually deploys. A preview is where that matters most: it runs the production dashboard build and real feature flags.
Why the preview's tenant is synthetic and local to its database, rather than a test organization and user created in the identity provider:
What a preview gives up: telemetry pages show the organization's empty state, and there is no real identity provider sign-in in the loop (
login-smokecovers that locally).Threat model
The token is a secret given to the API server at startup and to the test runner. It is compared in constant time. It carries no individual attribution.
Local stack: the developer or CI job generates the token for its own stack; nothing sets one by default.
mise run seedwritesPreview: the token is one secret per environment, handed only to previews that opted in.
TestDemoSeedSafety) cover tenant isolation, and a test here runs the reset beside another tenantConfineUser), or use it on dev or another preview, where its user and organization do not existStated residuals, none of which this change tries to engineer away:
cache.Suffixthe session manager already takes) would close it; it is left out because it changes session wiring for every user of that preview.auth.switchScopesorauth.enterDemo, not a 403: the refusal is made where the session is written, and those handlers were left untouched.A sign-in code is never stored: the cache holds a boolean marker under a hash of the code together with the dashboard origin, the organization id and the user's address it was minted for, so a server configured for anything else (another preview sharing the cache) does not know it. It is redeemed with an atomic get-and-delete and carried in a query parameter the request log already redacts.
What the sign-in skips compared with a normal login: the identity provider entirely, so there is no provider session and nothing to record (
signins.RecordExchange). Because of that it gives no session where the organization has sign-in enforcement on.The startup guard
attachE2EControl(server/cmd/gram/deps_e2e.go) makes the one decision, before anything is mounted. The preview check readscurrent_database()on the pool the reset will write through: the name is the fact that makes a reset contained, it is produced by the preview's own database setup, and dev and production are connected to a database without a pull request number, so they refuse whatever else they are given. Tests drive the decision against a real mux for each environment, database name and host combination, and check that only the API server declares the flag.A local stack is admitted by its environment name. A deployed image mislabelled
localstill refuses to start with a token: local mode finds its test user throughgit config user.email, and the image has neither git nor a git identity. A developer's own binary pointed at a hosted database in local mode is not something this guard can tell from a local stack.A build tag was considered and rejected: the image that reaches production is the one promoted from dev, so the same binary has to run in both.
What changed since the local-only revision, and why
sessions.Manager.Issue), which the earlier review asked for and was deferred.Not proven, and what a preview still needs
demoseed_safetytag. On a local stack the root and a golden-shaped flow pass through the routes (reseed, sign-in through a link, a seeded page opened).preview-e2elabel that opts a pull request in. Until that is reviewed, merged and applied, the surface is off on every preview, including this pull request's.