Skip to content

chore: proposal - Reflow target for deployed hosts and a manual preview job - #9442

Draft
ThomasRooney wants to merge 5 commits into
chore/e2e-control-surfacefrom
chore/reflow-deployed-target
Draft

ThomasRooney wants to merge 5 commits into
chore/e2e-control-surfacefrom
chore/reflow-deployed-target

Conversation

@ThomasRooney

@ThomasRooney ThomasRooney commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Proposal for discussion. Draft, not for merge as it stands. Stacked on #9441 and useless without it. #9441 now admits a pull request preview that opted in, as well as a local stack, so the preview half of this pull request has something to call, but only once the infrastructure change that gives a preview its token is applied. Neither workflow is triggered by any event: one is callable only, the other is manual dispatch, and the one secret they need does not exist. Neither has been run.

Summary

  • mise run reflow:target without --url behaves as before and additionally writes an OpenAPI binding holding the control token, so the two shared flows below run against this worktree's stack. The control surface is opt-in: when GRAM_E2E_CONTROL_TOKEN is not set the task stops and prints how to generate a private token and restart the server.
  • mise run reflow:target --url https://<host> --expect-build <sha> aims the Reflow target at a deployed host instead: a pull request preview that carries the preview-e2e label. A deployed target gets a Browser binding and an OpenAPI binding and no database binding: nothing in CI holds a DSN for a deployed environment.
  • A target holds one credential, the control token. The API key the first revision also wrote is gone: the suite's root no longer reads the public API.
  • mise run reflow:golden --label <tag> narrows every selection (the single run, the shard list and the repeat) to golden flows carrying a tag.
  • .github/workflows/reflow-deployed.yml (workflow_call only) and .github/workflows/reflow-preview.yml (manual dispatch, a pull request number): the deployed job and its preview caller. The first refuses hosts outside the known non-production set, waits for the host to answer, and records its runs on a branch name of their own with no base branch, so a pass from a local-stack run is never carried into a deployed run or the reverse. The second refuses forks and pull requests without both the preview and preview-e2e labels.

The existing reflow-golden.yml job is unchanged.

Motivation

The golden flows run against a stack the CI job boots from the branch, and two shared flows tie all of them to it: the root checks the seed with direct SQL, and the shared sign-in clicks through the dev-idp. With the control surface in #9441 both go through the product instead, and the rest of the suite runs on top of them unchanged apart from one frontmatter line (below). Running them against the image a pull request deploys is the reason for the deployed half.

The two shared flows are proposed as replacements for gram-seed-state and sign-in. They are not saved in the collection and replace nothing yet. Both were run against a local stack through this task.

The sign-in step, in full (the digest pins server/internal/e2econtrol/openapi.yaml as it stands on #9441):

### Sign in
```rfl timeout=120s
api.call_value (
  operationId="e2eSignInLink"
  spec="sha256:886174f8414001a9ffe6bd8bd3e5737f18861921b57dd585f10f5fa054bba89f"
  expect=json(
    {"status":200}
  )
  select="/path"
) -> sign_in_path
open ${sign_in_path}
expect testid=user-menu-trigger visible timeout=50000ms
```

The root, step by step, each a typed call with its expectation in the flow:

  1. e2eTenant reports a build containing the expected commit (waits up to 15 minutes).
  2. e2eReseed returns 204: the seed, its own postflight checks and the member fixtures all completed.
  3. e2eTenant gives the organization slug, project slug, seeded id prefix and build, which become the root's outputs. No slug or id is an input.
  4. It publishes project_path, server_host and mcp_server_slug as the current root does.

The seeded state itself is asserted in the dashboard by the flows that depend on the root.

Known costs and gaps:

  • A flow that uses the new sign-in must declare the api: OpenAPI provider, so each Browser golden flow that signs in gains one frontmatter line.
  • Read against a preview's synthetic tenant, 36 of the 46 Browser golden flows that sign in would run with their own steps unchanged. Of the rest: three hard-code local seed identifiers (a seeded user id, a gateway id, a deployment id) and need to read them from the root; three read SQL in their own steps; one asserts telemetry, which a preview does not seed; three need a look on a real preview (a feature flag, a staff-only panel used as a snapshot mask, the default environment). Snapshot baselines recorded on a local stack will differ on a preview wherever a page shows the organization's name, host or members.
  • With no base branch a deployed run executes every portable flow, in shards, since it cannot borrow passes from local-stack runs.
  • The job runs the pull request's own Reflow tasks while holding the control token, which is why forks are refused. The token's reach is synthetic tenants on opted-in previews.
  • Unverified: reflow ci run <slugs> --golden --label portable (the shard path passes slugs and a label together), and the tool list install_args: node aube jq in the callable workflow.

@changeset-bot

changeset-bot Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 836a2f6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5 issues found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".github/workflows/reflow-deployed.yml">

<violation number="1" location=".github/workflows/reflow-deployed.yml:92">
P2: This checkout is shallow, but `reflow ci run` needs repository history to calculate changes and compare CI runs. Add `fetch-depth: 0` so later deployed runs do not fail or misclassify changed flows.</violation>

<violation number="2" location=".github/workflows/reflow-deployed.yml:111">
P1: This allowlist accepts URLs whose wildcard crosses `/`, such as `https://pr-attacker.example/.dev.getgram.ai`, so the workflow can send its E2E secrets to an untrusted host. Validate an anchored hostname with a Bash regex before making any request.</violation>

<violation number="3" location=".github/workflows/reflow-deployed.yml:152">
P2: Bound each readiness request with connect and total-operation timeouts. Without them, a stalled allowlisted host can keep this step running until the job-level 90-minute timeout, rather than failing after the documented 15-minute wait.</violation>
</file>

<file name=".mise-tasks/reflow/golden.mts">

<violation number="1" location=".mise-tasks/reflow/golden.mts:95">
P3: `goldenSlugs` compares tags case-sensitively, but `reflow ci run --label` matches labels case-insensitively. `--label Portable` therefore aborts during preflight instead of running the matching suite; compare tags case-insensitively.</violation>
</file>

<file name=".github/workflows/reflow-preview.yml">

<violation number="1" location=".github/workflows/reflow-preview.yml:71">
P1: Do not pass secrets to a job that checks out and executes the PR-selected revision. A same-repository PR can alter the `mise` tasks or Reflow files and exfiltrate `E2E_CONTROL_TOKEN`, `E2E_API_KEY`, or the Reflow token; the fork check does not make same-repository PR code trusted. Run the secret-bearing deployment/control steps from a trusted revision, or add an explicit trusted-maintainer/approval gate and keep PR-controlled test code isolated from credentials.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment on lines +111 to +117
case "$TARGET_URL" in
https://pr-*.dev.getgram.ai|https://dev.getgram.ai|https://dev.ai.speakeasy.com) ;;
*)
echo "::error::$TARGET_URL is not a known non-production host"
exit 1
;;
esac

@cubic-dev-ai cubic-dev-ai Bot Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This allowlist accepts URLs whose wildcard crosses /, such as https://pr-attacker.example/.dev.getgram.ai, so the workflow can send its E2E secrets to an untrusted host. Validate an anchored hostname with a Bash regex before making any request.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/reflow-deployed.yml, line 111:

<comment>This allowlist accepts URLs whose wildcard crosses `/`, such as `https://pr-attacker.example/.dev.getgram.ai`, so the workflow can send its E2E secrets to an untrusted host. Validate an anchored hostname with a Bash regex before making any request.</comment>

<file context>
@@ -0,0 +1,229 @@
+
+      - name: Refuse a production host
+        run: |
+          case "$TARGET_URL" in
+            https://pr-*.dev.getgram.ai|https://dev.getgram.ai|https://dev.ai.speakeasy.com) ;;
+            *)
</file context>
Suggested change
case "$TARGET_URL" in
https://pr-*.dev.getgram.ai|https://dev.getgram.ai|https://dev.ai.speakeasy.com) ;;
*)
echo "::error::$TARGET_URL is not a known non-production host"
exit 1
;;
esac
if [[ ! "$TARGET_URL" =~ ^https://pr-[A-Za-z0-9-]+\.dev\.getgram\.ai$ && "$TARGET_URL" != "https://dev.getgram.ai" && "$TARGET_URL" != "https://dev.ai.speakeasy.com" ]]; then
echo "::error::$TARGET_URL is not a known non-production host"
exit 1
fi
Fix with cubic

with:
url: https://pr-${{ inputs.pr_number }}.dev.getgram.ai
expect_build: ${{ needs.resolve.outputs.short_sha }}
source_ref: ${{ needs.resolve.outputs.head_sha }}

@cubic-dev-ai cubic-dev-ai Bot Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Do not pass secrets to a job that checks out and executes the PR-selected revision. A same-repository PR can alter the mise tasks or Reflow files and exfiltrate E2E_CONTROL_TOKEN, E2E_API_KEY, or the Reflow token; the fork check does not make same-repository PR code trusted. Run the secret-bearing deployment/control steps from a trusted revision, or add an explicit trusted-maintainer/approval gate and keep PR-controlled test code isolated from credentials.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/reflow-preview.yml, line 71:

<comment>Do not pass secrets to a job that checks out and executes the PR-selected revision. A same-repository PR can alter the `mise` tasks or Reflow files and exfiltrate `E2E_CONTROL_TOKEN`, `E2E_API_KEY`, or the Reflow token; the fork check does not make same-repository PR code trusted. Run the secret-bearing deployment/control steps from a trusted revision, or add an explicit trusted-maintainer/approval gate and keep PR-controlled test code isolated from credentials.</comment>

<file context>
@@ -0,0 +1,79 @@
+    with:
+      url: https://pr-${{ inputs.pr_number }}.dev.getgram.ai
+      expect_build: ${{ needs.resolve.outputs.short_sha }}
+      source_ref: ${{ needs.resolve.outputs.head_sha }}
+      record_branch: preview/pr-${{ inputs.pr_number }}
+      reflow_collection: ${{ vars.REFLOW_COLLECTION_ID }}
</file context>
Fix with cubic

uses: useblacksmith/checkout@6fd481652155169ed4d2f25ebaf97464f685175f # v1
with:
repository: speakeasy-api/gram
ref: ${{ inputs.source_ref }}

@cubic-dev-ai cubic-dev-ai Bot Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This checkout is shallow, but reflow ci run needs repository history to calculate changes and compare CI runs. Add fetch-depth: 0 so later deployed runs do not fail or misclassify changed flows.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/reflow-deployed.yml, line 92:

<comment>This checkout is shallow, but `reflow ci run` needs repository history to calculate changes and compare CI runs. Add `fetch-depth: 0` so later deployed runs do not fail or misclassify changed flows.</comment>

<file context>
@@ -0,0 +1,229 @@
+        uses: useblacksmith/checkout@6fd481652155169ed4d2f25ebaf97464f685175f # v1
+        with:
+          repository: speakeasy-api/gram
+          ref: ${{ inputs.source_ref }}
+          persist-credentials: false
+
</file context>
Fix with cubic

GRAM_E2E_CONTROL_TOKEN: ${{ secrets.E2E_CONTROL_TOKEN }}
run: |
for attempt in $(seq 1 60); do
if curl -fsS -o /dev/null -H "Gram-E2E-Token: $GRAM_E2E_CONTROL_TOKEN" "$TARGET_URL/rpc/e2e.tenant"; then

@cubic-dev-ai cubic-dev-ai Bot Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Bound each readiness request with connect and total-operation timeouts. Without them, a stalled allowlisted host can keep this step running until the job-level 90-minute timeout, rather than failing after the documented 15-minute wait.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/reflow-deployed.yml, line 152:

<comment>Bound each readiness request with connect and total-operation timeouts. Without them, a stalled allowlisted host can keep this step running until the job-level 90-minute timeout, rather than failing after the documented 15-minute wait.</comment>

<file context>
@@ -0,0 +1,229 @@
+          GRAM_E2E_CONTROL_TOKEN: ${{ secrets.E2E_CONTROL_TOKEN }}
+        run: |
+          for attempt in $(seq 1 60); do
+            if curl -fsS -o /dev/null -H "Gram-E2E-Token: $GRAM_E2E_CONTROL_TOKEN" "$TARGET_URL/rpc/e2e.tenant"; then
+              exit 0
+            fi
</file context>
Suggested change
if curl -fsS -o /dev/null -H "Gram-E2E-Token: $GRAM_E2E_CONTROL_TOKEN" "$TARGET_URL/rpc/e2e.tenant"; then
if curl --connect-timeout 5 --max-time 15 -fsS -o /dev/null -H "Gram-E2E-Token: $GRAM_E2E_CONTROL_TOKEN" "$TARGET_URL/rpc/e2e.tenant"; then
Fix with cubic

...page.flows.filter((f) => f.golden != null).map((f) => f.slug),
...page.flows
.filter((f) => f.golden != null)
.filter((f) => !label || (f.tags ?? []).includes(label))

@cubic-dev-ai cubic-dev-ai Bot Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: goldenSlugs compares tags case-sensitively, but reflow ci run --label matches labels case-insensitively. --label Portable therefore aborts during preflight instead of running the matching suite; compare tags case-insensitively.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .mise-tasks/reflow/golden.mts, line 95:

<comment>`goldenSlugs` compares tags case-sensitively, but `reflow ci run --label` matches labels case-insensitively. `--label Portable` therefore aborts during preflight instead of running the matching suite; compare tags case-insensitively.</comment>

<file context>
@@ -85,7 +90,10 @@ async function goldenSlugs(): Promise<string[]> {
-      ...page.flows.filter((f) => f.golden != null).map((f) => f.slug),
+      ...page.flows
+        .filter((f) => f.golden != null)
+        .filter((f) => !label || (f.tags ?? []).includes(label))
+        .map((f) => f.slug),
     );
</file context>
Suggested change
.filter((f) => !label || (f.tags ?? []).includes(label))
.filter((f) => !label || (f.tags ?? []).some((tag) => tag.toLowerCase() === label.toLowerCase()))
Fix with cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant