Repository navigation
chore: proposal - Reflow target for deployed hosts and a manual preview job - #9442
ThomasRooney wants to merge 5 commits into
Conversation
|
There was a problem hiding this comment.
5 issues found across 4 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".github/workflows/reflow-deployed.yml">
<violation number="1" location=".github/workflows/reflow-deployed.yml:92">
P2: This checkout is shallow, but `reflow ci run` needs repository history to calculate changes and compare CI runs. Add `fetch-depth: 0` so later deployed runs do not fail or misclassify changed flows.</violation>
<violation number="2" location=".github/workflows/reflow-deployed.yml:111">
P1: This allowlist accepts URLs whose wildcard crosses `/`, such as `https://pr-attacker.example/.dev.getgram.ai`, so the workflow can send its E2E secrets to an untrusted host. Validate an anchored hostname with a Bash regex before making any request.</violation>
<violation number="3" location=".github/workflows/reflow-deployed.yml:152">
P2: Bound each readiness request with connect and total-operation timeouts. Without them, a stalled allowlisted host can keep this step running until the job-level 90-minute timeout, rather than failing after the documented 15-minute wait.</violation>
</file>
<file name=".mise-tasks/reflow/golden.mts">
<violation number="1" location=".mise-tasks/reflow/golden.mts:95">
P3: `goldenSlugs` compares tags case-sensitively, but `reflow ci run --label` matches labels case-insensitively. `--label Portable` therefore aborts during preflight instead of running the matching suite; compare tags case-insensitively.</violation>
</file>
<file name=".github/workflows/reflow-preview.yml">
<violation number="1" location=".github/workflows/reflow-preview.yml:71">
P1: Do not pass secrets to a job that checks out and executes the PR-selected revision. A same-repository PR can alter the `mise` tasks or Reflow files and exfiltrate `E2E_CONTROL_TOKEN`, `E2E_API_KEY`, or the Reflow token; the fork check does not make same-repository PR code trusted. Run the secret-bearing deployment/control steps from a trusted revision, or add an explicit trusted-maintainer/approval gate and keep PR-controlled test code isolated from credentials.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| case "$TARGET_URL" in | ||
| https://pr-*.dev.getgram.ai|https://dev.getgram.ai|https://dev.ai.speakeasy.com) ;; | ||
| *) | ||
| echo "::error::$TARGET_URL is not a known non-production host" | ||
| exit 1 | ||
| ;; | ||
| esac |
There was a problem hiding this comment.
P1: This allowlist accepts URLs whose wildcard crosses /, such as https://pr-attacker.example/.dev.getgram.ai, so the workflow can send its E2E secrets to an untrusted host. Validate an anchored hostname with a Bash regex before making any request.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/reflow-deployed.yml, line 111:
<comment>This allowlist accepts URLs whose wildcard crosses `/`, such as `https://pr-attacker.example/.dev.getgram.ai`, so the workflow can send its E2E secrets to an untrusted host. Validate an anchored hostname with a Bash regex before making any request.</comment>
<file context>
@@ -0,0 +1,229 @@
+
+ - name: Refuse a production host
+ run: |
+ case "$TARGET_URL" in
+ https://pr-*.dev.getgram.ai|https://dev.getgram.ai|https://dev.ai.speakeasy.com) ;;
+ *)
</file context>
| case "$TARGET_URL" in | |
| https://pr-*.dev.getgram.ai|https://dev.getgram.ai|https://dev.ai.speakeasy.com) ;; | |
| *) | |
| echo "::error::$TARGET_URL is not a known non-production host" | |
| exit 1 | |
| ;; | |
| esac | |
| if [[ ! "$TARGET_URL" =~ ^https://pr-[A-Za-z0-9-]+\.dev\.getgram\.ai$ && "$TARGET_URL" != "https://dev.getgram.ai" && "$TARGET_URL" != "https://dev.ai.speakeasy.com" ]]; then | |
| echo "::error::$TARGET_URL is not a known non-production host" | |
| exit 1 | |
| fi |
| with: | ||
| url: https://pr-${{ inputs.pr_number }}.dev.getgram.ai | ||
| expect_build: ${{ needs.resolve.outputs.short_sha }} | ||
| source_ref: ${{ needs.resolve.outputs.head_sha }} |
There was a problem hiding this comment.
P1: Do not pass secrets to a job that checks out and executes the PR-selected revision. A same-repository PR can alter the mise tasks or Reflow files and exfiltrate E2E_CONTROL_TOKEN, E2E_API_KEY, or the Reflow token; the fork check does not make same-repository PR code trusted. Run the secret-bearing deployment/control steps from a trusted revision, or add an explicit trusted-maintainer/approval gate and keep PR-controlled test code isolated from credentials.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/reflow-preview.yml, line 71:
<comment>Do not pass secrets to a job that checks out and executes the PR-selected revision. A same-repository PR can alter the `mise` tasks or Reflow files and exfiltrate `E2E_CONTROL_TOKEN`, `E2E_API_KEY`, or the Reflow token; the fork check does not make same-repository PR code trusted. Run the secret-bearing deployment/control steps from a trusted revision, or add an explicit trusted-maintainer/approval gate and keep PR-controlled test code isolated from credentials.</comment>
<file context>
@@ -0,0 +1,79 @@
+ with:
+ url: https://pr-${{ inputs.pr_number }}.dev.getgram.ai
+ expect_build: ${{ needs.resolve.outputs.short_sha }}
+ source_ref: ${{ needs.resolve.outputs.head_sha }}
+ record_branch: preview/pr-${{ inputs.pr_number }}
+ reflow_collection: ${{ vars.REFLOW_COLLECTION_ID }}
</file context>
| uses: useblacksmith/checkout@6fd481652155169ed4d2f25ebaf97464f685175f # v1 | ||
| with: | ||
| repository: speakeasy-api/gram | ||
| ref: ${{ inputs.source_ref }} |
There was a problem hiding this comment.
P2: This checkout is shallow, but reflow ci run needs repository history to calculate changes and compare CI runs. Add fetch-depth: 0 so later deployed runs do not fail or misclassify changed flows.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/reflow-deployed.yml, line 92:
<comment>This checkout is shallow, but `reflow ci run` needs repository history to calculate changes and compare CI runs. Add `fetch-depth: 0` so later deployed runs do not fail or misclassify changed flows.</comment>
<file context>
@@ -0,0 +1,229 @@
+ uses: useblacksmith/checkout@6fd481652155169ed4d2f25ebaf97464f685175f # v1
+ with:
+ repository: speakeasy-api/gram
+ ref: ${{ inputs.source_ref }}
+ persist-credentials: false
+
</file context>
| GRAM_E2E_CONTROL_TOKEN: ${{ secrets.E2E_CONTROL_TOKEN }} | ||
| run: | | ||
| for attempt in $(seq 1 60); do | ||
| if curl -fsS -o /dev/null -H "Gram-E2E-Token: $GRAM_E2E_CONTROL_TOKEN" "$TARGET_URL/rpc/e2e.tenant"; then |
There was a problem hiding this comment.
P2: Bound each readiness request with connect and total-operation timeouts. Without them, a stalled allowlisted host can keep this step running until the job-level 90-minute timeout, rather than failing after the documented 15-minute wait.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .github/workflows/reflow-deployed.yml, line 152:
<comment>Bound each readiness request with connect and total-operation timeouts. Without them, a stalled allowlisted host can keep this step running until the job-level 90-minute timeout, rather than failing after the documented 15-minute wait.</comment>
<file context>
@@ -0,0 +1,229 @@
+ GRAM_E2E_CONTROL_TOKEN: ${{ secrets.E2E_CONTROL_TOKEN }}
+ run: |
+ for attempt in $(seq 1 60); do
+ if curl -fsS -o /dev/null -H "Gram-E2E-Token: $GRAM_E2E_CONTROL_TOKEN" "$TARGET_URL/rpc/e2e.tenant"; then
+ exit 0
+ fi
</file context>
| if curl -fsS -o /dev/null -H "Gram-E2E-Token: $GRAM_E2E_CONTROL_TOKEN" "$TARGET_URL/rpc/e2e.tenant"; then | |
| if curl --connect-timeout 5 --max-time 15 -fsS -o /dev/null -H "Gram-E2E-Token: $GRAM_E2E_CONTROL_TOKEN" "$TARGET_URL/rpc/e2e.tenant"; then |
| ...page.flows.filter((f) => f.golden != null).map((f) => f.slug), | ||
| ...page.flows | ||
| .filter((f) => f.golden != null) | ||
| .filter((f) => !label || (f.tags ?? []).includes(label)) |
There was a problem hiding this comment.
P3: goldenSlugs compares tags case-sensitively, but reflow ci run --label matches labels case-insensitively. --label Portable therefore aborts during preflight instead of running the matching suite; compare tags case-insensitively.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .mise-tasks/reflow/golden.mts, line 95:
<comment>`goldenSlugs` compares tags case-sensitively, but `reflow ci run --label` matches labels case-insensitively. `--label Portable` therefore aborts during preflight instead of running the matching suite; compare tags case-insensitively.</comment>
<file context>
@@ -85,7 +90,10 @@ async function goldenSlugs(): Promise<string[]> {
- ...page.flows.filter((f) => f.golden != null).map((f) => f.slug),
+ ...page.flows
+ .filter((f) => f.golden != null)
+ .filter((f) => !label || (f.tags ?? []).includes(label))
+ .map((f) => f.slug),
);
</file context>
| .filter((f) => !label || (f.tags ?? []).includes(label)) | |
| .filter((f) => !label || (f.tags ?? []).some((tag) => tag.toLowerCase() === label.toLowerCase())) |
…s its deployed follow-up
Summary
mise run reflow:targetwithout--urlbehaves as before and additionally writes an OpenAPI binding holding the control token, so the two shared flows below run against this worktree's stack. The control surface is opt-in: whenGRAM_E2E_CONTROL_TOKENis not set the task stops and prints how to generate a private token and restart the server.mise run reflow:target --url https://<host> --expect-build <sha>aims the Reflow target at a deployed host instead: a pull request preview that carries thepreview-e2elabel. A deployed target gets a Browser binding and an OpenAPI binding and no database binding: nothing in CI holds a DSN for a deployed environment.mise run reflow:golden --label <tag>narrows every selection (the single run, the shard list and the repeat) to golden flows carrying a tag..github/workflows/reflow-deployed.yml(workflow_callonly) and.github/workflows/reflow-preview.yml(manual dispatch, a pull request number): the deployed job and its preview caller. The first refuses hosts outside the known non-production set, waits for the host to answer, and records its runs on a branch name of their own with no base branch, so a pass from a local-stack run is never carried into a deployed run or the reverse. The second refuses forks and pull requests without both thepreviewandpreview-e2elabels.The existing
reflow-golden.ymljob is unchanged.Motivation
The golden flows run against a stack the CI job boots from the branch, and two shared flows tie all of them to it: the root checks the seed with direct SQL, and the shared sign-in clicks through the dev-idp. With the control surface in #9441 both go through the product instead, and the rest of the suite runs on top of them unchanged apart from one frontmatter line (below). Running them against the image a pull request deploys is the reason for the deployed half.
The two shared flows are proposed as replacements for
gram-seed-stateandsign-in. They are not saved in the collection and replace nothing yet. Both were run against a local stack through this task.The sign-in step, in full (the digest pins
server/internal/e2econtrol/openapi.yamlas it stands on #9441):The root, step by step, each a typed call with its expectation in the flow:
e2eTenantreports a build containing the expected commit (waits up to 15 minutes).e2eReseedreturns 204: the seed, its own postflight checks and the member fixtures all completed.e2eTenantgives the organization slug, project slug, seeded id prefix and build, which become the root's outputs. No slug or id is an input.project_path,server_hostandmcp_server_slugas the current root does.The seeded state itself is asserted in the dashboard by the flows that depend on the root.
Known costs and gaps:
api: OpenAPIprovider, so each Browser golden flow that signs in gains one frontmatter line.reflow ci run <slugs> --golden --label portable(the shard path passes slugs and a label together), and the tool listinstall_args: node aube jqin the callable workflow.