fix(sse): cover opencode-zen with the egress-IP lockout and feed body-text rules

changelog.d/fixes/16797-opencode-zen-missing-egress-bucket.md

@@ -0,0 +1 @@
+- **fix(sse):** opt-in `opencode-zen` egress-IP lockout (`OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED`) and opt-in full-text error-rule bodies for the opencode family (`OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED`) and `cloudflare-ai` (`CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED`), all off by default so the release behavior is unchanged ([#16797](https://github.com/diegosouzapw/OmniRoute/pull/16797)) — thanks @maxmad64bis

docs/reference/FEATURE_FLAGS.md

@@ -46,7 +46,7 @@ A boolean flag is considered **enabled** when its effective value is `"true"`,
 
 ## Flag Catalog
 
-100 flags across 6 categories. **Default** is the definition default — the value
+103 flags across 6 categories. **Default** is the definition default — the value
 used when neither a DB override nor an environment variable is present.
 
 ### Security (10)
@@ -64,7 +64,7 @@ used when neither a DB override nor an environment variable is present.
 | `AUTH_LOG_INCLUDE_ACCOUNT_ID`           | boolean | `false`  | Include account prefix in AUTH log lines (e.g. "Using <provider> account: abc12345..."). Disabled by default so account identifiers are redacted from shared/multi-tenant process logs. Independent from Debug Mode; flipping Debug Mode does not reveal this. |
 | `OMNIROUTE_OIDC_DISABLE_PASSWORD_LOGIN` | boolean | `false`  | When OIDC is enabled, disable password login so users can only authenticate via OIDC Single Sign-On. When disabled (default), both password login and OIDC are available.                                                                                      |
 
-### Network (24)
+### Network (27)
 
 | Key                                             | Type    | Default | Restart | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
 | ----------------------------------------------- | ------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -87,6 +87,9 @@ used when neither a DB override nor an environment variable is present.
 | `FLUSH_EMPTY_RETRY_ENABLED`                     | boolean | `false` |         | On translated streaming turns, when the upstream turn carries no usable content (reasoning-only completion or zero valuable chunks), issue bounded retries through the normal credential path (up to `STREAM_RECOVERY.EMPTY_TURN_RETRY_MAX`) before anything is exposed to the client. Off by default: empty turns keep the current behavior (empty 200 or empty-content 502).                                                                                                                                      |
 | `STREAM_LATE_OUTPUT_PROBE_ENABLED`              | boolean | `false` |         | Streaming output probe: after the first 64 forwarded chunks keep sampling one chunk in 16 for the first visible output, so output that starts late still marks the stream as having emitted output. A lone 5xx after that point then stays request-scoped instead of recording a model-only lockout, and `firstOutputMs` is back-filled to the first output chunk of the current sampling gap. Off by default: only the first 64 forwarded chunks are probed, exactly as before.                                    |
 | `OPENCODE_POOL_RESELECT`                        | boolean | `false` |         | OpenCode rotation: after a 429 from an egress-bucketed provider on a proxy-less account under an ambient pool context, ask the connection pool for another member for the next attempt instead of retrying the same egress address. Orders, never excludes: an exhausted pool keeps the current behavior. Off by default: every 429 rotates to the next account exactly as before.                                                                                                                                  |
+| `OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED`           | boolean | `false` |         | Add `opencode-zen` to the default egress-IP lockout family: a 429 on one Zen connection also cools every sibling Zen connection sharing its egress IP (#16797). Off by default: each Zen connection cools down on its own, as before. An explicit `OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS` value always wins.                                                                                                                                                                                                           |
+| `OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED`        | boolean | `false` |         | The built-in OpenCode error rules (`opencode-zen`, `opencode-go`, `opencode-cli`) match on the full upstream error text, so body markers such as `organization_quota_exceeded` or "monthly usage limit reached" classify a 429 as quota exhausted with a connection-scoped lock (#16797). Off by default: the rules only see the structured error code, as before.                                                                                                                                                  |
+| `CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED`   | boolean | `false` |         | The built-in Cloudflare Workers AI error rule matches on the full upstream error text, so the 429 "daily free allocation" body is classified as quota exhausted instead of a short rate limit (#16797). Off by default: the rule only sees the structured error code, as before.                                                                                                                                                                                                                                    |
 | `OPENCODE_RATE_LIMITED_429_EARLY_STOP`          | boolean | `false` |         | OpenCode rotation: stop the account wave at the first 429 classified as a real rate limit (parseable `Retry-After`, or a body naming a rate/usage limit) and return that upstream 429 unchanged. Unclassified 429s keep rotating. Off by default: the free tier is limited per egress IP (#9611), so every 429 rotates and an exhausted wave returns the last upstream 429.                                                                                                                                         |
 | `MITM_DISABLE_TLS_VERIFY`                       | boolean | `false` | ✓       | Disable TLS certificate verification for the MITM proxy. **Danger.**                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS`         | boolean | `false` |         | Turns off the outbound URL guard's host checks, cloud-metadata block included, on provider URL validation, model discovery, provider-node base URLs, and the proxy-fallback test, and allows private webhook targets. Local and LAN URLs already pass by default on the validation, discovery, and provider-node paths (`OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS`); the proxy-fallback test and private webhook targets consult only `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` and block local/LAN hosts while it is off. |
@@ -241,7 +244,7 @@ Returns every flag with its effective value, source, and a summary.
       "requiresRestart": false,
       "warningLevel": "caution",
     },
-    // ... all 100 flags
+    // ... all 103 flags
   ],
   "summary": {
     "total": 56,

open-sse/config/providerErrorRules.ts

@@ -411,8 +411,70 @@ const EGRESS_BUCKETED_LOCK_PROVIDERS = egressIpLockProvidersFromEnv(
   process.env.OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS
 );
 
+/**
+ * Feature-flag reader injected by a module that is allowed to read the flag store
+ * (`open-sse/services/accountFallback.ts` registers `isFeatureFlagEnabled`). This file
+ * stays free of `src/` imports: it is reachable from client bundles and the
+ * open-sse → src boundary is ratcheted. Unregistered or throwing reader = flag off, so
+ * every flag below fails closed to the release default.
+ */
+let featureFlagReader: ((key: string) => boolean) | null = null;
+
+export function setProviderErrorRuleFlagReader(reader: ((key: string) => boolean) | null): void {
+  featureFlagReader = reader;
+}
+
+function isRuleFlagEnabled(key: string): boolean {
+  if (!featureFlagReader) return false;
+  try {
+    return featureFlagReader(key) === true;
+  } catch {
+    return false;
+  }
+}
+
+/**
+ * Opt-in (#16797): OpenCode Zen joins the default egress-IP lock family. Off by default
+ * because Zen is keyed (account-bucketed paid plans would see one 429 park every sibling
+ * connection sharing an egress IP). An explicit OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS value
+ * (a list or `none`) always wins over the flag.
+ */
+export const OPENCODE_ZEN_EGRESS_IP_LOCK_FLAG = "OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED";
+const OPENCODE_ZEN_PROVIDER_ID = "opencode-zen";
+
+/**
+ * Pure resolver for the effective egress-lock set (exported for tests): the explicit env
+ * value wins; otherwise the default family, plus `opencode-zen` when the opt-in flag is on.
+ */
+export function resolveEgressIpLockProviders(
+  raw: string | undefined,
+  zenOptIn: boolean
+): Set<string> {
+  const set = egressIpLockProvidersFromEnv(raw);
+  if (zenOptIn && (raw === undefined || raw.trim() === "")) set.add(OPENCODE_ZEN_PROVIDER_ID);
+  return set;
+}
+
+const EGRESS_LOCK_ENV_IS_EXPLICIT = Boolean(process.env.OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS?.trim());
+const EGRESS_BUCKETED_LOCK_PROVIDERS_WITH_ZEN = resolveEgressIpLockProviders(
+  process.env.OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS,
+  true
+);
+
+/** The effective set: the flag is read per call so a dashboard override applies live. */
+function effectiveEgressBucketedLockProviders(): Set<string> {
+  if (EGRESS_LOCK_ENV_IS_EXPLICIT) return EGRESS_BUCKETED_LOCK_PROVIDERS;
+  return isRuleFlagEnabled(OPENCODE_ZEN_EGRESS_IP_LOCK_FLAG)
+    ? EGRESS_BUCKETED_LOCK_PROVIDERS_WITH_ZEN
+    : EGRESS_BUCKETED_LOCK_PROVIDERS;
+}
+
 export function isEgressBucketedLockScope(provider: string | null | undefined): boolean {
-  return !!provider && EGRESS_BUCKETED_LOCK_PROVIDERS.has(provider.toLowerCase());
+  if (!provider) return false;
+  const key = provider.toLowerCase();
+  if (EGRESS_BUCKETED_LOCK_PROVIDERS.has(key)) return true;
+  // Only an opencode-zen lookup ever reads the flag store (hot path for everyone else).
+  return key === OPENCODE_ZEN_PROVIDER_ID && effectiveEgressBucketedLockProviders().has(key);
 }
 
 /**
@@ -424,7 +486,7 @@ export function isEgressBucketedLockScope(provider: string | null | undefined):
  * applied.
  */
 export function egressBucketedLockProviders(): string[] {
-  return [...EGRESS_BUCKETED_LOCK_PROVIDERS].sort();
+  return [...effectiveEgressBucketedLockProviders()].sort();
 }
 
 /**
@@ -444,6 +506,31 @@ export function egressBucketedLockProviders(): string[] {
  */
 const FULL_TEXT_RULE_PROVIDERS = new Set(["agentrouter"]);
 
+/**
+ * Opt-in widenings of FULL_TEXT_RULE_PROVIDERS (#16797), one flag per provider family.
+ * The opencode and cloudflare-ai catalog rules above match on body-text markers
+ * ("organization_quota_exceeded", "monthly usage limit reached", "daily free
+ * allocation", ...) that the stripped structured error never carries, so today they
+ * cannot fire through checkFallbackError. Letting them fire changes the classification
+ * (rate_limit → quota_exhausted) and the cooldown/lock scope, hence default-off flags.
+ */
+export const OPENCODE_FULL_TEXT_ERROR_RULES_FLAG = "OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED";
+export const CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_FLAG =
+  "CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED";
+const OPT_IN_FULL_TEXT_RULE_PROVIDERS: ReadonlyArray<{
+  flag: string;
+  providers: ReadonlySet<string>;
+}> = [
+  { flag: OPENCODE_FULL_TEXT_ERROR_RULES_FLAG, providers: new Set(OPENCODE_RULE_FAMILY) },
+  { flag: CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_FLAG, providers: new Set(["cloudflare-ai"]) },
+];
+
+function usesFullTextRuleBody(key: string): boolean {
+  if (FULL_TEXT_RULE_PROVIDERS.has(key)) return true;
+  const optIn = OPT_IN_FULL_TEXT_RULE_PROVIDERS.find((entry) => entry.providers.has(key));
+  return !!optIn && isRuleFlagEnabled(optIn.flag);
+}
+
 /**
  * True when an operator has declared at least one rule for this provider via
  * `settings.providerErrorRules` (injected through `setOperatorProviderErrorRules`).
@@ -469,8 +556,7 @@ export function resolveRuleMatchBody(
 ): unknown {
   if (
     provider &&
-    (FULL_TEXT_RULE_PROVIDERS.has(provider.toLowerCase()) ||
-      hasOperatorRuleForProvider(provider)) &&
+    (usesFullTextRuleBody(provider.toLowerCase()) || hasOperatorRuleForProvider(provider)) &&
     errorText
   ) {
     return errorText;

open-sse/services/accountFallback.ts

@@ -20,6 +20,7 @@ import {
   getProviderErrorRuleMatch,
   resolveRuleMatchBody,
   honorsRuleLockScope,
+  setProviderErrorRuleFlagReader,
 } from "../config/providerErrorRules.ts";
 import * as rot from "./rotationConfig.ts";
 import { isContentPolicyRefusal } from "../utils/contentPolicyError.ts";
@@ -117,8 +118,16 @@ import { isCreditsExhaustedWithSharedWallet } from "./accountFallback/sharedWall
 import { CREDITS_EXHAUSTED_SIGNALS } from "./accountFallback/creditExhaustionSignals.ts";
 export { CREDITS_EXHAUSTED_SIGNALS } from "./accountFallback/creditExhaustionSignals.ts";
 import { isMistralAmbiguous401 } from "./accountFallback/mistralAmbiguousAuth.ts";
-import { isMistralAmbiguous401SoftLockoutEnabled } from "@/shared/utils/featureFlags";
+import {
+  isFeatureFlagEnabled,
+  isMistralAmbiguous401SoftLockoutEnabled,
+} from "@/shared/utils/featureFlags";
 import { boundedMap } from "@/lib/quota/boundedMap";
+// #16797: providerErrorRules.ts stays free of src/ imports; its opt-in flags
+// (OpenCode Zen egress-IP lock, full-text rule bodies) read the flag store through
+// this injected reader. Not registered = flags off (release default).
+setProviderErrorRuleFlagReader(isFeatureFlagEnabled);
+
 export type ProviderProfile = {
   baseCooldownMs: number;
   useUpstreamRetryHints: boolean;

src/shared/constants/featureFlagDefinitions.ts

@@ -372,6 +372,42 @@ export const FEATURE_FLAG_DEFINITIONS: FeatureFlagDefinition[] = [
     requiresRestart: false,
     warningLevel: "caution",
   },
+  {
+    key: "OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED",
+    label: "OpenCode Zen Egress-IP Lockout",
+    description:
+      "Add OpenCode Zen (opencode-zen) to the default egress-IP lockout family, so a 429 on one Zen connection also cools every sibling Zen connection that shares its egress IP. Off by default: Zen is keyed and each connection cools down on its own, exactly as before. An explicit OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS value always wins over this flag.",
+    descriptionI18nKey: "featureFlagOpencodeZenEgressIpLockEnabledDescription",
+    category: "network",
+    defaultValue: "false",
+    type: "boolean",
+    requiresRestart: false,
+    warningLevel: "caution",
+  },
+  {
+    key: "OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED",
+    label: "OpenCode Full-Text Error Rules",
+    description:
+      'Let the built-in OpenCode error rules (opencode-zen, opencode-go, opencode-cli) match on the full upstream error text, so body markers such as organization_quota_exceeded or "monthly usage limit reached" classify a 429 as quota exhausted with a connection-scoped lock. Off by default: those rules only see the structured error code and such a 429 stays a short rate limit, exactly as before.',
+    descriptionI18nKey: "featureFlagOpencodeFullTextErrorRulesEnabledDescription",
+    category: "network",
+    defaultValue: "false",
+    type: "boolean",
+    requiresRestart: false,
+    warningLevel: "caution",
+  },
+  {
+    key: "CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED",
+    label: "Cloudflare Workers AI Full-Text Error Rules",
+    description:
+      'Let the built-in Cloudflare Workers AI error rule match on the full upstream error text, so the 429 "daily free allocation" body is classified as quota exhausted instead of a short rate limit. Off by default: the rule only sees the structured error code, exactly as before.',
+    descriptionI18nKey: "featureFlagCloudflareAiFullTextErrorRulesEnabledDescription",
+    category: "network",
+    defaultValue: "false",
+    type: "boolean",
+    requiresRestart: false,
+    warningLevel: "caution",
+  },
   {
     key: "MITM_DISABLE_TLS_VERIFY",
     label: "Disable TLS Verify (MITM)",

tests/unit/egress-ip-lock-10880.test.ts

@@ -217,6 +217,56 @@ test("three connections sharing one egress IP: ONE upstream call for the whole p
   assert.equal(c!.testStatus, "unavailable");
 });
 
+// #16797: opencode-zen is keyed, so it joins the egress family only behind the
+// default-off OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED flag. Flag off = the tip's behavior
+// (every Zen connection is tried and cooled on its own); flag on = one call per pool.
+async function rotateZenPool(): Promise<{ upstreamCalls: number; siblingCooled: boolean }> {
+  await resetStorage();
+  const connA = await seedConnection("opencode-zen");
+  const connB = await seedConnection("opencode-zen");
+  seedProxyLog(connA, SHARED_EGRESS_IP, "opencode-zen");
+  seedProxyLog(connB, SHARED_EGRESS_IP, "opencode-zen");
+  // RATE_LIMIT_TEXT_429 carries no quota text, so a lone connection keeps a short
+  // scaled cooldown and the sibling's state is decided only by the egress branch.
+  const first = await auth.markAccountUnavailable(
+    connA,
+    429,
+    '{"error":{"message":"rate limit reached"}}',
+    "opencode-zen",
+    "model-x",
+    null,
+    {}
+  );
+  assert.equal(first.shouldFallback, true);
+  const b = await providersDb.getProviderConnectionById(connB);
+  const siblingCooled =
+    !!b?.rateLimitedUntil && new Date(String(b.rateLimitedUntil)).getTime() > Date.now();
+  const { upstreamCalls } = await rotateWholePool(
+    [connB],
+    '{"error":{"message":"rate limit reached"}}',
+    "opencode-zen"
+  );
+  return { upstreamCalls: upstreamCalls + 1, siblingCooled };
+}
+
+test("#16797 flag off (default): an opencode-zen 429 leaves its egress siblings alone", async () => {
+  delete process.env.OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED;
+  const { upstreamCalls, siblingCooled } = await rotateZenPool();
+  assert.equal(siblingCooled, false, "zen sibling must not be pre-cooled by default");
+  assert.equal(upstreamCalls, 2, "each zen connection is still tried on its own");
+});
+
+test("#16797 flag on: an opencode-zen 429 cools the connections sharing its egress IP", async () => {
+  process.env.OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED = "true";
+  try {
+    const { upstreamCalls, siblingCooled } = await rotateZenPool();
+    assert.equal(siblingCooled, true, "zen sibling cooled before being tried");
+    assert.equal(upstreamCalls, 1, "exactly one upstream call for the zen pool");
+  } finally {
+    delete process.env.OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED;
+  }
+});
+
 test("counter-test: the SAME rotation burns N calls for a non-allowlisted provider", async () => {
   await resetStorage();
   const connA = await seedConnection("ollama-cloud");