diff --git a/changelog.d/fixes/16797-opencode-zen-missing-egress-bucket.md b/changelog.d/fixes/16797-opencode-zen-missing-egress-bucket.md new file mode 100644 index 000000000000..9fd503138e7e --- /dev/null +++ b/changelog.d/fixes/16797-opencode-zen-missing-egress-bucket.md @@ -0,0 +1 @@ +- **fix(sse):** opt-in `opencode-zen` egress-IP lockout (`OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED`) and opt-in full-text error-rule bodies for the opencode family (`OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED`) and `cloudflare-ai` (`CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED`), all off by default so the release behavior is unchanged ([#16797](https://github.com/diegosouzapw/OmniRoute/pull/16797)) — thanks @maxmad64bis diff --git a/docs/reference/FEATURE_FLAGS.md b/docs/reference/FEATURE_FLAGS.md index d08a8020a1c8..dc816afe354b 100644 --- a/docs/reference/FEATURE_FLAGS.md +++ b/docs/reference/FEATURE_FLAGS.md @@ -46,7 +46,7 @@ A boolean flag is considered **enabled** when its effective value is `"true"`, ## Flag Catalog -100 flags across 6 categories. **Default** is the definition default — the value +103 flags across 6 categories. **Default** is the definition default — the value used when neither a DB override nor an environment variable is present. ### Security (10) @@ -64,7 +64,7 @@ used when neither a DB override nor an environment variable is present. | `AUTH_LOG_INCLUDE_ACCOUNT_ID` | boolean | `false` | Include account prefix in AUTH log lines (e.g. "Using account: abc12345..."). Disabled by default so account identifiers are redacted from shared/multi-tenant process logs. Independent from Debug Mode; flipping Debug Mode does not reveal this. | | `OMNIROUTE_OIDC_DISABLE_PASSWORD_LOGIN` | boolean | `false` | When OIDC is enabled, disable password login so users can only authenticate via OIDC Single Sign-On. When disabled (default), both password login and OIDC are available. | -### Network (24) +### Network (27) | Key | Type | Default | Restart | Description | | ----------------------------------------------- | ------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -87,6 +87,9 @@ used when neither a DB override nor an environment variable is present. | `FLUSH_EMPTY_RETRY_ENABLED` | boolean | `false` | | On translated streaming turns, when the upstream turn carries no usable content (reasoning-only completion or zero valuable chunks), issue bounded retries through the normal credential path (up to `STREAM_RECOVERY.EMPTY_TURN_RETRY_MAX`) before anything is exposed to the client. Off by default: empty turns keep the current behavior (empty 200 or empty-content 502). | | `STREAM_LATE_OUTPUT_PROBE_ENABLED` | boolean | `false` | | Streaming output probe: after the first 64 forwarded chunks keep sampling one chunk in 16 for the first visible output, so output that starts late still marks the stream as having emitted output. A lone 5xx after that point then stays request-scoped instead of recording a model-only lockout, and `firstOutputMs` is back-filled to the first output chunk of the current sampling gap. Off by default: only the first 64 forwarded chunks are probed, exactly as before. | | `OPENCODE_POOL_RESELECT` | boolean | `false` | | OpenCode rotation: after a 429 from an egress-bucketed provider on a proxy-less account under an ambient pool context, ask the connection pool for another member for the next attempt instead of retrying the same egress address. Orders, never excludes: an exhausted pool keeps the current behavior. Off by default: every 429 rotates to the next account exactly as before. | +| `OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED` | boolean | `false` | | Add `opencode-zen` to the default egress-IP lockout family: a 429 on one Zen connection also cools every sibling Zen connection sharing its egress IP (#16797). Off by default: each Zen connection cools down on its own, as before. An explicit `OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS` value always wins. | +| `OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED` | boolean | `false` | | The built-in OpenCode error rules (`opencode-zen`, `opencode-go`, `opencode-cli`) match on the full upstream error text, so body markers such as `organization_quota_exceeded` or "monthly usage limit reached" classify a 429 as quota exhausted with a connection-scoped lock (#16797). Off by default: the rules only see the structured error code, as before. | +| `CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED` | boolean | `false` | | The built-in Cloudflare Workers AI error rule matches on the full upstream error text, so the 429 "daily free allocation" body is classified as quota exhausted instead of a short rate limit (#16797). Off by default: the rule only sees the structured error code, as before. | | `OPENCODE_RATE_LIMITED_429_EARLY_STOP` | boolean | `false` | | OpenCode rotation: stop the account wave at the first 429 classified as a real rate limit (parseable `Retry-After`, or a body naming a rate/usage limit) and return that upstream 429 unchanged. Unclassified 429s keep rotating. Off by default: the free tier is limited per egress IP (#9611), so every 429 rotates and an exhausted wave returns the last upstream 429. | | `MITM_DISABLE_TLS_VERIFY` | boolean | `false` | ✓ | Disable TLS certificate verification for the MITM proxy. **Danger.** | | `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` | boolean | `false` | | Turns off the outbound URL guard's host checks, cloud-metadata block included, on provider URL validation, model discovery, provider-node base URLs, and the proxy-fallback test, and allows private webhook targets. Local and LAN URLs already pass by default on the validation, discovery, and provider-node paths (`OMNIROUTE_ALLOW_LOCAL_PROVIDER_URLS`); the proxy-fallback test and private webhook targets consult only `OMNIROUTE_ALLOW_PRIVATE_PROVIDER_URLS` and block local/LAN hosts while it is off. | @@ -241,7 +244,7 @@ Returns every flag with its effective value, source, and a summary. "requiresRestart": false, "warningLevel": "caution", }, - // ... all 100 flags + // ... all 103 flags ], "summary": { "total": 56, diff --git a/open-sse/config/providerErrorRules.ts b/open-sse/config/providerErrorRules.ts index e6c70433e801..d3d01f823e52 100644 --- a/open-sse/config/providerErrorRules.ts +++ b/open-sse/config/providerErrorRules.ts @@ -411,8 +411,70 @@ const EGRESS_BUCKETED_LOCK_PROVIDERS = egressIpLockProvidersFromEnv( process.env.OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS ); +/** + * Feature-flag reader injected by a module that is allowed to read the flag store + * (`open-sse/services/accountFallback.ts` registers `isFeatureFlagEnabled`). This file + * stays free of `src/` imports: it is reachable from client bundles and the + * open-sse → src boundary is ratcheted. Unregistered or throwing reader = flag off, so + * every flag below fails closed to the release default. + */ +let featureFlagReader: ((key: string) => boolean) | null = null; + +export function setProviderErrorRuleFlagReader(reader: ((key: string) => boolean) | null): void { + featureFlagReader = reader; +} + +function isRuleFlagEnabled(key: string): boolean { + if (!featureFlagReader) return false; + try { + return featureFlagReader(key) === true; + } catch { + return false; + } +} + +/** + * Opt-in (#16797): OpenCode Zen joins the default egress-IP lock family. Off by default + * because Zen is keyed (account-bucketed paid plans would see one 429 park every sibling + * connection sharing an egress IP). An explicit OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS value + * (a list or `none`) always wins over the flag. + */ +export const OPENCODE_ZEN_EGRESS_IP_LOCK_FLAG = "OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED"; +const OPENCODE_ZEN_PROVIDER_ID = "opencode-zen"; + +/** + * Pure resolver for the effective egress-lock set (exported for tests): the explicit env + * value wins; otherwise the default family, plus `opencode-zen` when the opt-in flag is on. + */ +export function resolveEgressIpLockProviders( + raw: string | undefined, + zenOptIn: boolean +): Set { + const set = egressIpLockProvidersFromEnv(raw); + if (zenOptIn && (raw === undefined || raw.trim() === "")) set.add(OPENCODE_ZEN_PROVIDER_ID); + return set; +} + +const EGRESS_LOCK_ENV_IS_EXPLICIT = Boolean(process.env.OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS?.trim()); +const EGRESS_BUCKETED_LOCK_PROVIDERS_WITH_ZEN = resolveEgressIpLockProviders( + process.env.OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS, + true +); + +/** The effective set: the flag is read per call so a dashboard override applies live. */ +function effectiveEgressBucketedLockProviders(): Set { + if (EGRESS_LOCK_ENV_IS_EXPLICIT) return EGRESS_BUCKETED_LOCK_PROVIDERS; + return isRuleFlagEnabled(OPENCODE_ZEN_EGRESS_IP_LOCK_FLAG) + ? EGRESS_BUCKETED_LOCK_PROVIDERS_WITH_ZEN + : EGRESS_BUCKETED_LOCK_PROVIDERS; +} + export function isEgressBucketedLockScope(provider: string | null | undefined): boolean { - return !!provider && EGRESS_BUCKETED_LOCK_PROVIDERS.has(provider.toLowerCase()); + if (!provider) return false; + const key = provider.toLowerCase(); + if (EGRESS_BUCKETED_LOCK_PROVIDERS.has(key)) return true; + // Only an opencode-zen lookup ever reads the flag store (hot path for everyone else). + return key === OPENCODE_ZEN_PROVIDER_ID && effectiveEgressBucketedLockProviders().has(key); } /** @@ -424,7 +486,7 @@ export function isEgressBucketedLockScope(provider: string | null | undefined): * applied. */ export function egressBucketedLockProviders(): string[] { - return [...EGRESS_BUCKETED_LOCK_PROVIDERS].sort(); + return [...effectiveEgressBucketedLockProviders()].sort(); } /** @@ -444,6 +506,31 @@ export function egressBucketedLockProviders(): string[] { */ const FULL_TEXT_RULE_PROVIDERS = new Set(["agentrouter"]); +/** + * Opt-in widenings of FULL_TEXT_RULE_PROVIDERS (#16797), one flag per provider family. + * The opencode and cloudflare-ai catalog rules above match on body-text markers + * ("organization_quota_exceeded", "monthly usage limit reached", "daily free + * allocation", ...) that the stripped structured error never carries, so today they + * cannot fire through checkFallbackError. Letting them fire changes the classification + * (rate_limit → quota_exhausted) and the cooldown/lock scope, hence default-off flags. + */ +export const OPENCODE_FULL_TEXT_ERROR_RULES_FLAG = "OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED"; +export const CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_FLAG = + "CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED"; +const OPT_IN_FULL_TEXT_RULE_PROVIDERS: ReadonlyArray<{ + flag: string; + providers: ReadonlySet; +}> = [ + { flag: OPENCODE_FULL_TEXT_ERROR_RULES_FLAG, providers: new Set(OPENCODE_RULE_FAMILY) }, + { flag: CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_FLAG, providers: new Set(["cloudflare-ai"]) }, +]; + +function usesFullTextRuleBody(key: string): boolean { + if (FULL_TEXT_RULE_PROVIDERS.has(key)) return true; + const optIn = OPT_IN_FULL_TEXT_RULE_PROVIDERS.find((entry) => entry.providers.has(key)); + return !!optIn && isRuleFlagEnabled(optIn.flag); +} + /** * True when an operator has declared at least one rule for this provider via * `settings.providerErrorRules` (injected through `setOperatorProviderErrorRules`). @@ -469,8 +556,7 @@ export function resolveRuleMatchBody( ): unknown { if ( provider && - (FULL_TEXT_RULE_PROVIDERS.has(provider.toLowerCase()) || - hasOperatorRuleForProvider(provider)) && + (usesFullTextRuleBody(provider.toLowerCase()) || hasOperatorRuleForProvider(provider)) && errorText ) { return errorText; diff --git a/open-sse/services/accountFallback.ts b/open-sse/services/accountFallback.ts index e55dba31f3c5..7d765032df12 100644 --- a/open-sse/services/accountFallback.ts +++ b/open-sse/services/accountFallback.ts @@ -20,6 +20,7 @@ import { getProviderErrorRuleMatch, resolveRuleMatchBody, honorsRuleLockScope, + setProviderErrorRuleFlagReader, } from "../config/providerErrorRules.ts"; import * as rot from "./rotationConfig.ts"; import { isContentPolicyRefusal } from "../utils/contentPolicyError.ts"; @@ -117,8 +118,16 @@ import { isCreditsExhaustedWithSharedWallet } from "./accountFallback/sharedWall import { CREDITS_EXHAUSTED_SIGNALS } from "./accountFallback/creditExhaustionSignals.ts"; export { CREDITS_EXHAUSTED_SIGNALS } from "./accountFallback/creditExhaustionSignals.ts"; import { isMistralAmbiguous401 } from "./accountFallback/mistralAmbiguousAuth.ts"; -import { isMistralAmbiguous401SoftLockoutEnabled } from "@/shared/utils/featureFlags"; +import { + isFeatureFlagEnabled, + isMistralAmbiguous401SoftLockoutEnabled, +} from "@/shared/utils/featureFlags"; import { boundedMap } from "@/lib/quota/boundedMap"; +// #16797: providerErrorRules.ts stays free of src/ imports; its opt-in flags +// (OpenCode Zen egress-IP lock, full-text rule bodies) read the flag store through +// this injected reader. Not registered = flags off (release default). +setProviderErrorRuleFlagReader(isFeatureFlagEnabled); + export type ProviderProfile = { baseCooldownMs: number; useUpstreamRetryHints: boolean; diff --git a/src/shared/constants/featureFlagDefinitions.ts b/src/shared/constants/featureFlagDefinitions.ts index 4784fa1431ce..cdfcc6fb95d1 100644 --- a/src/shared/constants/featureFlagDefinitions.ts +++ b/src/shared/constants/featureFlagDefinitions.ts @@ -372,6 +372,42 @@ export const FEATURE_FLAG_DEFINITIONS: FeatureFlagDefinition[] = [ requiresRestart: false, warningLevel: "caution", }, + { + key: "OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED", + label: "OpenCode Zen Egress-IP Lockout", + description: + "Add OpenCode Zen (opencode-zen) to the default egress-IP lockout family, so a 429 on one Zen connection also cools every sibling Zen connection that shares its egress IP. Off by default: Zen is keyed and each connection cools down on its own, exactly as before. An explicit OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS value always wins over this flag.", + descriptionI18nKey: "featureFlagOpencodeZenEgressIpLockEnabledDescription", + category: "network", + defaultValue: "false", + type: "boolean", + requiresRestart: false, + warningLevel: "caution", + }, + { + key: "OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED", + label: "OpenCode Full-Text Error Rules", + description: + 'Let the built-in OpenCode error rules (opencode-zen, opencode-go, opencode-cli) match on the full upstream error text, so body markers such as organization_quota_exceeded or "monthly usage limit reached" classify a 429 as quota exhausted with a connection-scoped lock. Off by default: those rules only see the structured error code and such a 429 stays a short rate limit, exactly as before.', + descriptionI18nKey: "featureFlagOpencodeFullTextErrorRulesEnabledDescription", + category: "network", + defaultValue: "false", + type: "boolean", + requiresRestart: false, + warningLevel: "caution", + }, + { + key: "CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED", + label: "Cloudflare Workers AI Full-Text Error Rules", + description: + 'Let the built-in Cloudflare Workers AI error rule match on the full upstream error text, so the 429 "daily free allocation" body is classified as quota exhausted instead of a short rate limit. Off by default: the rule only sees the structured error code, exactly as before.', + descriptionI18nKey: "featureFlagCloudflareAiFullTextErrorRulesEnabledDescription", + category: "network", + defaultValue: "false", + type: "boolean", + requiresRestart: false, + warningLevel: "caution", + }, { key: "MITM_DISABLE_TLS_VERIFY", label: "Disable TLS Verify (MITM)", diff --git a/tests/unit/egress-ip-lock-10880.test.ts b/tests/unit/egress-ip-lock-10880.test.ts index 6b93e8e95a82..9207c81fbc06 100644 --- a/tests/unit/egress-ip-lock-10880.test.ts +++ b/tests/unit/egress-ip-lock-10880.test.ts @@ -217,6 +217,56 @@ test("three connections sharing one egress IP: ONE upstream call for the whole p assert.equal(c!.testStatus, "unavailable"); }); +// #16797: opencode-zen is keyed, so it joins the egress family only behind the +// default-off OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED flag. Flag off = the tip's behavior +// (every Zen connection is tried and cooled on its own); flag on = one call per pool. +async function rotateZenPool(): Promise<{ upstreamCalls: number; siblingCooled: boolean }> { + await resetStorage(); + const connA = await seedConnection("opencode-zen"); + const connB = await seedConnection("opencode-zen"); + seedProxyLog(connA, SHARED_EGRESS_IP, "opencode-zen"); + seedProxyLog(connB, SHARED_EGRESS_IP, "opencode-zen"); + // RATE_LIMIT_TEXT_429 carries no quota text, so a lone connection keeps a short + // scaled cooldown and the sibling's state is decided only by the egress branch. + const first = await auth.markAccountUnavailable( + connA, + 429, + '{"error":{"message":"rate limit reached"}}', + "opencode-zen", + "model-x", + null, + {} + ); + assert.equal(first.shouldFallback, true); + const b = await providersDb.getProviderConnectionById(connB); + const siblingCooled = + !!b?.rateLimitedUntil && new Date(String(b.rateLimitedUntil)).getTime() > Date.now(); + const { upstreamCalls } = await rotateWholePool( + [connB], + '{"error":{"message":"rate limit reached"}}', + "opencode-zen" + ); + return { upstreamCalls: upstreamCalls + 1, siblingCooled }; +} + +test("#16797 flag off (default): an opencode-zen 429 leaves its egress siblings alone", async () => { + delete process.env.OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED; + const { upstreamCalls, siblingCooled } = await rotateZenPool(); + assert.equal(siblingCooled, false, "zen sibling must not be pre-cooled by default"); + assert.equal(upstreamCalls, 2, "each zen connection is still tried on its own"); +}); + +test("#16797 flag on: an opencode-zen 429 cools the connections sharing its egress IP", async () => { + process.env.OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED = "true"; + try { + const { upstreamCalls, siblingCooled } = await rotateZenPool(); + assert.equal(siblingCooled, true, "zen sibling cooled before being tried"); + assert.equal(upstreamCalls, 1, "exactly one upstream call for the zen pool"); + } finally { + delete process.env.OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED; + } +}); + test("counter-test: the SAME rotation burns N calls for a non-allowlisted provider", async () => { await resetStorage(); const connA = await seedConnection("ollama-cloud"); diff --git a/tests/unit/feature-flags-settings.test.ts b/tests/unit/feature-flags-settings.test.ts index a4eb009e3b4c..a1a19cc9203e 100644 --- a/tests/unit/feature-flags-settings.test.ts +++ b/tests/unit/feature-flags-settings.test.ts @@ -55,11 +55,12 @@ const { // opt-in flag to count unpriced usage as $0 in USD quotas, default off) to 83; // REASONING_REPLAY_ENABLED (#12486, default on) to 84; COMBO_AUTO_PRUNE_STALE_STEPS // (#13505, default off) to 85; KIRO_TOOL_CALL_REPAIR_ENABLED (one-shot Kiro +// tool_call wrapper repair, default on) to 86. +const EXPECTED_FEATURE_FLAG_COUNT = 103; // tool_call wrapper repair, default on) to 86; CLAUDE_OAUTH_SUMMARIZED_THINKING_UNREDACTED // (#17003, lets Claude OAuth drop the redact-thinking beta for summarized thinking, // default off) to 87. // MODEL_ENDPOINT_CAPABILITY_GATE_ENABLED (#16929/#16956, default off) takes it to 88. -const EXPECTED_FEATURE_FLAG_COUNT = 100; // ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM (#16236, default off) takes it to 88. // MODELS_DEV_EFFORT_TIERS_ENABLED (#15944, default off) takes it to 88. // default off) to 87; AUTO_REQUEST_RESOLVED_CAPABILITIES (#16419, resolved-capability diff --git a/tests/unit/provider-error-rules-opt-in-flags-16797.test.ts b/tests/unit/provider-error-rules-opt-in-flags-16797.test.ts new file mode 100644 index 000000000000..65b14b9bda4b --- /dev/null +++ b/tests/unit/provider-error-rules-opt-in-flags-16797.test.ts @@ -0,0 +1,145 @@ +// #16797 — opt-in widenings of the provider error-rule allowlists, each behind its own +// default-off feature flag. Exercises the real providerErrorRules.ts predicates and the +// real checkFallbackError (accountFallback.ts registers the flag reader at load), with +// the flag off (release behavior, unchanged) and on. +import test from "node:test"; +import assert from "node:assert/strict"; + +delete process.env.OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS; +const FLAGS = [ + "OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED", + "OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED", + "CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED", +]; +for (const flag of FLAGS) delete process.env[flag]; + +const rules = await import("../../open-sse/config/providerErrorRules.ts"); +const { checkFallbackError } = await import("../../open-sse/services/accountFallback.ts"); +const core = await import("../../src/lib/db/core.ts"); +const { isFeatureFlagEnabled } = await import("../../src/shared/utils/featureFlags.ts"); + +test.after(() => { + for (const flag of FLAGS) delete process.env[flag]; + core.resetDbInstance(); +}); + +function withFlag(flag: string, fn: () => T): T { + process.env[flag] = "true"; + try { + return fn(); + } finally { + delete process.env[flag]; + } +} + +test("egress family: opencode-zen is out by default and in with the flag", () => { + assert.equal(rules.isEgressBucketedLockScope("opencode-zen"), false); + assert.deepEqual(rules.egressBucketedLockProviders(), ["opencode-cli", "opencode-go"]); + withFlag("OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED", () => { + assert.equal(rules.isEgressBucketedLockScope("opencode-zen"), true); + assert.equal(rules.isEgressBucketedLockScope("OpenCode-Zen"), true, "case-insensitive"); + assert.deepEqual(rules.egressBucketedLockProviders(), [ + "opencode-cli", + "opencode-go", + "opencode-zen", + ]); + }); + assert.equal(rules.isEgressBucketedLockScope("opencode-zen"), false, "read per call"); +}); + +test("egress family: an explicit OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS value wins over the flag", () => { + assert.deepEqual([...rules.resolveEgressIpLockProviders(undefined, false)].sort(), [ + "opencode-cli", + "opencode-go", + ]); + assert.deepEqual([...rules.resolveEgressIpLockProviders(" ", true)].sort(), [ + "opencode-cli", + "opencode-go", + "opencode-zen", + ]); + assert.equal(rules.resolveEgressIpLockProviders("none", true).size, 0); + assert.deepEqual([...rules.resolveEgressIpLockProviders("opencode-go", true)], ["opencode-go"]); +}); + +test("a throwing or missing flag reader fails closed to the release default", () => { + try { + rules.setProviderErrorRuleFlagReader(() => { + throw new Error("flag store unavailable"); + }); + process.env.OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED = "true"; + assert.equal(rules.isEgressBucketedLockScope("opencode-zen"), false); + rules.setProviderErrorRuleFlagReader(null); + assert.equal(rules.isEgressBucketedLockScope("opencode-zen"), false); + } finally { + delete process.env.OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED; + rules.setProviderErrorRuleFlagReader(isFeatureFlagEnabled); + } +}); + +const STRUCTURED = { code: "rate_limited", type: "requests" }; +const OPENCODE_QUOTA_BODY = '{"error":{"message":"organization_quota_exceeded"}}'; +const CLOUDFLARE_DAILY_BODY = + "you have used up your daily free allocation of 10,000 neurons, please upgrade"; + +test("full-text bodies: the opencode family keeps the structured body unless opted in", () => { + for (const provider of ["opencode-zen", "opencode-go", "opencode-cli"]) { + assert.equal(rules.resolveRuleMatchBody(provider, STRUCTURED, OPENCODE_QUOTA_BODY), STRUCTURED); + withFlag("OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED", () => { + assert.equal( + rules.resolveRuleMatchBody(provider, STRUCTURED, OPENCODE_QUOTA_BODY), + OPENCODE_QUOTA_BODY, + provider + ); + // The opencode flag never widens another provider. + assert.equal( + rules.resolveRuleMatchBody("cloudflare-ai", STRUCTURED, CLOUDFLARE_DAILY_BODY), + STRUCTURED + ); + }); + } + assert.equal( + rules.resolveRuleMatchBody("agentrouter", STRUCTURED, "用户额度不足"), + "用户额度不足" + ); +}); + +test("full-text bodies: cloudflare-ai keeps the structured body unless opted in", () => { + assert.equal( + rules.resolveRuleMatchBody("cloudflare-ai", STRUCTURED, CLOUDFLARE_DAILY_BODY), + STRUCTURED + ); + withFlag("CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED", () => { + assert.equal( + rules.resolveRuleMatchBody("cloudflare-ai", STRUCTURED, CLOUDFLARE_DAILY_BODY), + CLOUDFLARE_DAILY_BODY + ); + assert.equal( + rules.resolveRuleMatchBody("opencode-zen", STRUCTURED, OPENCODE_QUOTA_BODY), + STRUCTURED + ); + }); +}); + +test("checkFallbackError: an opencode-zen quota body stays a rate limit by default", () => { + const off = checkFallbackError(429, OPENCODE_QUOTA_BODY, 0, null, "opencode-zen", null); + assert.equal(off.shouldFallback, true); + assert.equal(off.reason, "rate_limit_exceeded"); + assert.equal(off.ruleScope, undefined); + const on = withFlag("OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED", () => + checkFallbackError(429, OPENCODE_QUOTA_BODY, 0, null, "opencode-zen", null) + ); + assert.equal(on.shouldFallback, true); + assert.equal(on.reason, "quota_exhausted"); + assert.equal(on.ruleScope, "connection"); +}); + +test("checkFallbackError: the cloudflare-ai daily allocation body is a quota only when opted in", () => { + const off = checkFallbackError(429, CLOUDFLARE_DAILY_BODY, 0, null, "cloudflare-ai", null); + assert.equal(off.shouldFallback, true); + assert.notEqual(off.reason, "quota_exhausted"); + const on = withFlag("CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED", () => + checkFallbackError(429, CLOUDFLARE_DAILY_BODY, 0, null, "cloudflare-ai", null) + ); + assert.equal(on.shouldFallback, true); + assert.equal(on.reason, "quota_exhausted"); +}); diff --git a/tests/unit/proxy-outcome-memory.test.ts b/tests/unit/proxy-outcome-memory.test.ts index 39018b7ee556..0103cf87f59f 100644 --- a/tests/unit/proxy-outcome-memory.test.ts +++ b/tests/unit/proxy-outcome-memory.test.ts @@ -8,7 +8,8 @@ import assert from "node:assert/strict"; const memory = await import("../../open-sse/utils/proxyRefusalMemory.ts"); const { noteProxyOutcome } = await import("../../src/sse/handlers/proxyOutcomeMemory.ts"); -const { egressBucketedLockProviders } = await import("../../open-sse/config/providerErrorRules.ts"); +const { egressBucketedLockProviders, setProviderErrorRuleFlagReader } = + await import("../../open-sse/config/providerErrorRules.ts"); const PROXY = { type: "http", host: "10.4.0.1", port: 8080 }; const KEY = memory.proxyEgressKey(PROXY); @@ -37,6 +38,21 @@ test("a refusal from a provider outside the scope writes nothing", () => { assert.equal(memory.__proxyRefusalMemorySizeForTesting(), 0); }); +test("#16797: with OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED on, an opencode-zen refusal sets the member aside", () => { + // The flag store is injected by accountFallback.ts at runtime; this unit wires a reader + // that reports only this flag on, so no database is opened here. + setProviderErrorRuleFlagReader((key) => key === "OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED"); + try { + noteProxyOutcome("opencode-zen", { proxy: PROXY, upstreamStatus: 429 }); + assert.equal(memory.isProxyAvoided(KEY), true); + } finally { + setProviderErrorRuleFlagReader(null); + } + memory.__resetProxyRefusalMemoryForTesting(); + noteProxyOutcome("opencode-zen", { proxy: PROXY, upstreamStatus: 429 }); + assert.equal(memory.__proxyRefusalMemorySizeForTesting(), 0, "flag off again: nothing written"); +}); + test("no received status, no proxy or no proxyInfo writes nothing", () => { noteProxyOutcome("opencode-go", { proxy: PROXY }); noteProxyOutcome("opencode-go", { proxy: PROXY, upstreamStatus: null }); diff --git a/tests/unit/server-owned-tool-loop-flag.test.ts b/tests/unit/server-owned-tool-loop-flag.test.ts index 8bffd8e0730f..0695966ee1dd 100644 --- a/tests/unit/server-owned-tool-loop-flag.test.ts +++ b/tests/unit/server-owned-tool-loop-flag.test.ts @@ -76,9 +76,12 @@ describe("feature-flags-settings count update", () => { // default off) to 82; USAGE_LIMIT_IGNORE_UNPRICED (#14799, default off) to 83; // REASONING_REPLAY_ENABLED (#12486, default on) to 84; COMBO_AUTO_PRUNE_STALE_STEPS // (#13505, default off) to 85; KIRO_TOOL_CALL_REPAIR_ENABLED (default on) to 86; + // OPENCODE_ZEN_EGRESS_IP_LOCK_ENABLED (#16797, default off) to 87; + // OPENCODE_FULL_TEXT_ERROR_RULES_ENABLED + CLOUDFLARE_AI_FULL_TEXT_ERROR_RULES_ENABLED + // (#16797, default off) to 89. + assert.equal(FEATURE_FLAG_DEFINITIONS.length, 103); // CLAUDE_OAUTH_SUMMARIZED_THINKING_UNREDACTED (#17003, default off) to 87; // MODEL_ENDPOINT_CAPABILITY_GATE_ENABLED (#16929/#16956, default off) to 88. - assert.equal(FEATURE_FLAG_DEFINITIONS.length, 100); // ANTHROPIC_COMPATIBLE_MID_CONVERSATION_SYSTEM (#16236, default off) to 88. // MODELS_DEV_EFFORT_TIERS_ENABLED (#15944, default off) to 88. // CREDENTIAL_HEALTH_RETEST_AUTO_DISABLED (#16190, default off) to 88.