feat(extension-pgvector): support variable length embeddings

docs/architecture docs/adrs/ADR 212 - AST-bound codec resolution.md

@@ -72,7 +72,7 @@ By the time anything reaches `ops.json`, all codec resolution has already happen
 
 ### Honest descriptor signatures
 
-With the triangulation removed, descriptors no longer need to accept `undefined` params as a proxy for "no params known". `PgVectorDescriptor.factory(params: VectorParams)` reads `params.length` directly; the defensive `(params as VectorParams | undefined)?.length` cast is deleted. `PgVectorCodec.length` narrows from `number | undefined` to `number`. The undimensioned `vectorColumn` helper (which relied on the representative-codec hack) is retired; users use `vector(N)` exclusively.
+With the triangulation removed, descriptors no longer need to accept `undefined` params as a proxy for "no params known". `PgVectorDescriptor.factory(params: VectorParams)` reads `params.length` directly; the defensive `(params as VectorParams | undefined)?.length` cast is deleted. `VectorParams` permits an optional `length`, so `PgVectorCodec.length` is `number | undefined`. The public `vector()` and `pgVectorColumn()` helpers explicitly supply `{}` for variable dimensions; `vector(N)` and `pgVectorColumn(N)` supply `{ length: N }` for a fixed dimension. An empty parameter object is a validated choice, not a proxy for unknown codec identity. All four conversion paths (`encode`, `decode`, `encodeJson`, and `decodeJson`) require 1–16,000 finite numbers for undimensioned vectors and exactly N finite numbers for dimensioned vectors.
 
 ## Consequences
 
@@ -88,7 +88,7 @@ With the triangulation removed, descriptors no longer need to accept `undefined`
 
 - **Builder construction sites must stamp `codec`.** Every column-bound `ParamRef` and `ProjectionItem` construction site calls `codecRefForColumn` at build time. This is a one-time cost per node (previously the builder was already calling `forColumn` to derive `refs`), but the call is now explicit and required rather than optional.
 - **Refs-less raw SQL paths require explicit codec.** `sql.value(42)` without an explicit codec argument fails at build time. Default-codec ergonomics (e.g. `defaultCodecForJsType` on the family adapter) are out of scope; tracked separately.
-- **Breaking: `vectorColumn` retired.** The undimensioned vector column helper is removed. Users migrate to `vector(N)`. This is acceptable because the undimensioned form only worked due to the representative-codec hack, and any future parameterized codec whose wire format depends on its params would have silently produced malformed output.
+- **Variable dimensions are explicit.** `vector()` emits the PostgreSQL type `vector` and the TypeScript type `Vector`; `vector(N)` emits `vector(N)` and `Vector<N>`. Their validated parameter objects distinguish them in the content-keyed resolver without a representative-codec fallback. Variable dimensions allow rows with different vector lengths, but distance operations still require compatible dimensions.
 
 ## References
 

packages/3-extensions/pgvector/README.md

@@ -84,7 +84,7 @@ export const contract = defineContract({
 });
 ```
 
-The `vector(N)` factory is registered through the unified `CodecDescriptor<{ length: number }>` shape — `paramsSchema` validates the dimension at the contract boundary and the curried `factory` materializes the runtime codec at context construction. The column's TypeScript type, `Vector<N>`, comes from the `ParameterizedOutputTypesForTypeParams` type function the `codec-types` entry exports; both `defineContract` and the emitted `contract.d.ts` apply it. See [ADR 208 — Higher-order codecs for parameterized types](../../../docs/architecture%20docs/adrs/ADR%20208%20-%20Higher-order%20codecs%20for%20parameterized%20types.md) for the descriptor model. Every pgvector column must declare an explicit dimension via `vector(N)`; the runtime codec is constructed against `{ length: N }`, so an undimensioned form has no honest descriptor signature.
+The `vector(N?)` factory is registered through the unified `CodecDescriptor<{ length?: number }>` shape — `paramsSchema` validates the dimension at the contract boundary when specified, and the curried `factory` materializes the runtime codec at context construction. The column's TypeScript type, `Vector` or `Vector<N>`, comes from the `ParameterizedOutputTypesForTypeParams` type function the `codec-types` entry exports; both `defineContract` and the emitted `contract.d.ts` apply it. See [ADR 208 — Higher-order codecs for parameterized types](../../../docs/architecture%20docs/adrs/ADR%20208%20-%20Higher-order%20codecs%20for%20parameterized%20types.md) for the descriptor model. Any pgvector column can declare an explicit dimension via `vector(N)` for runtime validation and `Vector<N>` typing, or use `vector()` to support variable dimensions. All codec paths enforce 1–16,000 finite numbers for `vector()` and exactly N finite numbers for `vector(N)`. Postgres requires compatible dimensions for distance operations. [ADR 212 — AST-bound codec resolution](../../../docs/architecture%20docs/adrs/ADR%20212%20-%20AST-bound%20codec%20resolution.md#honest-descriptor-signatures) describes how both forms retain explicit codec identity.
 
 ### Runtime Setup
 

packages/3-extensions/pgvector/src/core/authoring.ts

@@ -5,7 +5,7 @@ export const pgvectorAuthoringTypes = {
     Vector: {
       kind: 'typeConstructor',
       inferred: true,
-      args: [{ kind: 'value', dataType: 'pg/int4', name: 'length' }],
+      args: [{ kind: 'value', dataType: 'pg/int4', name: 'length', optional: true }],
       output: {
         codecId: 'pg/vector@1',
         typeParams: {

packages/3-extensions/pgvector/src/core/codecs.ts

@@ -4,10 +4,10 @@
  * Mirrors the patterns in `postgres/codecs-class.ts` and `sqlite/codecs-class.ts` for the single `pg/vector@1` codec. Three artifacts:
  *
  * 1. `PgVectorCodec` extends {@link CodecImpl} with the wire and value conversions inline. Conversions are simple enough (PostgreSQL `[1,2,3]` text format) that no shared helper module is warranted; the class body is the source of truth.
- * 2. `PgVectorDescriptor` extends {@link PostgresCodecDescriptor} with the codec id, traits, the `pgvector/vector` data type and its params schema, and explicit target behavior. The column's type, `Vector<${length}>`, comes from the pack's type function in `types/codec-types.ts`. The data type declares the type's name and the bounds of `length`.
- * 3. `pgVectorColumn(length)` per-codec column helper invoking `descriptor.factory({ length })` directly.
+ * 2. `PgVectorDescriptor` extends {@link PostgresCodecDescriptor} with the codec id, traits, the `pgvector/vector` data type and its params schema, and explicit target behavior. The column's type, `Vector` or `Vector<${length}>`, comes from the pack's type function in `types/codec-types.ts`. The data type declares the type's name and the bounds of `length`.
+ * 3. `pgVectorColumn(length?)` per-codec column helper invoking `descriptor.factory({ length })` directly.
  *
- * `length` threads into the runtime codec via the constructor so the wire methods enforce the declared dimension, and `pgvector/vector` checks it for every stored value. Without this, `vector(3)` and `vector(1536)` would produce codecs with identical behaviour and a dimension-mismatched value would round-trip undetected.
+ * When provided, `length` threads into the runtime codec via the constructor so the wire methods enforce the declared dimension, and `pgvector/vector` checks it for every stored value. Without this, `vector(3)` and `vector(1536)` would produce codecs with identical behaviour and a dimension-mismatched value would round-trip undetected.
  */
 
 import {
@@ -26,14 +26,15 @@ import {
   definePostgresCodecs,
   PostgresCodecDescriptor,
 } from '@internal/target-postgres/codec-descriptor';
+import { ifDefined } from '@internal/utils/defined';
 import type { StandardSchemaV1 } from '@standard-schema/spec';
-import { VECTOR_CODEC_ID } from './constants';
+import { VECTOR_CODEC_ID, VECTOR_MAX_DIM } from './constants';
 import { pgvectorVector, pgvectorVectorParams } from './data-types';
 import { pgVectorError } from './errors';
 
 type VectorConversionCode = 'RUNTIME.ENCODE_FAILED' | 'RUNTIME.DECODE_FAILED';
 
-type VectorParams = { readonly length: number };
+type VectorParams = { readonly length?: number };
 
 function parseVector(value: string): number[] {
   if (!value.startsWith('[') || !value.endsWith(']')) {
@@ -71,13 +72,15 @@ export class PgVectorCodec extends CodecImpl<
   string,
   number[]
 > {
-  readonly length: number;
+  readonly length: number | undefined;
 
-  constructor(descriptor: AnyCodecDescriptor, length: number) {
-    super(descriptor, pgvectorVector, { length });
+  /** Creates a codec with an optional exact dimension; undefined permits variable dimensions. */
+  constructor(descriptor: AnyCodecDescriptor, length: number | undefined) {
+    super(descriptor, pgvectorVector, ifDefined('length', length));
     this.length = length;
   }
 
+  /** Rejects non-finite elements and lengths outside the declared dimension or supported bounds. */
   assertVector(value: unknown, code: VectorConversionCode): asserts value is number[] {
     const meta = { codecId: VECTOR_CODEC_ID, expectedLength: this.length };
     if (!Array.isArray(value)) {
@@ -91,7 +94,14 @@ export class PgVectorCodec extends CodecImpl<
         throw pgVectorError(code, 'Vector value must contain only finite numbers', { meta });
       }
     }
-    if (value.length !== this.length) {
+    if (this.length === undefined && !this.acceptsLength(value.length)) {
+      throw pgVectorError(
+        code,
+        `Vector length must be between 1 and ${VECTOR_MAX_DIM}, got ${value.length}`,
+        { meta: { ...meta, receivedLength: value.length } },
+      );
+    }
+    if (this.length !== undefined && value.length !== this.length) {
       throw pgVectorError(
         code,
         `Vector length mismatch: expected ${this.length}, got ${value.length}`,
@@ -127,6 +137,13 @@ export class PgVectorCodec extends CodecImpl<
     this.assertVector(input, 'RUNTIME.ENCODE_FAILED');
     return this.dataTypeValueOf([...input]);
   }
+
+  /** Checks the exact declared dimension, or the supported 1–16,000 range when none is declared. */
+  private acceptsLength(length: number): boolean {
+    return this.length === undefined
+      ? length >= 1 && length <= VECTOR_MAX_DIM
+      : length === this.length;
+  }
 }
 
 /**
@@ -156,11 +173,30 @@ export class PgVectorDescriptor extends PostgresCodecDescriptor<VectorParams> {
 
 export const pgVectorDescriptor = new PgVectorDescriptor();
 
-/**
- * Per-codec column helper for `pg/vector@1`. Generic over `N extends number` so the column site preserves the dimension literal in `typeParams` (e.g. `pgVectorColumn(1536)` packs `typeParams: { length: 1536 }`).
- */
-export const pgVectorColumn = <N extends number>(length: N) =>
-  column(pgVectorDescriptor.factory({ length }), pgVectorDescriptor.codecId, { length });
+/** Creates a vector column with variable dimensions and empty type parameters. */
+export function pgVectorColumn(): ReturnType<typeof variableVectorColumn>;
+/** Creates a vector column that preserves its exact dimension as a type parameter. */
+export function pgVectorColumn<N extends number>(
+  length: N,
+): ReturnType<typeof fixedVectorColumn<N>>;
+/** Creates a vector column whose dimension may be absent, binding variable dimensions when it is. */
+export function pgVectorColumn(
+  length?: number,
+): ReturnType<typeof variableVectorColumn> | ReturnType<typeof fixedVectorColumn<number>>;
+/** Selects a variable or fixed dimension codec factory from the optional length. */
+export function pgVectorColumn(length?: number) {
+  return length === undefined ? variableVectorColumn() : fixedVectorColumn(length);
+}
+
+/** Binds an undimensioned codec using an explicit empty parameter object. */
+function variableVectorColumn() {
+  return column(pgVectorDescriptor.factory({}), pgVectorDescriptor.codecId, {});
+}
+
+/** Binds a dimensioned codec while preserving the dimension literal in its type parameters. */
+function fixedVectorColumn<N extends number>(length: N) {
+  return column(pgVectorDescriptor.factory({ length }), pgVectorDescriptor.codecId, { length });
+}
 
 pgVectorColumn satisfies ColumnHelperFor<PgVectorDescriptor>;
 pgVectorColumn satisfies ColumnHelperForStrict<PgVectorDescriptor>;

packages/3-extensions/pgvector/src/core/data-types.ts

@@ -39,10 +39,10 @@ function elementNumber(element: JsonValue): number {
 }
 
 export const pgvectorVectorParams = arktype({
-  length: `number.integer >= 1 & number.integer <= ${VECTOR_MAX_DIM}` as const,
+  'length?': `number.integer >= 1 & number.integer <= ${VECTOR_MAX_DIM}` as const,
 });
 
-/** A `vector(n)` value is an array of exactly `n` finite numbers, each one a `real` holds exactly. */
+/** A `vector(n)` value is an array of exactly `n` finite numbers, and a `vector` value one of 1 to 16,000, each one a `real` holds exactly. */
 function isFiniteNumber(element: JsonValue): element is number {
   return typeof element === 'number' && Number.isFinite(element);
 }
@@ -51,9 +51,11 @@ const readVector: DataTypeReader = (json, params) => {
   const length = typeof params['length'] === 'number' ? params['length'] : undefined;
   const expected =
     length === undefined
-      ? 'an array of finite numbers'
+      ? `an array of 1 to ${VECTOR_MAX_DIM} finite numbers`
       : `an array of ${counted(length, 'finite number')}`;
-  if (!Array.isArray(json) || (length !== undefined && json.length !== length)) {
+  const acceptsLength = (received: number) =>
+    length === undefined ? received >= 1 && received <= VECTOR_MAX_DIM : received === length;
+  if (!Array.isArray(json) || !acceptsLength(json.length)) {
     return refuseJsonValue('pgvector/vector', expected, json);
   }
   if (!json.every(isFiniteNumber)) return refuseJsonValue('pgvector/vector', expected, json);
@@ -88,7 +90,10 @@ function realText(value: number): string {
 export const pgvectorVector = sqlDataType('pgvector/vector', {
   params: pgvectorVectorParams,
   read: readVector,
-  texts: [{ text: 'vector({length})', written: true, catalog: true }],
+  texts: [
+    { text: 'vector', written: true, catalog: true },
+    { text: 'vector({length})', written: true, catalog: true },
+  ],
   listCast: {
     of: [pgInt2.id, pgInt4.id, pgInt8.id, pgNumeric.id],
     cast: (elements) => elements.map(elementNumber),

packages/3-extensions/pgvector/src/exports/column-types.ts

@@ -1,10 +1,23 @@
 /**
- * Column type descriptor factory for pgvector extension. `vector(N)` is the canonical authoring surface; every pgvector column must declare a dimension via this factory.
+ * Column type descriptor factory for pgvector extension. `vector(N?)` is the canonical authoring surface; each pgvector column may optionally declare a dimension via this factory. When provided, the dimension threads into the runtime codec through `paramsSchema.length` and into the DDL via the data type declaration (e.g. `vector(1536)`).
  */
 
 import type { ColumnTypeDescriptor } from '@internal/framework-components/codec';
 import { VECTOR_CODEC_ID } from '../core/constants';
 
+/**
+ * Factory for creating non-dimensioned vector column descriptors.
+ *
+ * @example
+ * ```typescript
+ * .column('embedding', { type: vector(), nullable: false })
+ * // Produces: typeParams: {}
+ * ```
+ * @returns A column type descriptor without `typeParams.length` set
+ */
+export function vector(): ColumnTypeDescriptor<typeof VECTOR_CODEC_ID> & {
+  readonly typeParams: Record<string, never>;
+};
 /**
  * Factory for creating dimensioned vector column descriptors.
  *
@@ -18,7 +31,17 @@ import { VECTOR_CODEC_ID } from '../core/constants';
  */
 export function vector<N extends number>(
   length: N,
-): ColumnTypeDescriptor<typeof VECTOR_CODEC_ID> & { readonly typeParams: { readonly length: N } } {
+): ColumnTypeDescriptor<typeof VECTOR_CODEC_ID> & { readonly typeParams: { readonly length: N } };
+/** Creates a vector descriptor whose dimension may be absent, leaving variable dimensions when it is. */
+export function vector(length?: number): ColumnTypeDescriptor<typeof VECTOR_CODEC_ID> & {
+  readonly typeParams: Record<string, never> | { readonly length: number };
+};
+
+/** Creates a vector descriptor, omitting the dimension parameter when no length is supplied. */
+export function vector(length?: number): ColumnTypeDescriptor<typeof VECTOR_CODEC_ID> {
+  if (length === undefined) {
+    return { codecId: VECTOR_CODEC_ID, typeParams: {} };
+  }
   return {
     codecId: VECTOR_CODEC_ID,
     typeParams: { length },

packages/3-extensions/pgvector/src/types/codec-types.ts

@@ -23,11 +23,11 @@ export type CodecTypes = CoreCodecTypes;
 type OutputTypeMap<TypeParams> = {
   readonly 'pg/vector@1': TypeParams extends { readonly length: infer N extends number }
     ? Vector<N>
-    : never;
+    : Vector;
 };
 
 /**
- * The output type of a `vector(N)` column, `Vector<N>`: the only declaration of it, which a contract written with `defineContract` and the `contract.d.ts` that `prisma contract emit` writes both apply.
+ * The output type of a `vector()` or `vector(N)` column, `Vector` or `Vector<N>`: the only declaration of it, which a contract written with `defineContract` and the `contract.d.ts` that `prisma contract emit` writes both apply.
  */
 export interface ParameterizedOutputTypesForTypeParams extends ParameterizedOutputTypesFunction {
   readonly outputTypes: OutputTypeMap<this['typeParams']>;

packages/3-extensions/pgvector/test/codec-conformance.integration.test.ts

@@ -62,6 +62,17 @@ function vectorCase(
 }
 
 const cases: readonly PostgresCodecConformanceCase[] = [
+  ...[[1], [1, 2, 3]].map(
+    (value): PostgresCodecConformanceCase => ({
+      codecId: 'pg/vector@1',
+      descriptor: pgVectorDescriptor,
+      dataType: pgvectorVector,
+      label: `variable vector with ${value.length} dimensions`,
+      value,
+      typeParams: {},
+      setupSql: INSTALL_VECTOR,
+    }),
+  ),
   vectorCase('three dimensions', [1, 2, 3]),
   // A vector's text form separates elements with commas and wraps them in
   // brackets, so a value has to carry negatives and fractions before the