diff --git a/docs/architecture docs/adrs/ADR 212 - AST-bound codec resolution.md b/docs/architecture docs/adrs/ADR 212 - AST-bound codec resolution.md index 461a617ae94a..9b7c708884df 100644 --- a/docs/architecture docs/adrs/ADR 212 - AST-bound codec resolution.md +++ b/docs/architecture docs/adrs/ADR 212 - AST-bound codec resolution.md @@ -72,7 +72,7 @@ By the time anything reaches `ops.json`, all codec resolution has already happen ### Honest descriptor signatures -With the triangulation removed, descriptors no longer need to accept `undefined` params as a proxy for "no params known". `PgVectorDescriptor.factory(params: VectorParams)` reads `params.length` directly; the defensive `(params as VectorParams | undefined)?.length` cast is deleted. `PgVectorCodec.length` narrows from `number | undefined` to `number`. The undimensioned `vectorColumn` helper (which relied on the representative-codec hack) is retired; users use `vector(N)` exclusively. +With the triangulation removed, descriptors no longer need to accept `undefined` params as a proxy for "no params known". `PgVectorDescriptor.factory(params: VectorParams)` reads `params.length` directly; the defensive `(params as VectorParams | undefined)?.length` cast is deleted. `VectorParams` permits an optional `length`, so `PgVectorCodec.length` is `number | undefined`. The public `vector()` and `pgVectorColumn()` helpers explicitly supply `{}` for variable dimensions; `vector(N)` and `pgVectorColumn(N)` supply `{ length: N }` for a fixed dimension. An empty parameter object is a validated choice, not a proxy for unknown codec identity. All four conversion paths (`encode`, `decode`, `encodeJson`, and `decodeJson`) require 1–16,000 finite numbers for undimensioned vectors and exactly N finite numbers for dimensioned vectors. ## Consequences @@ -88,7 +88,7 @@ With the triangulation removed, descriptors no longer need to accept `undefined` - **Builder construction sites must stamp `codec`.** Every column-bound `ParamRef` and `ProjectionItem` construction site calls `codecRefForColumn` at build time. This is a one-time cost per node (previously the builder was already calling `forColumn` to derive `refs`), but the call is now explicit and required rather than optional. - **Refs-less raw SQL paths require explicit codec.** `sql.value(42)` without an explicit codec argument fails at build time. Default-codec ergonomics (e.g. `defaultCodecForJsType` on the family adapter) are out of scope; tracked separately. -- **Breaking: `vectorColumn` retired.** The undimensioned vector column helper is removed. Users migrate to `vector(N)`. This is acceptable because the undimensioned form only worked due to the representative-codec hack, and any future parameterized codec whose wire format depends on its params would have silently produced malformed output. +- **Variable dimensions are explicit.** `vector()` emits the PostgreSQL type `vector` and the TypeScript type `Vector`; `vector(N)` emits `vector(N)` and `Vector`. Their validated parameter objects distinguish them in the content-keyed resolver without a representative-codec fallback. Variable dimensions allow rows with different vector lengths, but distance operations still require compatible dimensions. ## References diff --git a/packages/3-extensions/pgvector/README.md b/packages/3-extensions/pgvector/README.md index 41d36ab2cc2e..3d6522a9912f 100644 --- a/packages/3-extensions/pgvector/README.md +++ b/packages/3-extensions/pgvector/README.md @@ -84,7 +84,7 @@ export const contract = defineContract({ }); ``` -The `vector(N)` factory is registered through the unified `CodecDescriptor<{ length: number }>` shape — `paramsSchema` validates the dimension at the contract boundary and the curried `factory` materializes the runtime codec at context construction. The column's TypeScript type, `Vector`, comes from the `ParameterizedOutputTypesForTypeParams` type function the `codec-types` entry exports; both `defineContract` and the emitted `contract.d.ts` apply it. See [ADR 208 — Higher-order codecs for parameterized types](../../../docs/architecture%20docs/adrs/ADR%20208%20-%20Higher-order%20codecs%20for%20parameterized%20types.md) for the descriptor model. Every pgvector column must declare an explicit dimension via `vector(N)`; the runtime codec is constructed against `{ length: N }`, so an undimensioned form has no honest descriptor signature. +The `vector(N?)` factory is registered through the unified `CodecDescriptor<{ length?: number }>` shape — `paramsSchema` validates the dimension at the contract boundary when specified, and the curried `factory` materializes the runtime codec at context construction. The column's TypeScript type, `Vector` or `Vector`, comes from the `ParameterizedOutputTypesForTypeParams` type function the `codec-types` entry exports; both `defineContract` and the emitted `contract.d.ts` apply it. See [ADR 208 — Higher-order codecs for parameterized types](../../../docs/architecture%20docs/adrs/ADR%20208%20-%20Higher-order%20codecs%20for%20parameterized%20types.md) for the descriptor model. Any pgvector column can declare an explicit dimension via `vector(N)` for runtime validation and `Vector` typing, or use `vector()` to support variable dimensions. All codec paths enforce 1–16,000 finite numbers for `vector()` and exactly N finite numbers for `vector(N)`. Postgres requires compatible dimensions for distance operations. [ADR 212 — AST-bound codec resolution](../../../docs/architecture%20docs/adrs/ADR%20212%20-%20AST-bound%20codec%20resolution.md#honest-descriptor-signatures) describes how both forms retain explicit codec identity. ### Runtime Setup diff --git a/packages/3-extensions/pgvector/src/core/authoring.ts b/packages/3-extensions/pgvector/src/core/authoring.ts index c197ce0c288b..33437cce054d 100644 --- a/packages/3-extensions/pgvector/src/core/authoring.ts +++ b/packages/3-extensions/pgvector/src/core/authoring.ts @@ -5,7 +5,7 @@ export const pgvectorAuthoringTypes = { Vector: { kind: 'typeConstructor', inferred: true, - args: [{ kind: 'value', dataType: 'pg/int4', name: 'length' }], + args: [{ kind: 'value', dataType: 'pg/int4', name: 'length', optional: true }], output: { codecId: 'pg/vector@1', typeParams: { diff --git a/packages/3-extensions/pgvector/src/core/codecs.ts b/packages/3-extensions/pgvector/src/core/codecs.ts index fc741e0d07b3..f253737c41e4 100644 --- a/packages/3-extensions/pgvector/src/core/codecs.ts +++ b/packages/3-extensions/pgvector/src/core/codecs.ts @@ -4,10 +4,10 @@ * Mirrors the patterns in `postgres/codecs-class.ts` and `sqlite/codecs-class.ts` for the single `pg/vector@1` codec. Three artifacts: * * 1. `PgVectorCodec` extends {@link CodecImpl} with the wire and value conversions inline. Conversions are simple enough (PostgreSQL `[1,2,3]` text format) that no shared helper module is warranted; the class body is the source of truth. - * 2. `PgVectorDescriptor` extends {@link PostgresCodecDescriptor} with the codec id, traits, the `pgvector/vector` data type and its params schema, and explicit target behavior. The column's type, `Vector<${length}>`, comes from the pack's type function in `types/codec-types.ts`. The data type declares the type's name and the bounds of `length`. - * 3. `pgVectorColumn(length)` per-codec column helper invoking `descriptor.factory({ length })` directly. + * 2. `PgVectorDescriptor` extends {@link PostgresCodecDescriptor} with the codec id, traits, the `pgvector/vector` data type and its params schema, and explicit target behavior. The column's type, `Vector` or `Vector<${length}>`, comes from the pack's type function in `types/codec-types.ts`. The data type declares the type's name and the bounds of `length`. + * 3. `pgVectorColumn(length?)` per-codec column helper invoking `descriptor.factory({ length })` directly. * - * `length` threads into the runtime codec via the constructor so the wire methods enforce the declared dimension, and `pgvector/vector` checks it for every stored value. Without this, `vector(3)` and `vector(1536)` would produce codecs with identical behaviour and a dimension-mismatched value would round-trip undetected. + * When provided, `length` threads into the runtime codec via the constructor so the wire methods enforce the declared dimension, and `pgvector/vector` checks it for every stored value. Without this, `vector(3)` and `vector(1536)` would produce codecs with identical behaviour and a dimension-mismatched value would round-trip undetected. */ import { @@ -26,14 +26,15 @@ import { definePostgresCodecs, PostgresCodecDescriptor, } from '@internal/target-postgres/codec-descriptor'; +import { ifDefined } from '@internal/utils/defined'; import type { StandardSchemaV1 } from '@standard-schema/spec'; -import { VECTOR_CODEC_ID } from './constants'; +import { VECTOR_CODEC_ID, VECTOR_MAX_DIM } from './constants'; import { pgvectorVector, pgvectorVectorParams } from './data-types'; import { pgVectorError } from './errors'; type VectorConversionCode = 'RUNTIME.ENCODE_FAILED' | 'RUNTIME.DECODE_FAILED'; -type VectorParams = { readonly length: number }; +type VectorParams = { readonly length?: number }; function parseVector(value: string): number[] { if (!value.startsWith('[') || !value.endsWith(']')) { @@ -71,13 +72,15 @@ export class PgVectorCodec extends CodecImpl< string, number[] > { - readonly length: number; + readonly length: number | undefined; - constructor(descriptor: AnyCodecDescriptor, length: number) { - super(descriptor, pgvectorVector, { length }); + /** Creates a codec with an optional exact dimension; undefined permits variable dimensions. */ + constructor(descriptor: AnyCodecDescriptor, length: number | undefined) { + super(descriptor, pgvectorVector, ifDefined('length', length)); this.length = length; } + /** Rejects non-finite elements and lengths outside the declared dimension or supported bounds. */ assertVector(value: unknown, code: VectorConversionCode): asserts value is number[] { const meta = { codecId: VECTOR_CODEC_ID, expectedLength: this.length }; if (!Array.isArray(value)) { @@ -91,7 +94,14 @@ export class PgVectorCodec extends CodecImpl< throw pgVectorError(code, 'Vector value must contain only finite numbers', { meta }); } } - if (value.length !== this.length) { + if (this.length === undefined && !this.acceptsLength(value.length)) { + throw pgVectorError( + code, + `Vector length must be between 1 and ${VECTOR_MAX_DIM}, got ${value.length}`, + { meta: { ...meta, receivedLength: value.length } }, + ); + } + if (this.length !== undefined && value.length !== this.length) { throw pgVectorError( code, `Vector length mismatch: expected ${this.length}, got ${value.length}`, @@ -127,6 +137,13 @@ export class PgVectorCodec extends CodecImpl< this.assertVector(input, 'RUNTIME.ENCODE_FAILED'); return this.dataTypeValueOf([...input]); } + + /** Checks the exact declared dimension, or the supported 1–16,000 range when none is declared. */ + private acceptsLength(length: number): boolean { + return this.length === undefined + ? length >= 1 && length <= VECTOR_MAX_DIM + : length === this.length; + } } /** @@ -156,11 +173,30 @@ export class PgVectorDescriptor extends PostgresCodecDescriptor { export const pgVectorDescriptor = new PgVectorDescriptor(); -/** - * Per-codec column helper for `pg/vector@1`. Generic over `N extends number` so the column site preserves the dimension literal in `typeParams` (e.g. `pgVectorColumn(1536)` packs `typeParams: { length: 1536 }`). - */ -export const pgVectorColumn = (length: N) => - column(pgVectorDescriptor.factory({ length }), pgVectorDescriptor.codecId, { length }); +/** Creates a vector column with variable dimensions and empty type parameters. */ +export function pgVectorColumn(): ReturnType; +/** Creates a vector column that preserves its exact dimension as a type parameter. */ +export function pgVectorColumn( + length: N, +): ReturnType>; +/** Creates a vector column whose dimension may be absent, binding variable dimensions when it is. */ +export function pgVectorColumn( + length?: number, +): ReturnType | ReturnType>; +/** Selects a variable or fixed dimension codec factory from the optional length. */ +export function pgVectorColumn(length?: number) { + return length === undefined ? variableVectorColumn() : fixedVectorColumn(length); +} + +/** Binds an undimensioned codec using an explicit empty parameter object. */ +function variableVectorColumn() { + return column(pgVectorDescriptor.factory({}), pgVectorDescriptor.codecId, {}); +} + +/** Binds a dimensioned codec while preserving the dimension literal in its type parameters. */ +function fixedVectorColumn(length: N) { + return column(pgVectorDescriptor.factory({ length }), pgVectorDescriptor.codecId, { length }); +} pgVectorColumn satisfies ColumnHelperFor; pgVectorColumn satisfies ColumnHelperForStrict; diff --git a/packages/3-extensions/pgvector/src/core/data-types.ts b/packages/3-extensions/pgvector/src/core/data-types.ts index f3a8bcf6f8c4..b2cbe35033e8 100644 --- a/packages/3-extensions/pgvector/src/core/data-types.ts +++ b/packages/3-extensions/pgvector/src/core/data-types.ts @@ -39,10 +39,10 @@ function elementNumber(element: JsonValue): number { } export const pgvectorVectorParams = arktype({ - length: `number.integer >= 1 & number.integer <= ${VECTOR_MAX_DIM}` as const, + 'length?': `number.integer >= 1 & number.integer <= ${VECTOR_MAX_DIM}` as const, }); -/** A `vector(n)` value is an array of exactly `n` finite numbers, each one a `real` holds exactly. */ +/** A `vector(n)` value is an array of exactly `n` finite numbers, and a `vector` value one of 1 to 16,000, each one a `real` holds exactly. */ function isFiniteNumber(element: JsonValue): element is number { return typeof element === 'number' && Number.isFinite(element); } @@ -51,9 +51,11 @@ const readVector: DataTypeReader = (json, params) => { const length = typeof params['length'] === 'number' ? params['length'] : undefined; const expected = length === undefined - ? 'an array of finite numbers' + ? `an array of 1 to ${VECTOR_MAX_DIM} finite numbers` : `an array of ${counted(length, 'finite number')}`; - if (!Array.isArray(json) || (length !== undefined && json.length !== length)) { + const acceptsLength = (received: number) => + length === undefined ? received >= 1 && received <= VECTOR_MAX_DIM : received === length; + if (!Array.isArray(json) || !acceptsLength(json.length)) { return refuseJsonValue('pgvector/vector', expected, json); } if (!json.every(isFiniteNumber)) return refuseJsonValue('pgvector/vector', expected, json); @@ -88,7 +90,10 @@ function realText(value: number): string { export const pgvectorVector = sqlDataType('pgvector/vector', { params: pgvectorVectorParams, read: readVector, - texts: [{ text: 'vector({length})', written: true, catalog: true }], + texts: [ + { text: 'vector', written: true, catalog: true }, + { text: 'vector({length})', written: true, catalog: true }, + ], listCast: { of: [pgInt2.id, pgInt4.id, pgInt8.id, pgNumeric.id], cast: (elements) => elements.map(elementNumber), diff --git a/packages/3-extensions/pgvector/src/exports/column-types.ts b/packages/3-extensions/pgvector/src/exports/column-types.ts index 0d714395c885..251f3056e28c 100644 --- a/packages/3-extensions/pgvector/src/exports/column-types.ts +++ b/packages/3-extensions/pgvector/src/exports/column-types.ts @@ -1,10 +1,23 @@ /** - * Column type descriptor factory for pgvector extension. `vector(N)` is the canonical authoring surface; every pgvector column must declare a dimension via this factory. + * Column type descriptor factory for pgvector extension. `vector(N?)` is the canonical authoring surface; each pgvector column may optionally declare a dimension via this factory. When provided, the dimension threads into the runtime codec through `paramsSchema.length` and into the DDL via the data type declaration (e.g. `vector(1536)`). */ import type { ColumnTypeDescriptor } from '@internal/framework-components/codec'; import { VECTOR_CODEC_ID } from '../core/constants'; +/** + * Factory for creating non-dimensioned vector column descriptors. + * + * @example + * ```typescript + * .column('embedding', { type: vector(), nullable: false }) + * // Produces: typeParams: {} + * ``` + * @returns A column type descriptor without `typeParams.length` set + */ +export function vector(): ColumnTypeDescriptor & { + readonly typeParams: Record; +}; /** * Factory for creating dimensioned vector column descriptors. * @@ -18,7 +31,17 @@ import { VECTOR_CODEC_ID } from '../core/constants'; */ export function vector( length: N, -): ColumnTypeDescriptor & { readonly typeParams: { readonly length: N } } { +): ColumnTypeDescriptor & { readonly typeParams: { readonly length: N } }; +/** Creates a vector descriptor whose dimension may be absent, leaving variable dimensions when it is. */ +export function vector(length?: number): ColumnTypeDescriptor & { + readonly typeParams: Record | { readonly length: number }; +}; + +/** Creates a vector descriptor, omitting the dimension parameter when no length is supplied. */ +export function vector(length?: number): ColumnTypeDescriptor { + if (length === undefined) { + return { codecId: VECTOR_CODEC_ID, typeParams: {} }; + } return { codecId: VECTOR_CODEC_ID, typeParams: { length }, diff --git a/packages/3-extensions/pgvector/src/types/codec-types.ts b/packages/3-extensions/pgvector/src/types/codec-types.ts index c9a417f9e27e..e012dd99baa4 100644 --- a/packages/3-extensions/pgvector/src/types/codec-types.ts +++ b/packages/3-extensions/pgvector/src/types/codec-types.ts @@ -23,11 +23,11 @@ export type CodecTypes = CoreCodecTypes; type OutputTypeMap = { readonly 'pg/vector@1': TypeParams extends { readonly length: infer N extends number } ? Vector - : never; + : Vector; }; /** - * The output type of a `vector(N)` column, `Vector`: the only declaration of it, which a contract written with `defineContract` and the `contract.d.ts` that `prisma contract emit` writes both apply. + * The output type of a `vector()` or `vector(N)` column, `Vector` or `Vector`: the only declaration of it, which a contract written with `defineContract` and the `contract.d.ts` that `prisma contract emit` writes both apply. */ export interface ParameterizedOutputTypesForTypeParams extends ParameterizedOutputTypesFunction { readonly outputTypes: OutputTypeMap; diff --git a/packages/3-extensions/pgvector/test/codec-conformance.integration.test.ts b/packages/3-extensions/pgvector/test/codec-conformance.integration.test.ts index ba787cafd740..f7703c45c3fa 100644 --- a/packages/3-extensions/pgvector/test/codec-conformance.integration.test.ts +++ b/packages/3-extensions/pgvector/test/codec-conformance.integration.test.ts @@ -62,6 +62,17 @@ function vectorCase( } const cases: readonly PostgresCodecConformanceCase[] = [ + ...[[1], [1, 2, 3]].map( + (value): PostgresCodecConformanceCase => ({ + codecId: 'pg/vector@1', + descriptor: pgVectorDescriptor, + dataType: pgvectorVector, + label: `variable vector with ${value.length} dimensions`, + value, + typeParams: {}, + setupSql: INSTALL_VECTOR, + }), + ), vectorCase('three dimensions', [1, 2, 3]), // A vector's text form separates elements with commas and wraps them in // brackets, so a value has to carry negatives and fractions before the diff --git a/packages/3-extensions/pgvector/test/contract-infer.test.ts b/packages/3-extensions/pgvector/test/contract-infer.test.ts index d171b833e78b..e2362097acaf 100644 --- a/packages/3-extensions/pgvector/test/contract-infer.test.ts +++ b/packages/3-extensions/pgvector/test/contract-infer.test.ts @@ -11,14 +11,14 @@ import { import { describe, expect, it } from 'vitest'; import pgvectorControl from '../src/exports/control'; -function inferFrom(extensions: readonly (typeof pgvectorControl)[]) { +function inferFrom(extensions: readonly (typeof pgvectorControl)[], typeText = 'vector(3)') { const stack = createControlStack({ family: sqlFamilyDescriptor, target: postgresTarget, adapter: postgresAdapter, extensions, }); - const vector = { text: 'vector(3)', kind: undefined, schema: 'public', name: 'vector' }; + const vector = { text: typeText, kind: undefined, schema: 'public', name: 'vector' }; const tree = new PostgresDatabaseSchemaNode({ namespaces: { public: new PostgresNamespaceSchemaNode({ @@ -40,13 +40,13 @@ function inferFrom(extensions: readonly (typeof pgvectorControl)[]) { }, embedding: { name: 'embedding', - typeText: 'vector(3)', + typeText, nullable: false, reportedType: vector, }, history: { name: 'history', - typeText: 'vector(3)', + typeText, nullable: false, many: true, reportedType: vector, @@ -70,21 +70,25 @@ function inferFrom(extensions: readonly (typeof pgvectorControl)[]) { } describe('contract infer over a pgvector column', () => { - it('prints pgvector.Vector(3) when the pack is configured', async () => { - const fields = flatPslModels(await inferFrom([pgvectorControl])())[0]?.fields ?? []; - expect( - fields.map((field) => ({ - name: field.name, - path: field.typeConstructor?.path ?? [field.typeName], - args: field.typeConstructor?.args.map((arg) => arg.value) ?? [], - list: field.list, - })), - ).toEqual([ - { name: 'id', path: ['Int'], args: [], list: false }, - { name: 'embedding', path: ['pgvector', 'Vector'], args: ['3'], list: false }, - { name: 'history', path: ['pgvector', 'Vector'], args: ['3'], list: true }, - ]); - }); + it.each(['vector(3)', 'vector'])( + 'prints the constructor for %s when the pack is configured', + async (typeText) => { + const args = typeText === 'vector' ? [] : ['3']; + const fields = flatPslModels(await inferFrom([pgvectorControl], typeText)())[0]?.fields ?? []; + expect( + fields.map((field) => ({ + name: field.name, + path: field.typeConstructor?.path ?? [field.typeName], + args: field.typeConstructor?.args.map((arg) => arg.value) ?? [], + list: field.list, + })), + ).toEqual([ + { name: 'id', path: ['Int'], args: [], list: false }, + { name: 'embedding', path: ['pgvector', 'Vector'], args, list: false }, + { name: 'history', path: ['pgvector', 'Vector'], args, list: true }, + ]); + }, + ); it('fails and names each vector column when the pack is not configured', async () => { await expect(inferFrom([])()).rejects.toThrow( diff --git a/packages/3-extensions/pgvector/test/data-type-declarations.test.ts b/packages/3-extensions/pgvector/test/data-type-declarations.test.ts index f66653724dc0..ed0938aa17b0 100644 --- a/packages/3-extensions/pgvector/test/data-type-declarations.test.ts +++ b/packages/3-extensions/pgvector/test/data-type-declarations.test.ts @@ -38,6 +38,7 @@ describe('the pgvector data type declaration', () => { it('declares its texts and claims no kind', () => { expect(vectorType().sql.texts).toEqual([ + { text: 'vector', written: true, catalog: true }, { text: 'vector({length})', written: true, catalog: true }, ]); expect(vectorType().sql.claimsKind).toBeUndefined(); @@ -48,6 +49,11 @@ describe('the pgvector data type declaration', () => { expect(renderSqlCatalogText(vectorType(), { length: 1536 })).toBe('vector(1536)'); }); + it('writes and reports an unspecified length', () => { + expect(renderSqlTypeName(vectorType(), {})).toBe('vector'); + expect(renderSqlCatalogText(vectorType(), {})).toBe('vector'); + }); + it('has the base name vector', () => { expect(sqlBaseName(vectorType(), { length: 3 })).toBe('vector'); expect(sqlBaseName(vectorType(), {})).toBe('vector'); @@ -63,7 +69,10 @@ describe('the pgvector data type declaration', () => { dataType: 'pgvector/vector', typeParams: { length: 3 }, }); - expect(resolveReportedSqlType(reported('vector'), dataTypes)).toBeUndefined(); + expect(resolveReportedSqlType(reported('vector'), dataTypes)).toEqual({ + dataType: 'pgvector/vector', + typeParams: {}, + }); expect(resolveReportedSqlType(reported('vector(16001)'), dataTypes)).toBeUndefined(); }); @@ -71,12 +80,9 @@ describe('the pgvector data type declaration', () => { expect(renderSqlTypeName(vectorType(), { length })).toBe(`vector(${length})`); }); - it.each([[{ length: 0 }], [{ length: 16001 }], [{ length: 1.5 }], [{}]])( - 'refuses %j', - (params) => { - expect(() => renderSqlTypeName(vectorType(), params)).toThrow(invalidParams); - }, - ); + it.each([[{ length: 0 }], [{ length: 16001 }], [{ length: 1.5 }]])('refuses %j', (params) => { + expect(() => renderSqlTypeName(vectorType(), params)).toThrow(invalidParams); + }); it('normalizes to itself', () => { const { normalize } = vectorType().sql; diff --git a/packages/3-extensions/pgvector/test/default-encoding.test.ts b/packages/3-extensions/pgvector/test/default-encoding.test.ts index adda1593df20..d1c026a23d3b 100644 --- a/packages/3-extensions/pgvector/test/default-encoding.test.ts +++ b/packages/3-extensions/pgvector/test/default-encoding.test.ts @@ -57,6 +57,35 @@ describe('literal defaults on a codec contributed through extensions', () => { }); }); + it.each(['typeHelper', 'column'] as const)( + 'stores a default without a dimension through %s', + (form) => { + const contract = defineContract( + { extensions: { pgvector } }, + ({ field, model, type, gen }) => { + const embedding = field.column(form === 'typeHelper' ? type.pgvector.Vector() : vector()); + return { + models: { + Doc: model('Doc', { + fields: { + id: field.column(type.Char(36)).id().onCreate(gen.uuid()), + embedding: embedding.default([1, 2]), + }, + }), + }, + }; + }, + ); + expect( + contract.storage.namespaces['public']?.entries.table?.['Doc']?.columns['embedding'], + ).toMatchObject({ + dataType: 'pgvector/vector', + codecId: 'pg/vector@1', + default: { kind: 'literal', value: [1, 2] }, + }); + }, + ); + describe('on a column whose type parameters the data type does not accept', () => { it('reports the type parameters, not the default', () => { expect(() => @@ -65,7 +94,7 @@ describe('literal defaults on a codec contributed through extensions', () => { Doc: model('Doc', { fields: { id: field.column(type.Char(36)).id().onCreate(gen.uuid()), - embedding: field.column({ codecId: 'pg/vector@1' } as const).default([1, 2, 3]), + embedding: field.column(vector(0)).default([1, 2, 3]), }, }), }, @@ -73,8 +102,7 @@ describe('literal defaults on a codec contributed through extensions', () => { ).toThrow( expect.objectContaining({ code: 'CONTRACT.TYPE_PARAMS_INVALID', - message: - 'Field "Doc.embedding" has type parameters that its data type does not accept: pgvector/vector: length must be a number (was missing)', + message: expect.stringContaining('length'), meta: { dataType: 'pgvector/vector', parameters: ['length'], diff --git a/packages/3-extensions/pgvector/test/nearest-collection-method.types.test-d.ts b/packages/3-extensions/pgvector/test/nearest-collection-method.types.test-d.ts index 9ba487468743..e6ce38ab4d4c 100644 --- a/packages/3-extensions/pgvector/test/nearest-collection-method.types.test-d.ts +++ b/packages/3-extensions/pgvector/test/nearest-collection-method.types.test-d.ts @@ -41,6 +41,22 @@ describe('the pgvector nearest-neighbour collection methods', () => { }>(); }); + test('accepts vectors of any length for an undimensioned field', () => { + type Methods = MethodsFor<{ + readonly embedding: { + readonly nullable: false; + readonly type: { + readonly kind: 'scalar'; + readonly codecId: 'pg/vector@1'; + readonly typeParams: Record; + }; + }; + }>; + expectTypeOf().toEqualTypeOf< + readonly [vector: Vector, maxDistance?: number] + >(); + }); + test('are none for a model with no vector field', () => { expectTypeOf>>().toEqualTypeOf>(); }); diff --git a/packages/3-extensions/pgvector/test/parameterized-output-types.test-d.ts b/packages/3-extensions/pgvector/test/parameterized-output-types.test-d.ts index 12f332eb7efb..d5e012a3181c 100644 --- a/packages/3-extensions/pgvector/test/parameterized-output-types.test-d.ts +++ b/packages/3-extensions/pgvector/test/parameterized-output-types.test-d.ts @@ -19,7 +19,6 @@ test('a vector column with a length reads as a vector of that length', () => { }); test('a vector column without a length reads as the plain vector type', () => { - expectTypeOf>>().toEqualTypeOf< - CodecTypes['pg/vector@1']['output'] - >(); + expectTypeOf>>().not.toBeAny(); + expectTypeOf>>().toEqualTypeOf(); }); diff --git a/packages/3-extensions/pgvector/test/variable-dimensions.test.ts b/packages/3-extensions/pgvector/test/variable-dimensions.test.ts new file mode 100644 index 000000000000..d31d65aae5a1 --- /dev/null +++ b/packages/3-extensions/pgvector/test/variable-dimensions.test.ts @@ -0,0 +1,76 @@ +import { + instantiateAuthoringTypeConstructor, + validateAuthoringHelperArguments, +} from '@internal/framework-components/authoring'; +import { renderSqlTypeName } from '@internal/sql-contract/data-type'; +import { describe, expect, it } from 'vitest'; +import { pgvectorAuthoringTypes } from '../src/core/authoring'; +import { pgVectorColumn, pgVectorDescriptor } from '../src/core/codecs'; +import { VECTOR_MAX_DIM } from '../src/core/constants'; +import { pgvectorVector } from '../src/core/data-types'; +import { vector } from '../src/exports/column-types'; +import { fromContractJson, toContractJson } from './contract-json'; + +describe('variable dimensions', () => { + it('authors a vector without dimension metadata', () => { + expect(vector()).toEqual({ codecId: 'pg/vector@1', typeParams: {} }); + expect(pgVectorColumn().typeParams).toEqual({}); + }); + + it.each([ + [undefined, {}], + [3, { length: 3 }], + ])('authors a vector from an optional length of %s', (length, typeParams) => { + const optionalLength: number | undefined = length; + expect(vector(optionalLength)).toEqual({ codecId: 'pg/vector@1', typeParams }); + expect(pgVectorColumn(optionalLength).typeParams).toEqual(typeParams); + }); + + it('authors PSL vectors without a length argument', () => { + validateAuthoringHelperArguments( + 'pgvector.Vector', + pgvectorAuthoringTypes.pgvector.Vector.args, + [], + ); + expect( + instantiateAuthoringTypeConstructor(pgvectorAuthoringTypes.pgvector.Vector, []), + ).toMatchObject({ + codecId: 'pg/vector@1', + }); + }); + + it('validates and renders undimensioned contracts', async () => { + expect(await pgVectorDescriptor.paramsSchema['~standard'].validate({})).toEqual({ value: {} }); + expect(renderSqlTypeName(pgvectorVector, {})).toBe('vector'); + }); + + it('rejects invalid JSON elements without a declared length', () => { + const codec = pgVectorColumn().codecFactory({ name: 'embedding' }); + for (const value of [[null], ['1'], [Number.POSITIVE_INFINITY], [Number.NaN]]) { + expect(() => fromContractJson(codec, value)).toThrow(); + } + }); + + it.each([0, VECTOR_MAX_DIM + 1])('rejects length %i through every codec path', async (length) => { + const codec = pgVectorColumn().codecFactory({ name: 'embedding' }); + const value = Array(length).fill(1); + const wire = `[${value.join(',')}]`; + await expect(codec.toWire(value, {})).rejects.toThrow(); + await expect(codec.fromWire(wire, {})).rejects.toThrow(); + expect(() => toContractJson(codec, value)).toThrow(); + expect(() => fromContractJson(codec, value)).toThrow( + `pgvector/vector JSON value must be an array of 1 to ${VECTOR_MAX_DIM} finite numbers`, + ); + }); + + it('accepts different lengths through every codec path', async () => { + const codec = pgVectorColumn().codecFactory({ name: 'embedding' }); + for (const value of [[1], [1, 2, 3], [1, 2], Array(VECTOR_MAX_DIM).fill(1)]) { + const wire = `[${value.join(',')}]`; + expect(await codec.toWire(value, {})).toBe(wire); + expect(await codec.fromWire(wire, {})).toEqual(value); + expect(toContractJson(codec, value)).toEqual(value); + expect(fromContractJson(codec, value)).toEqual(value); + } + }); +}); diff --git a/packages/9-public/@prisma/orm-extension-pgvector/README.md b/packages/9-public/@prisma/orm-extension-pgvector/README.md index 52c2ac1ac320..51177af4c85c 100644 --- a/packages/9-public/@prisma/orm-extension-pgvector/README.md +++ b/packages/9-public/@prisma/orm-extension-pgvector/README.md @@ -11,11 +11,11 @@ pnpm add @prisma/orm-extension-pgvector | Namespace | Surface | | --- | --- | | `/pack` | the extension pack an application composes into `extensions: [...]` — pure, no runtime imports | -| `/column-types` | the `Vector(n)` column author | +| `/column-types` | the `vector()` or `vector(n)` column author | | `/codec-types`, `/operation-types` | types emitted contracts reference | | `/runtime` | the runtime extension that registers the codec and operations | | `/control` | the control descriptor and baseline migration that install the server extension | ## Responsibilities -Dimensioned vector storage and search: the `pg/vector@1` codec (`number[]` at runtime, `Vector` in `contract.d.ts`), similarity operations such as `cosineDistance`, and a baseline migration that runs `CREATE EXTENSION IF NOT EXISTS vector` when the pack is composed into an application. +Variable or fixed-dimension vector storage and search: the `pg/vector@1` codec (`number[]` at runtime, `Vector` in `contract.d.ts`), similarity operations such as `cosineDistance`, and a baseline migration that runs `CREATE EXTENSION IF NOT EXISTS vector` when the pack is composed into an application. diff --git a/test/integration/test/authoring/psl.pgvector-dbinit.test.ts b/test/integration/test/authoring/psl.pgvector-dbinit.test.ts index fe9d647cd97f..10ad21207d82 100644 --- a/test/integration/test/authoring/psl.pgvector-dbinit.test.ts +++ b/test/integration/test/authoring/psl.pgvector-dbinit.test.ts @@ -133,16 +133,16 @@ describe( } } - it( - 'dbInit succeeds for a PSL-emitted pgvector named type schema', - async () => { + it.each(['1536', ''])( + 'dbInit succeeds for a PSL-emitted pgvector named type schema with dimension %s', + async (dimension) => { const emittedContract = await emitPgvectorContract(`types { - Embedding1536 = pgvector.Vector(1536) + Embedding = pgvector.Vector(${dimension}) } model Document { id Int @id @default(autoincrement()) - embedding Embedding1536 + embedding Embedding } `); @@ -166,7 +166,9 @@ model Document { .filter((s) => s.language === 'sql') .map((s) => s.text) .join(';\n\n') ?? ''; - expect(ddl).toContain('vector(1536)'); + expect(ddl).toMatch( + dimension ? `"embedding" vector(${dimension})` : /"embedding" vector(?!\()/, + ); expect(ddl).not.toContain('"vector(1536)"'); const apply = await client.dbInit({ @@ -181,6 +183,17 @@ model Document { ); } }); + if (dimension === '') { + await withClient(connectionString, async (client) => { + await client.query( + `INSERT INTO "Document" ("embedding") VALUES ('[1]'), ('[1,2,3]')`, + ); + const result = await client.query<{ dimensions: number }>( + `SELECT vector_dims("embedding") AS dimensions FROM "Document" ORDER BY "id"`, + ); + expect(result.rows).toEqual([{ dimensions: 1 }, { dimensions: 3 }]); + }); + } }); }, timeouts.spinUpPpgDev, diff --git a/test/integration/test/packaging/extension-tarball.test.ts b/test/integration/test/packaging/extension-tarball.test.ts index 77754855cc36..417701647229 100644 --- a/test/integration/test/packaging/extension-tarball.test.ts +++ b/test/integration/test/packaging/extension-tarball.test.ts @@ -81,6 +81,27 @@ describe('an extension pack installed next to the facade it extends', () => { expect(runInScratch(scratch, script)).toContain(`resolved ${subpaths.length}`); }); + it('supports variable dimensions from the installed package', () => { + expect( + runInScratch( + scratch, + ` + import { strict as assert } from 'node:assert'; + import { vector } from '${extension}/column-types'; + import runtime from '${extension}/runtime'; + assert.deepEqual(vector().typeParams, {}); + const descriptor = runtime.codecs().find(codec => codec.codecId === 'pg/vector@1'); + const codec = descriptor.factory({})({ name: 'embedding' }); + for (const value of [[1], [1, 2, 3]]) { + assert.deepEqual(await codec.fromWire(await codec.toWire(value, {}), {}), value); + assert.deepEqual(codec.fromDataTypeValue(codec.toDataTypeValue(value)), value); + } + console.log('variable dimensions ok'); + `, + ), + ).toContain('variable dimensions ok'); + }); + it('requires its target shell as an exact-pinned peer, not a dependency', () => { const manifest: unknown = JSON.parse(readFileSync(join(installedDir, 'package.json'), 'utf8')); const { dependencies, peerDependencies } = Object(manifest) as {