Merge v0.10.0

.agents/skills/hyperindex/SKILL.md

@@ -45,12 +45,13 @@ Use production by default for consumer examples. `api.indexer.hypercerts.dev` is
 ## Before answering
 
 1. If the user asks for an exact field, filter, enum, or union and you are not sure, introspect the endpoint first.
-2. Prefer schema-specific queries such as `orgHypercertsClaimActivity` over generic `records` when the collection has a typed query.
-3. Use `recordTimeline` when the caller needs one newest-first feed across multiple collections. It requires `where.collection.in`, supports optional `where.did.in` author filtering, and does not expose `totalCount`.
-4. Always include pagination (`first`, `after`, `pageInfo { hasNextPage endCursor }`) in list examples.
-5. Keep selection sets small. Add fields only when needed for the workflow.
-6. Use inline fragments for union fields such as descriptions, images, attachment content, proof fields, and strong references.
-7. Do not assume the target endpoint exposes every feature described in this skill. `main` may be ahead of staging, and staging may be ahead of production. When a query depends on newer schema features such as `recordTimeline`, nested filters, author labels, or collection-specific fields, introspect the target endpoint first. If the feature is missing, tell the user which endpoint lacks it and offer the closest fallback, such as `search(query: ..., collection: ...)` plus client-side filtering.
+2. Prefer schema-specific queries such as `orgHypercertsClaimActivity` over generic `records` when the collection has a typed query and the caller only needs schema-valid records.
+3. Use generic `records(collection: ...)` when the caller needs raw JSON, operational debugging, unknown-schema records, or records hidden from typed GraphQL by validation metadata.
+4. Use `recordTimeline` when the caller needs one newest-first feed across multiple collections. It requires `where.collection.in`, supports optional `where.did.in` author filtering, and does not expose `totalCount`.
+5. Always include pagination (`first`, `after`, `pageInfo { hasNextPage endCursor }`) in list examples.
+6. Keep selection sets small. Add fields only when needed for the workflow.
+7. Use inline fragments for union fields such as descriptions, images, attachment content, proof fields, and strong references.
+8. Do not assume the target endpoint exposes every feature described in this skill. `main` may be ahead of staging, and staging may be ahead of production. When a query depends on newer schema features such as `recordTimeline`, nested filters, author labels, validation metadata, or collection-specific fields, introspect the target endpoint first. If the feature is missing, tell the user which endpoint lacks it and offer the closest fallback, such as `search(query: ..., collection: ...)` plus client-side filtering.
 
 Detailed schema reference: [references/schema-reference.md](references/schema-reference.md)
 
@@ -165,13 +166,47 @@ query EndorsementClosure($did: String!) {
 }
 ```
 
-`where.did.eq` is required and selects the root DID. The endorsement closure DID filter exposes only `eq`, not `in`, because each request is rooted at one DID. Optional `where.degree.eq` returns only one hop distance; the value must be `1`, `2`, or `3`. Omit `where.degree` to return all supported degrees. Results are sorted by degree then DID. `certifiedProfileData` resolves the reached account's Certified profile when one exists. `viaAccounts` lists up to 64 previous-ring accounts that led to the account, including each predecessor DID and optional Certified profile data; it is empty for direct degree-1 accounts. `truncated: true` means the server-side account cap was reached. The resolver computes active endorsement edges from current Certified badge award, definition, and response records at request time; it only counts badge awards whose subject is the `app.certified.defs#did` account DID union member, ignores record strongRef subjects, respects badge-definition `allowedIssuers` allowlists, and does not use a persisted edge table.
+`where.did.eq` is required and selects the root DID. The endorsement closure DID filter exposes only `eq`, not `in`, because each request is rooted at one DID. Optional `where.degree.eq` returns only one hop distance; the value must be `1`, `2`, or `3`. Omit `where.degree` to return all supported degrees. Results are sorted by degree then DID. `certifiedProfileData` resolves the reached account's Certified profile when one exists. `viaAccounts` lists up to 64 previous-ring accounts that led to the account, including each predecessor DID and optional Certified profile data; it is empty for direct degree-1 accounts. `truncated: true` means the server-side account cap was reached. The resolver computes active endorsement edges from current Certified badge award, definition, and response records whose `validationStatus` is `valid`; invalid records cannot create or suppress edges. It only counts badge awards whose subject is the `app.certified.defs#did` account DID union member, ignores record strongRef subjects, respects badge-definition `allowedIssuers` allowlists, and does not use a persisted edge table.
 
 If a workflow needs unsupported nested matching, use one of these patterns:
 
 - Use typed nested/presence filters to narrow the set, then filter client-side.
 - Use `search(query: ..., collection: ...)` to find records whose JSON contains a referenced AT-URI or string.
-- Use `records(collection: ...)` as a fallback for collections without typed schema coverage.
+- Use `records(collection: ...)` as a fallback for collections without typed schema coverage or for raw debugging.
+
+## Validation gate and generic records
+
+Hyperindex stores every observed AT Protocol record in its raw record table. Typed GraphQL collection fields only expose rows whose `validationStatus` is `valid`, meaning Indigo validated the record against the startup Lexicon set used to generate the running schema. Records whose status is `invalid`, `unknown_schema`, or `validation_error` are hidden from typed list queries, typed `ByUri` queries, typed counts, relationship hydration, and typed create/update subscriptions. Typed delete subscriptions emit only for rows that were valid before deletion.
+
+Use generic `records(collection: ...)` or `search(...)` for operational visibility into raw rows, including records hidden from typed GraphQL. Generic record nodes expose validation metadata:
+
+```graphql
+query RawRecords($collection: String!) {
+  records(collection: $collection, first: 20) {
+    edges {
+      node {
+        uri
+        cid
+        did
+        collection
+        rkey
+        validationStatus
+        validationError
+        validatedAt
+        lexiconHash
+        value
+      }
+    }
+    pageInfo { hasNextPage endCursor }
+  }
+}
+```
+
+Validation is local-only during ingestion. Hyperindex uses Indigo to check records against the Lexicons loaded at startup and does not resolve `_lexicon` DNS records, DID documents, PDS-hosted schema records, or other remote schema sources while classifying records. `lexiconHash` is a validation fingerprint for the saved collection Lexicon and any transitive referenced Lexicons used during classification.
+
+Public typed GraphQL, record validation, and default Jetstream collection filters use one fixed Lexicon set loaded at startup. Lexicon upload/register/delete changes only saved configuration; restart or redeploy Hyperindex to apply the change everywhere together. For multiple backend replicas, coordinate Lexicon-changing rollouts so old and new startup snapshots never serve concurrently against the shared validation metadata.
+
+Generic `recordEvents` receives every observed raw create/update/delete event, including invalid and unknown-schema records. Typed collection subscriptions filter that stream to valid create/update rows and deletes that were valid before removal.
 
 ## Generic record timeline
 

.agents/skills/hyperindex/references/schema-reference.md

@@ -1,6 +1,6 @@
 # Hyperindex GraphQL Schema Reference
 
-Last updated on 2026-07-28 for pending record author identity changes. Baseline generated from live introspection of `https://api.indexer.hypercerts.dev/graphql` on 2026-06-10; `author` metadata described below reflects the current branch and may not yet be deployed.
+Last updated on 2026-07-30 for pending record validation gate and record author identity changes. Baseline generated from live introspection of `https://api.indexer.hypercerts.dev/graphql` on 2026-06-10; validation and `author` metadata described below reflect the current branch and may not yet be deployed.
 
 ## Endpoints
 
@@ -49,7 +49,7 @@ Last updated on 2026-07-28 for pending record author identity changes. Baseline
 | `orgHypercertsFundingReceiptByUri` | uri: `String!` | Get a single org.hypercerts.funding.receipt by AT-URI |
 | `orgHypercertsWorkscopeTag` | after: `String`, before: `String`, first: `Int`, last: `Int`, sortBy: `OrgHypercertsWorkscopeTagSortField`, sortDirection: `SortDirection`, where: `OrgHypercertsWorkscopeTagWhereInput` | Query org.hypercerts.workscope.tag records |
 | `orgHypercertsWorkscopeTagByUri` | uri: `String!` | Get a single org.hypercerts.workscope.tag by AT-URI |
-| `records` | after: `String`, before: `String`, collection: `String!`, first: `Int`, last: `Int` | Query records from any collection (useful for collections without lexicon schemas) |
+| `records` | after: `String`, before: `String`, collection: `String!`, first: `Int`, last: `Int` | Query raw records from any collection, including records hidden from typed GraphQL by validation metadata. |
 | `recordTimeline` | after: `String`, first: `Int`, where: `RecordTimelineWhereInput!` | Query a newest-first page of current records across selected collections, optionally filtered by author DIDs. |
 | `externalLabels` | activeOnly: `Boolean`, sources: `[String!]`, subjects: `[String!]!`, values: `[String!]` | Query locally ingested external ATProto labels by DID or AT-URI subject. |
 | `search` | after: `String`, collection: `String`, first: `Int`, query: `String!` | Search records by text content |
@@ -81,7 +81,33 @@ Last updated on 2026-07-28 for pending record author identity changes. Baseline
 | `org.hypercerts.funding.receipt` | `orgHypercertsFundingReceipt` | `orgHypercertsFundingReceiptByUri` | `OrgHypercertsFundingReceipt` |
 | `org.hypercerts.workscope.tag` | `orgHypercertsWorkscopeTag` | `orgHypercertsWorkscopeTagByUri` | `OrgHypercertsWorkscopeTag` |
 
-Typed list queries accept Relay-style pagination arguments (`first`, `after`, `last`, `before`), plus `where`, `sortBy`, and `sortDirection` when the collection exposes those inputs.
+Typed list queries accept Relay-style pagination arguments (`first`, `after`, `last`, `before`), plus `where`, `sortBy`, and `sortDirection` when the collection exposes those inputs. Typed list queries, typed single-record queries, typed collection counts, relationship hydration, and typed create/update subscriptions only expose rows whose saved validation status is `valid`. Typed delete subscriptions emit only for rows that were valid before deletion. Typed single-record queries return `null` when a raw row exists but is `invalid`, `unknown_schema`, or `validation_error`.
+
+## Record validation gate
+
+Hyperindex stores every observed AT Protocol record in the raw `record` table. Validation metadata controls whether the row is safe to serve through generated, typed GraphQL fields:
+
+| Status | Typed GraphQL visibility | Meaning |
+| --- | --- | --- |
+| `valid` | Visible | Indigo confirmed that the record conforms to Hyperindex's saved startup Lexicon snapshot, with the record key validated locally against that same snapshot. |
+| `invalid` | Hidden | The startup snapshot contains the collection Lexicon, but the record does not conform to it. |
+| `unknown_schema` | Hidden | The startup snapshot contains no saved Lexicon for the collection. |
+| `validation_error` | Hidden | Hyperindex could not complete validation against the startup snapshot because of malformed data, an incomplete local Lexicon set, or an internal validation error. |
+
+Validation uses saved Lexicons only. Normal ingestion does not resolve `_lexicon` DNS records, DID documents, PDS-hosted schema records, or any other remote schema source while classifying records.
+
+The generic `records(collection: ...)` query returns all raw records for the collection, including rows hidden from typed GraphQL. `search(...)` uses the same raw visibility and metadata contract. Generic record nodes expose these validation metadata fields:
+
+| Field | Type | Notes |
+| --- | --- | --- |
+| `validationStatus` | `String!` | One of `valid`, `invalid`, `unknown_schema`, or `validation_error`. |
+| `validationError` | `String` | Explanation of the failed or hidden validation state, when available. |
+| `validatedAt` | `String` | Timestamp when Hyperindex last classified the record against its saved startup Lexicon snapshot. |
+| `lexiconHash` | `String` | SHA-256 validation fingerprint for the exact saved collection Lexicon bytes and transitive referenced Lexicons in that startup snapshot. |
+
+Public typed GraphQL, Indigo record validation, and default Jetstream collection filters use one fixed Lexicon set loaded at startup. Lexicon upload/register/delete changes only saved configuration; restart or redeploy Hyperindex to apply the change everywhere together.
+
+Generic `recordEvents` receives all observed raw events. Typed collection subscriptions filter those events to valid create/update records and deletes that were valid before removal.
 
 ## Generic record timeline
 
@@ -173,6 +199,8 @@ Rows are ordered by top-level record JSON `createdAt` descending, then `uri` des
 | `did` | `String!` | DID of the predecessor account. |
 | `certifiedProfileData` | `AppCertifiedActorProfile` | Certified profile for the predecessor account, or null when no profile record exists. |
 
+`endorsementClosure` only uses badge awards, badge definitions, and badge responses whose `validationStatus` is `valid`. Invalid or unclassified awards and definitions cannot create edges, and invalid or unclassified responses cannot suppress them.
+
 Example:
 
 ```graphql

.changes/unreleased/add-record-validation-gate.yaml

@@ -0,0 +1,4 @@
+kind: changed
+body: Add an Indigo-backed record validation gate that atomically stores raw records with validation metadata while typed GraphQL and typed subscriptions expose only valid data. Generic records, search, and raw subscriptions retain invalid and unknown-schema data for diagnostics; admin Lexicon changes are validated and staged for the next process snapshot, so operators must restart or redeploy to apply them and coordinate multi-replica rollouts so old and new snapshots do not serve concurrently. PostgreSQL operators with large record tables can pre-create the validation refresh paging index before rollout to avoid a write-blocking startup index build.
+custom:
+  Affects: user

.changes/unreleased/fix-endorsement-validation-gate.yaml

@@ -0,0 +1,4 @@
+kind: fixed
+body: Fix `endorsementClosure` so only validated badge awards, badge definitions, and badge responses affect the computed trust graph. Invalid or unclassified raw records remain available through generic queries but can no longer create or suppress endorsement edges.
+custom:
+  Affects: user

README.md

@@ -56,7 +56,7 @@ Lexicons define the AT Protocol record types you want to index. Hyperindex suppo
 
 Or place lexicon JSON files in a directory and set the `LEXICON_DIR` environment variable.
 
-After registering by NSID or uploading a ZIP file, restart/redeploy the backend indexer for the new lexicons to appear in the public GraphQL schema and query list. The admin lexicon list updates immediately, but typed GraphQL queries are generated at backend startup.
+After registering, uploading, or deleting Lexicons, restart/redeploy the backend indexer. The admin Lexicon list updates immediately, but public GraphQL, Indigo record validation, and default Jetstream collection filters all keep using the Lexicon set loaded at startup until the process restarts. For multi-replica deployments, coordinate Lexicon-changing rollouts so replicas using the old and new startup snapshots never serve concurrently against the shared validation metadata.
 
 **Example lexicons:**
 - `org.hypercerts.claim.activity` - Hypercert claim activity

client/src/app/lexicons/page.tsx

@@ -276,7 +276,7 @@ export default function LexiconsPage() {
     },
     onSuccess: (response) => {
       const count = response.uploadLexicons;
-      setSuccess(`Uploaded ${count} lexicon${count !== 1 ? "s" : ""}`);
+      setSuccess(`Uploaded ${count} lexicon${count !== 1 ? "s" : ""}. Restart or redeploy Hyperindex to apply the change.`);
       setError(null);
       setZipFile(null);
       if (zipFileInputRef.current) {
@@ -297,7 +297,7 @@ export default function LexiconsPage() {
       graphqlClient.request(DELETE_LEXICON, { nsid }),
     onMutate: (nsid) => setDeletingNsid(nsid),
     onSuccess: (_, nsid) => {
-      setSuccess(`Deleted ${nsid}`);
+      setSuccess(`Deleted ${nsid}. Restart or redeploy Hyperindex to apply the change.`);
       setError(null);
       if (expandedId === nsid) setExpandedId(null);
       queryClient.invalidateQueries({ queryKey: ["lexicons"] });
@@ -352,10 +352,10 @@ export default function LexiconsPage() {
       queryClient.invalidateQueries({ queryKey: ["lexicons"] });
       if (!firstError) {
         setNsidInput("");
-        setSuccess(`Registered ${completed} lexicon${completed !== 1 ? "s" : ""}`);
-        setTimeout(() => setSuccess(null), 3000);
+        setSuccess(`Registered ${completed} lexicon${completed !== 1 ? "s" : ""}. Restart or redeploy Hyperindex to apply the change.`);
+        setTimeout(() => setSuccess(null), 5000);
       } else {
-        setSuccess(`Registered ${completed}/${nsids.length} lexicons (stopped on error)`);
+        setSuccess(`Registered ${completed}/${nsids.length} lexicons before the error. Restart or redeploy Hyperindex to apply the saved changes.`);
       }
     }
   };
@@ -478,7 +478,7 @@ export default function LexiconsPage() {
               Register published lexicons by NSID
             </h3>
             <p className="mt-1 text-xs" style={{ color: "var(--muted-foreground)" }}>
-              Resolve published AT Protocol lexicons by NSID and add them to this AppView.
+              Resolve and save published AT Protocol Lexicons by NSID. Restart or redeploy Hyperindex to apply them.
             </p>
             <form onSubmit={handleRegister} className="mt-4 flex flex-col gap-3 sm:flex-row sm:items-start">
               <label htmlFor="lexicon-nsids" className="sr-only">
@@ -521,7 +521,7 @@ export default function LexiconsPage() {
             </h3>
             <div className="mt-1 space-y-1 text-xs" style={{ color: "var(--muted-foreground)" }}>
               <p>Upload a .zip containing one or more lexicon .json files. Lexicons do not need to be published yet.</p>
-              <p>Each JSON file must contain a top-level id field. A backend restart may be required before new lexicons appear in the public GraphQL schema.</p>
+              <p>Each JSON file must contain a top-level id field. Restart or redeploy Hyperindex to apply saved changes to GraphQL, validation, and ingestion.</p>
             </div>
             <form onSubmit={handleUpload} className="mt-4 flex flex-col gap-3 sm:flex-row sm:items-center">
               <label htmlFor="lexicon-zip-file" className="sr-only">