Skip to content

Drop the plaintext personal access token column (D3 of 3: drop identity_access_tokens.token) - #3203

Draft
jeremy wants to merge 6 commits into
oauth-token-digests-d2from
oauth-token-digests-d3
Draft

jeremy wants to merge 6 commits into
oauth-token-digests-d2from
oauth-token-digests-d3

Conversation

@jeremy

@jeremy jeremy commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Caution

Explicitly gated. Don't merge this until D2 (#3202) is deployed to production and to beta (every beta destination: they share fizzy_production), and to staging, and every process that might name token on INSERT has been replaced. Production runs with partial_inserts off (load_defaults 8.1), so a process running D1 or anything older names every column it knows on every INSERT. Once token is gone, such a process fails to create personal access tokens (Unknown column 'token'), fails OAuth refreshes (D1's rotation writes token: nil), and answers an unknown Bearer token with a 500 instead of a 401 (D1's plaintext fallback queries token). D2 ignores the column, so only D2 or later may be running when this migrates.

Warning

This PR contains migrations. Don't merge it directly. It has to go through the beta integration branch (oauth-integration-2026-10-15) and be migrated by hand on beta before it lands. beta1 runs against fizzy_production (saas/config/database.yml ends beta: *production), so migrating on beta1 is migrating production. The rows to add to that branch's migration inventory are below; this PR doesn't push to the integration branch.

Stacked on #3202 (train: … → #3169 → #3192 → #3202 → this).

Implements step D3, the last, of hashing tokens at rest (rubric TK-2, bearer artifacts stored only as digests):

Step What it does Status
D1 (#3192) Add digest columns, backfill, read by digest with a plaintext fallback, keep writing personal plaintext after the OAuth stack
D2 (#3202) Rerun the backfill, stop writing and reading plaintext, ignore token after D1 is deployed to production
D3 (this PR) Drop identity_access_tokens.token after D2 is deployed everywhere

What D3 does

  • DropIdentityAccessTokenPlaintext: on MySQL, ALTER TABLE identity_access_tokens DROP INDEX index_identity_access_tokens_on_refresh_token, DROP COLUMN refresh_token, DROP COLUMN token, ALGORITHM=INPLACE, LOCK=NONE, a table rebuild; elsewhere remove_index and remove_column. The plaintext refresh_token column was kept out of Store tokens and client secrets as digests (D1 of 3: read by digest, plaintext fallback) #3192's DropPlaintextOauthCredentials, so that no migration dropped a column production main had cached; Store tokens and client secrets as digests (D1 of 3: read by digest, plaintext fallback) #3192 ignores it and empties it, so it goes here. MySQL 8.4 would otherwise drop it INSTANT, from the data dictionary alone, leaving each row's plaintext in the table's pages until the row is next rewritten, which for a personal access token is usually never (updated_at doesn't move on use). With it go the plaintext personal access tokens D1 and D2 left beside their digests. No token is stored in plaintext any more.
  • down restores both columns empty: nullable strings, refresh_token with its unique index, as they were before any token was written. The values are unrecoverable: only digests remain, so code that reads only plaintext (production main before D1) can't authenticate any token again. D1 and D2 read digests and keep working.
  • Identity::AccessToken no longer needs ignored_columns, so it's removed. db/schema.rb and db/schema_sqlite.rb drop both columns and the index, and move to 2026_10_17_130000.
  • Tests: D1's and D2's backfill tests are deleted, since both write plaintext into a column that no longer exists. Both migrations stay, and run in order on a fresh db:migrate before this one. Identity::AccessTokenTest asserts both columns are gone. test/migrations/drop_identity_access_token_plaintext_test.rb caches the table's columns as D2's code does (ignoring both), runs the migration, and creates a token with them.
  • Docs: the conformance record's token row says step 3 is done, and docs/docker-deployment.md now says tokens are stored only as digests, not just looked up by them.

Migration inventory rows for the integration branch

For docs/oauth/integration-migrations.md on oauth-integration-2026-10-15. This PR doesn't change that branch. Labeled D3-1 so a sibling PR's rows after #3192's 8–10 don't collide:

# Version / file PR What it does Production table MySQL 8.4 DDL
D3-1 20261017130000_drop_identity_access_token_plaintext.rb this MySQL: ALTER TABLE identity_access_tokens DROP INDEX index_identity_access_tokens_on_refresh_token, DROP COLUMN refresh_token, DROP COLUMN token, ALGORITHM=INPLACE, LOCK=NONE. down adds both back, nullable and empty, with the index identity_access_tokens: ~3,500 rows INPLACE, LOCK=NONE: a table rebuild that allows concurrent reads and writes, forced rather than MySQL's default INSTANT so the plaintext leaves the pages (see above). Sub-second at ~3,500 rows and 0.65 MB; on 8.4.11 the rebuild was confirmed by the table's InnoDB TABLE_ID changing and TOTAL_ROW_VERSIONS resetting to 0. token has no index of its own; refresh_token's goes in the same statement

Notes for the runbook:

  • The lock that matters. Like every online ALTER, the rebuild takes an exclusive metadata lock on identity_access_tokens, which every Bearer request reads, briefly at its start and again at its end. Each time it waits for every open transaction that has touched the table, and while it waits, every new query on the table queues behind it: API traffic stalls. The MIGRATE=1 connection has no max_execution_time, and on main nothing sets lock_wait_timeout (the integration branch sets 5 s). Check information_schema.innodb_trx just before, and if the step sits for more than a few seconds, cancel it rather than let the queue build; it's one statement, so it's safe to rerun.
  • Before running it, confirm nothing older than D2 is running: production, every beta destination and staging on D2's image, no lingering old containers or console sessions (bin/kamal app details -d <dest>). A D1 process left running is the failure in the caution above.
  • Undo by version: bin/rails db:migrate:down:primary VERSION=20261017130000. It restores empty columns, not the tokens. Never db:rollback (see the integration doc). Undoing D2's 20261017120000 is a no-op either way.
  • After this, neither backfill should be redone by hand: D2's rerun is a no-op without the column, and D1's (20261015120100) would fail looking for it.
  • Self-hosted installs that upgrade with Kamal from a version older than D2 migrate in the new container while the old one still serves, so until the cutover (seconds, once the new container passes its health check) the old one fails as in the caution above. That's bounded by the cutover, and D2 and D3 reach main separately, so an install that follows main passes through D2 first. Stopping the old container before migrating would close the window and cost every upgrade its zero-downtime cutover, so it isn't done.
  • Plaintext outside the table: backups and binlogs taken before this migration still hold it, until their retention runs out. On SQLite, remove_column rebuilds the table, but the old pages stay in the file as free pages until a VACUUM.
  • Staging has its own database, so it migrates separately, by the same gate: staging on D2 first.

Verified

On thelio at the dcb914ceb head; this PR's diff over #3202 is byte-identical since. Gated on exit status (CI's commands: bin/rails db:setup then bin/rails test, against a dedicated mysql:8.4 container):

Suite Result
SaaS (MySQL 8.4) 2285 tests, 0 failures, 0 errors
OSS (SQLite) 2117 tests, 0 failures, 0 errors
OSS (MySQL 8.4, DATABASE_ADAPTER=mysql) 2117 tests, 0 failures, 0 errors

Migration round trip, SQLite and a dedicated MySQL 8.4 container: continuing #3202's (#3192's schema, a pre-D1 plaintext-only token, a D1 token, an OAuth grant): up through 20261017130000, the column is gone, every index is intact, both personal tokens and a newly created one authenticate, no updated_at moved, and the dump matches db/schema.rb / db/schema_sqlite.rb exactly. Then down newest first: the column comes back nullable and empty, and the dump differs from #3192's only in that the restored column is appended (and, on SQLite, declared as the original CreateIdentityAccessTokens declared it, without the dump's limit: 255). Then up again and redo of 20261017120000 with the column gone (a no-op): checks pass, dump matches.

#3191 shipped-client pins need the same two-line .token → .raw_token change #3202 describes. With it, they passed against this branch before the rebase onto dcb914ceb (OSS 19 tests, SaaS 28 with the auditor pins), and against #3202's current head. This branch's own diff over #3202 is unchanged since. The token string and the create JSON shape are unchanged.

Not done here, on purpose

  • No plaintext is carried anywhere: there's nothing to keep once every token is digested.
  • No new third party or personal-data flow.

@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T18:45:57.424528Z a420fae Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The destructive migration requires human confirmation that D2 replaced every older process and the shared-production manual migration gates are satisfied.

No open issues 🎉

What changed in this PR

Completes D3 of digest-only token storage by removing the remaining plaintext personal access token column.

Changes:

  • Adds the column-drop migration and updates both database schemas.
  • Removes the model’s column-ignore setting and updates tests for the final schema.
  • Documents completion of digest-only storage.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
test/​models/​identity/​access_token_test.rb Asserts the plaintext column is absent.
test/​migrations/​backfill_token_digests_test.rb Removes tests requiring the dropped column.
test/​migrations/​backfill_remaining_token_digests_test.rb Removes tests requiring the dropped column.
docs/​oauth/​spec_conformance.md Records completion of D3.
docs/​docker-deployment.md Documents digest-only storage.
db/​schema.rb Removes the MySQL plaintext column.
db/​schema_sqlite.rb Removes the SQLite plaintext column.
db/​migrate/​20261016130000_drop_identity_access_token_plaintext.rb Drops plaintext storage; rollback restores an empty column.
app/​models/​identity/​access_token.rb Removes the obsolete column-ignore setting.

Review effort: Balanced

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ee24b6f171

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread db/migrate/20261016130000_drop_identity_access_token_plaintext.rb Outdated
Comment thread docs/docker-deployment.md
Comment thread db/migrate/20261016130000_drop_identity_access_token_plaintext.rb Outdated
@jeremy
jeremy force-pushed the oauth-token-digests-d3 branch from ee24b6f to f339274 Compare October 17, 2026 07:43
@jeremy
jeremy force-pushed the oauth-token-digests-d2 branch from c8786a9 to 135b057 Compare October 17, 2026 07:43
@jeremy
jeremy requested a balanced review from Copilot October 17, 2026 08:16
@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The destructive migration requires human confirmation that D2 is deployed everywhere and the manual migration gates are satisfied.

No open issues 🎉

Review effort: Balanced

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: bd3431ce1c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jeremy
jeremy force-pushed the oauth-token-digests-d2 branch from 135b057 to b6675c2 Compare October 17, 2026 08:31
@jeremy
jeremy force-pushed the oauth-token-digests-d3 branch 2 times, most recently from 715995f to a7b7748 Compare October 17, 2026 08:40
@jeremy
jeremy requested a balanced review from Copilot October 17, 2026 08:44
@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: a7b77486fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The destructive migration requires human confirmation that D2 is deployed everywhere and the manual migration gates are satisfied.

No open issues 🎉

Review effort: Balanced

@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

Review threads: 3 resolved (2 fixed, 1 declined with the reasoning in the thread).
Fixed: Codex, rebuild the table so plaintext leaves the pages: MySQL now drops with ALGORITHM=INPLACE, LOCK=NONE, verified as a rebuild on 8.4.11. Codex, qualify the digest-only docs claim: docker-deployment.md names the refresh retry's encrypted pair.
Declined: Codex, keep the column through self-hosted rolling upgrades: bounded by the Kamal cutover, and D2's ignored_columns reaches main first; recorded in the body's runbook notes.
Open for a decision: the merge gate in the PR body (D2 on production, every beta and staging, no older process left), which Copilot also flags; Jeremy's call when D2 is deployed.

@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 10a6f57398

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A human must confirm every process runs D2 or later and the manual migration gates are satisfied before the destructive migration proceeds.

No open issues 🎉

Review effort: Balanced

@jeremy

jeremy commented Oct 17, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 5c04610946

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The destructive migration requires human confirmation that D2 runs everywhere and the manual migration prerequisites are satisfied.

No open issues 🎉

Review effort: Balanced

Step D3 of hashing tokens at rest. Tokens have been found only by digest,
and the column ignored, since step D2, so this drops identity_access_tokens.token
and the plaintext it still held for tokens issued before digests.

Undoing it restores an empty, nullable column: the tokens can't be
recovered from their digests.
…refresh retry's encrypted pair

An INSTANT drop only removes the column from the data dictionary, so a
token's plaintext would stay in its row until the row is rewritten. The
deployment docs now name the one credential kept other than as a digest:
a refresh's successor pair, encrypted for the 60-second retry window.
identity_access_tokens.refresh_token stayed past
DropPlaintextOauthCredentials, so that production code from before the
OAuth stack never had a cached column dropped under it. Every process
that runs this migration's prerequisite code ignores it, and
BackfillTokenDigests emptied it, so it goes in the same table rebuild as
token, with its unique index.

A migration test caches the table's columns as the code before this
migration does, runs the migration, and creates a token with them.
@jeremy
jeremy force-pushed the oauth-token-digests-d2 branch from 3ac03cc to 457a45e Compare October 18, 2026 05:56
@jeremy
jeremy force-pushed the oauth-token-digests-d3 branch from 5c04610 to a420fae Compare October 18, 2026 05:56
@jeremy
jeremy requested a balanced review from Copilot October 18, 2026 06:00
@jeremy

jeremy commented Oct 18, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human confirmation is required that D2 runs everywhere and the destructive migration follows the manual integration gates.

No open issues 🎉

Review effort: Balanced

@jeremy

jeremy commented Oct 18, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@jeremy

jeremy commented Oct 18, 2026

Copy link
Copy Markdown
Member Author

Restacked onto #3192 617edb87b ("Keep identity_access_tokens.refresh_token until it can go with token"), now at a420fae04. This PR also gains "Drop the plaintext refresh token column beside token": the migration now drops identity_access_tokens.refresh_token and its index in the same INPLACE rebuild as token, and a migration test creates a token with columns cached as D2's code caches them. On thelio: rubocop clean; SaaS (MySQL), OSS (SQLite) and OSS (MySQL) suites pass, 0 failures, 0 errors. CI is green and Copilot reviewed this head with no open issues. Codex didn't review it: two requests returned "You have reached your Codex usage limits for code reviews". Re-request it once the limit resets.

@jeremy

jeremy commented Oct 19, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: a420fae044

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants