Skip to content

feat: port crypto to the PSA API for Mbed TLS 4 / ESP-IDF 6 - #91

Open
davidcoulson wants to merge 2 commits into
yoziru:mainfrom
davidcoulson:pr/psa-crypto
Open

davidcoulson wants to merge 2 commits into
yoziru:mainfrom
davidcoulson:pr/psa-crypto

Conversation

@davidcoulson

@davidcoulson davidcoulson commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

ESP-IDF 6 ships Mbed TLS 4 (TF-PSA-Crypto), which removed the legacy ECDH/ECP, GCM, SHA-1 and CTR_DRBG APIs this library uses, so it no longer builds there (mbedtls/ecdh.h: No such file or directory). This moves the crypto onto the PSA Crypto API, which Mbed TLS 3.6 (ESP-IDF 5.3+) and 4.x both provide, so one source tree builds on both.

  • The P-256 key pair is a volatile PSA key. ECDH uses psa_raw_key_agreement (same X-coordinate output as before), and the session key is still SHA1(shared)[:16].
  • AES-128-GCM, SHA-1, SHA-256, HMAC-SHA256 and random bytes go through PSA, via new CryptoUtils helpers (aes_gcm_encrypt/aes_gcm_decrypt, sha256_hash). Peer::encrypt/decrypt_response and get_vin_advertisement_name use them.
  • Private key storage is unchanged. The same SEC1 EC PRIVATE KEY PEM is read and written through mbedtls_pk (mbedtls_pk_get_psa_attributes / mbedtls_pk_import_into_psa / mbedtls_pk_copy_from_psa). Keys saved by earlier releases still load, and newly written PEM is byte-identical to what 3.x wrote (there's a test for this), so upgrading or downgrading does not lose pairing.
  • mbedtls_pk_parse_key lost its RNG arguments in 4.x. That is the only #ifdef, keyed on TF_PSA_CRYPTO_VERSION_MAJOR.
  • The GCM error paths no longer leak the mbedtls_gcm_context.
  • The CryptoContext getters returning raw mbedtls_*_context pointers are removed. Nothing in the library or tests used them.
  • idf_component.yml: the minimum ESP-IDF is now 5.3, the first release with Mbed TLS 3.6 (the PSA import/export helpers above).

Second commit: Mbed TLS 4.x in CI

The TF_PSA_CRYPTO_VERSION_MAJOR branch was previously only compiled on hardware. The host build now takes TESLABLE_MBEDTLS_REPO/TESLABLE_MBEDTLS_TAG as cache variables and links tfpsacrypto when that target exists, and CI adds a "Build and Test (Mbed TLS 4.x)" leg against upstream v4.1.1. The tests that compared against the legacy mbedtls_gcm/sha256/md APIs (private headers in 4.x) now check known answers computed with the 3.6 implementation instead.

Verification

  • 332/332 host tests pass against both Mbed TLS 3.6.6 (espressif fork) and upstream Mbed TLS 4.1.1, including a new test_psa_crypto.cpp. It checks:
    • the PSA paths against the protocol spec vectors (ECDH session key, session-info key)
    • the PSA paths against known answers from the legacy 3.6 implementation (GCM ciphertext and tag, HMAC, SHA-256), and PEM bytes against mbedtls_pk on both majors
    • rejection of a tampered tag, AAD or ciphertext
    • key generation with ECDH agreement, save and reload, move semantics, and P-384 rejection
  • clang-format passes. No new clang-tidy findings in the touched files.
  • The library builds inside ESPHome 2026.9.1 for esp32c5 on ESP-IDF 5.5.1 (Mbed TLS 3.6) and ESP-IDF 6.1.0 (Mbed TLS 4.1). (The esp32c5 target itself is a separate PR.)
  • On a car: ESP32-C5, ESP-IDF 6.1.0 (Mbed TLS 4.1), Model S, paired with a key generated by this code. The VCSEC session handshake and polls work, and infotainment data (battery, range, charge state, TPMS, temperatures) decrypts and authenticates. The battery reading matches the Tesla Fleet API.

🤖 Generated with Claude Code

@yoziru

yoziru commented Oct 10, 2026

Copy link
Copy Markdown
Owner

Thanks for this — triaging the open PRs surfaced that your second commit is a critical fix, so I've extracted it into #102 and it will land on its own:

  • Verified against the vendored mbedtls 3.6.6 source that mbedtls_gcm_finish() in decrypt mode only writes the computed tag (output buffer) and returns 0 — the received tag was never compared, so response authenticity was never checked on the legacy path.
  • fix: verify AES-GCM response tags and use the response counter in the AAD #102 therefore adds an explicit constant-time tag comparison (which psa_aead_decrypt does for you), plus the response-counter AAD fix and rejection tests. The counter-override shape matches your e398afa (same decrypt_response parameter position), so your rebase should mostly be context.

After #102 lands, #91 needs a rebase onto main — main moved since Oct 7 (per-domain request hash in Client::get_last_request_hash, TX buffer sizing in Vehicle, retry/backoff changes, dead-code removal in MessageProcessor, test consolidation). Would you like to rebase it, or should I resolve the conflicts in a follow-up branch on top of your commits? There is real value in the PSA port (ESP-IDF 6 / Mbed TLS 4 support) and I'd like to get it merged.

yoziru added a commit that referenced this pull request Oct 10, 2026
… AAD (#102)

- Peer::decrypt_response now compares the computed GCM tag with the
  received one (constant-time). mbedtls_gcm_finish only computes the tag
  in decrypt mode, so response authenticity was never actually checked.
- The response AD is built with the counter the vehicle sends in
  AES_GCM_ResponseData instead of the peer's own request counter, which
  differs after a retry rebuilds the request. decrypt_response takes the
  response counter and construct_ad_buffer accepts a counter override.
- Tests: tampered tag and mismatched counter are rejected end to end;
  the VCSEC round trip signs with a differing counter and asserts
  rejection on mismatch; the unauthenticated mock decryption is now
  asserted to fail instead of being accepted.

The counter-override approach and test structure are adapted from
PR #91 by David Coulson, which found this while porting to PSA.

Co-authored-by: David Coulson <23066302+davidcoulson@users.noreply.github.com>

@yoziru yoziru left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Followed up on the earlier comment: #102 is on main (v6.3.0), so rebasing is now the only blocker for CI.

  1. Rebase onto main and drop e398afa. The red test is the stale copy of ResponseDecryption: at this head it asserts decrypt_result == OK || decrypt_result != OK (tests/test_protocol_handshake.cpp:219), so it cannot fail. Main's version asserts E_ERROR_DECRYPT for that invalid vector and should survive the conflict. I checked both files at this head and on main.

  2. The 4.x claim is not exercised anywhere. The host build fetches mbedtls-3.6.6-idf (CMakeLists.txt:147) and CI has no other Mbed TLS leg, so the #ifdef TF_PSA_CRYPTO_VERSION_MAJOR branch in src/crypto_context.cpp:22 is never compiled. Either add a 4.x matrix entry or scope the PR/release note to "3.6+, 4.x hardware-verified only". Otherwise this will rot on the next bump.

Separate, pre-existing: Peer::generate_nonce() returns void and encrypt() continues after a failed RNG (src/peer.cpp:116-125, 501). Not introduced here, but this PR swaps the RNG backend, so a follow-up is worth it.

The PSA translation itself looks sound: the new tests compare against the real legacy implementation (not mocks) and cover tag/AAD/ciphertext tampering and PEM bytes. I did not run the suite; CI is the evidence for 3.6.

davidcoulson and others added 2 commits November 6, 2026 08:50
ESP-IDF 6 ships Mbed TLS 4 (TF-PSA-Crypto), which removed the legacy
ECDH, ECP, GCM, SHA-1 and CTR_DRBG APIs this library used, so it no
longer compiled. Use PSA Crypto instead, which Mbed TLS 3.6 (ESP-IDF
5.3+) and 4.x both provide:

- P-256 key pair is a volatile PSA key; ECDH via psa_raw_key_agreement
- AES-128-GCM, SHA-1, SHA-256, HMAC-SHA256 and random via PSA
- private key storage is unchanged: the same SEC1 PEM is read and
  written through mbedtls_pk, so keys saved by earlier releases still
  load and newly saved keys are byte-identical
- mbedtls_pk_parse_key lost its RNG arguments in 4.x; gated on
  TF_PSA_CRYPTO_VERSION_MAJOR
- GCM error paths no longer leak the context

Minimum ESP-IDF is now 5.3 (first with Mbed TLS 3.6).

Tests: new test_psa_crypto checks the PSA paths against the protocol
spec vectors and against the legacy 3.6 APIs (GCM ciphertext and tag,
HMAC, SHA-256, PEM bytes), plus tamper rejection, key generation,
save/reload, move semantics and curve validation. Host suite 270/270.
Built with ESPHome for esp32c5 on ESP-IDF 5.5.1 and 6.1.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The TF_PSA_CRYPTO_VERSION_MAJOR branch in crypto_context.cpp was only
ever compiled on hardware. The host build now takes the Mbed TLS
repository and tag as cache variables, links tfpsacrypto when that
target exists (4.x moved the crypto library out of mbedcrypto), and
CI adds a test leg against upstream Mbed TLS v4.1.1.

The tests that used the legacy mbedtls_gcm/sha256/md APIs as a
reference implementation could not build on 4.x, where those headers
are private. They now check the PSA output against known answers
computed with the 3.6 implementation, and the PEM compatibility checks
call mbedtls_pk_parse_key with the right arity for each major.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@davidcoulson

Copy link
Copy Markdown
Contributor Author

Both addressed in the push.

  1. Rebased onto main and dropped e398afa. The branch is now the PSA port on top of fix: verify AES-GCM response tags and use the response counter in the AAD #102, with main's ResponseDecryption test (asserting E_ERROR_DECRYPT) kept as is.

  2. The 4.x branch is now exercised: CMakeLists.txt takes TESLABLE_MBEDTLS_REPO/TESLABLE_MBEDTLS_TAG as cache variables and links tfpsacrypto when that target exists, and there is a new "Build and Test (Mbed TLS 4.x)" matrix leg against upstream v4.1.1. The tests that used the legacy mbedtls_gcm/sha256/md APIs as a reference (private headers in 4.x) now check known answers computed with the 3.6 implementation, and the PEM checks call mbedtls_pk_parse_key with the right arity per major. Locally 332/332 pass on both 3.6.6 and 4.1.1; the library sources needed no change for 4.x.

Agreed on Peer::generate_nonce(); that is pre-existing and worth its own PR rather than widening this one.

@yoziru yoziru left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both earlier points check out: rebased onto main, and the 4.x leg runs the suite. Two new blockers before merge:

  1. build4/ is committed: 118 files / 27 MB of build output and binaries. Remove it and ignore the directory.
  2. The ECDH path logs the session key at VERBOSE (inline); it leaks key material and fails the Python bindings leg here.

Ran the host suite at head: 332/332 on 3.6.6; the bindings test fails as above. 4.x is CI-only here.

Comment thread src/crypto_context.cpp
break;
}
LOG_DEBUG("Tesla ECDH completed successfully");
LOG_VERBOSE("Session key: %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x",

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this log: it writes the derived session key to the log callback. test_raw_message_and_log_callbacks in bindings/python/tests/test_offline_flow.py asserts it never appears and fails at this head. main removed the same line in v6.0.0.

Comment thread tests/test_psa_crypto.cpp

EXPECT_STREQ(reinterpret_cast<char *>(pem_out), reinterpret_cast<char *>(legacy_pem));

uint8_t public_key[65];

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This compares PSA output to the same Mbed TLS version, so it passes even if 4.x changes the PEM format. A fixed 3.6 golden (with the trailing newline) would prove the downgrade path.

@yoziru

yoziru commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Follow-up from the correctness pass, since the first review is already posted.

The PSA port makes key storage process-global. The standalone build leaves MBEDTLS_THREADING_C off (mbedtls_config.h:3778) and caps volatile slots at 32 (crypto_extra.h:32), so two Clients on two threads can race in psa_import_key/psa_destroy_key (one can overwrite or destroy the other's live key), and more than 31 live keyed clients fail to load. ESP-IDF enables threading, so this is a host/multi-client concern, not the single-vehicle path; the old per-CryptoContext state was independent. Options: enable threading for the standalone build, serialize PSA calls, or document single-threaded use. I did not run a concurrency test; this is from the config and code.

Two smaller items: the GCM helpers import the key before constructing the std::vector, so if that allocation throws the slot leaks — allocate first. And get_vin_advertisement_name now returns "" if psa_crypto_init fails, where the old mbedtls_sha1 could not; worth a note with the PSA requirement.

Also worth a release note: removing the CryptoContext getters and changing CryptoUtils::generate_random_bytes's signature break downstream API; the PR body covers the getters but not the signature.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants