Repository navigation
Add stable decomposition clarification - #957
Merged
fdwr merged 2 commits intoOct 10, 2026
Merged
Conversation
fdwr
approved these changes
Sep 24, 2026
fdwr
left a comment
fdwr
left a comment
Collaborator
There was a problem hiding this comment.
馃憤 Minor thought, else LGTM.
| const sumExpX = | ||
| builder.reduceSum(builder.exp(builder.sub(input, maxX)), options); | ||
| return builder.add( | ||
| builder.log(sumExpX), builder.reshape(maxX, sumExpX.shape)); |
Collaborator
There was a problem hiding this comment.
I feel like adding some comment explaining how a "reduce log sum" became a "reduceMax" would be enlightening. e.g.
// This is more numerically stable than a direct decomposition:
// return builder.log(builder.reduceSum(builder.exp(input), options));
huningxin
approved these changes
Oct 8, 2026
huningxin
left a comment
huningxin
left a comment
Contributor
There was a problem hiding this comment.
LGTM % other reviewers' comments
github-actions Bot
added a commit
that referenced
this pull request
Oct 10, 2026
SHA: a89427e Reason: push, by fdwr Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the WG resolution of 2026-09-10 to "update the spec to use numerically more stable reduceLogSumExp decomposition" (https://www.w3.org/2026/09/10-webmachinelearning-minutes.html#214b).
Changes
reduceLogSumExp:log(reduceSum(exp(x - max))) + max.softplus:max(0, x) + log(1 + exp(-|x|))tanh:sign(x) * (1 - e) / (1 + e)withe = exp(-2|x|).EMULATEDnote text: adds a note to specify that reference decompositions aren't necessarily reference implementations.Justification
The previous forms return Infinity or NaN for inputs that the WPT conformance tests expect to succeed, more details in #947.
reduce_log_sum_exp.https.any.jshas the explicit 100.0 overflow and -100.0 underflow cases, and thefloat16tanhcases contain inputs of 5.586 and 5.957, beyond the point whereexp(2x)overflowsbinary16, so the oldtanhsample cannot pass them.reduceLogSumExp([100])float32reduceLogSumExp([-100])float16softplus(100)float32softplus(12)float16tanh(45)float32tanh(5.957)float16Inputs well inside range are unchanged to within one ULP. The stable forms match what frameworks and backends already do. rustnn/rustnn#213 uses the same max-shift for the CoreML lowering and passes the
float32andfloat16WPT overflow and underflow cases.Other decompositions
All 29 decomposition blocks were reviewed.
softmaxalready subtracts the max,sigmoidsaturates to 0 and 1 without NaN,eluexponentiatesmin(0, x), and the three normalizations use the two-pass variance.Preview | Diff