Skip to content

feat(ui): recipes, workouts, plans and calculators as interactive cards - #6086

Open
tylerbea wants to merge 34 commits into
amd:mainfrom
tylerbea:claude/gaia-recipe-artifact-ebf171
Open

tylerbea wants to merge 34 commits into
amd:mainfrom
tylerbea:claude/gaia-recipe-artifact-ebf171

Conversation

@tylerbea

@tylerbea tylerbea commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Asking GAIA for a recipe, a workout, a repair guide, a packing list, a trip plan, a tip split or a loan payment used to return a wall of text, with numbers the small local model often got wrong. Now the answer arrives as an interactive card in the Agent UI. Recipes rescale with servings, switch US/metric and run step timers in a full-screen Start cooking mode. Calculators compute in the app and update as you edit. A Try row on the new chat screen and a one-time tip on the first card show people these exist. Closes #6085.

Every card type was exercised live against Gemma-4-E4B. That run caught and fixed amounts the model writes last ("Olive oil (1 tablespoon)"), "165°F/74°C" pairs, its own step numbering, re-typed recipes under the card, a double-subtracted down payment and currency carried over from an earlier turn.

  • Card rendering: the SSE handler now names the card itself when it lifts one, so in-process turns draw cards and a failed call no longer draws an "Invalid payload" card. Reloaded chats rebuild cards from the saved tool steps.
  • TUI: draws both cards too: every line of a guide, and a calculator's results computed in the client.
  • Evals: new tool docstrings change the schema sent to Lemonade, so re-run gaia_tool_selection, tool_selection, gaia_core (itinerary and workout turns) and gaia_voice (the packing-list scenario expects a bulleted reply) on the next RC eval. No local run was possible here.

Before and after

Before After
Recipe request answered as plain text Recipe card with servings, units, timers and Start cooking
New chat with no examples New chat with the Try row of examples

Start cooking (full screen, one step at a time) · Loan calculator · Try row on a phone

Start cooking full-screen step with timer Loan calculator card in euros New chat Try row on a phone

Test plan

  • npx vitest run in src/gaia/apps/webui: 2099 passed. npx tsc --noEmit -p . is clean.
  • python -m pytest tests/unit/test_guide_tools.py tests/unit/test_calculator_tools.py tests/unit/test_sse_narration.py hub/agents/gaia/python/tests/test_capability_matrix.py: 187 passed.
  • python util/lint.py --black --isort --flake8 --tool-descriptions --agents passes.
  • Live with Gemma-4-E4B in the Agent UI: recipe, workout, how-to, checklist, itinerary, tip and loan cards all draw. Servings, units, timers, guided mode, ticks and calculator edits work, and the cards survive a reload. Checked at desktop and 375 px, in light and dark.
  • Reviewer: on a new chat, tap Try → Recipe, send it, then press Start cooking.

Recipes and other step-by-step answers now render as an interactive card:
servings that rescale amounts with friendly fractions and whole eggs,
US/metric units, timer chips on every duration, copy/share/print, and a
full-screen guided view with a big countdown.
… cards

A recipe or workout answer from a small local model was a long wall of
list text. The flagship now has four tools (show_recipe, show_workout,
show_how_to, show_checklist) that send the answer as a `guide` render
card, so the reply only adds a tip or two. A bad argument comes back as
{"ok": false, "error": ...} naming the fix, so the model can call again.

The flagship sidecar's /query stream never set tool_result.render, so no
card tool could ever draw over HTTP. Its translator now gets the render
map.

- GuideToolsMixin, registered in KNOWN_TOOLS as `guide`, wired into
  GaiaAgent and a `guide` bundle in the full profile's tool loader.
- tools_count 95 -> 99; capability matrix regenerated.
- `guide` row in the SSE contract §4.3; docs and changelogs updated.
Cards drew only through the sidecar relay, vanished on reload, and a small
model's real output broke scaling and temperatures.

- The SSE handler now names the card when it lifts one, so in-process turns
  draw it and a failed call never draws an "Invalid payload" card. The
  translator reads that instead of its own tool map.
- A reloaded chat rebuilds its cards from the saved tool steps, and the
  post-turn refetch keeps them instead of wiping them.
- Ingredients written "Olive oil (1 tablespoon)" now scale.
- "165°F/74°C" pairs no longer convert into duplicates, and doneness
  temperatures stay exact instead of snapping to 5 degrees.
- The tool result tells the model the card is on screen, so it stops
  re-typing the recipe under it.
…de cards

Gemma wrote "Step 1: Turn off the water", so the card read "1  Step 1: ...".
Trip plans now come back as a day-by-day card, and tip, bill-split and loan
questions open a calculator card that does the math, so a small model's
arithmetic never reaches the answer. The model only sends starting numbers.

- show_itinerary: a fifth guide type (days as steps, packing as items).
- CalculatorToolsMixin (KNOWN_TOOLS["calculator"]): show_tip_calculator and
  show_loan_calculator validate ranges and return a `calculator` render card;
  a bad number comes back as {"ok": false, "error": ...} naming the fix.
- Wired into GaiaAgent; `calculator` bundle; tools_count 99 -> 102; capability
  matrix regenerated; `calculator` row in the SSE contract §4.3.
- show_recipe's ingredients description trimmed back under the per-call
  token budget, which it had gone over.
- Itineraries draw as a timeline: one dot per day, titled by its "Day 1 -
  Lisbon" lead, with a checkable packing list. splitTitle accepts "Day N"
  leads only for this type; other types still treat a lead ending in a digit
  as body text.
- The calculator card computes tip, total and per-person share, or the
  monthly payment (standard amortization, 0% handled), total paid and
  interest. Number boxes and sliders recompute live, money is formatted in
  the person's locale and currency, and copy uses the guide's plain-text
  pattern. A bad payload shows the unsupported card.
Live with Gemma, a mortgage on a 300,000 EUR flat with 60,000 down came out
for a 180,000 loan: the model subtracted the down payment from `amount` and
the card subtracted it again. A dollar bill split right after showed euros,
since the card guessed currency from the browser region and the model then
carried EUR over from the earlier turn.

- `show_loan_calculator` takes `price` and says never to subtract the down
  payment.
- Both calculators take `currency` (ISO code, "$" is USD) and the card
  formats in it instead of guessing from the locale.
…ide cards

From a review of the branch:

- A finished turn kept only the cards rebuilt from saved steps, so a live
  confirmation or question card beside a recipe card vanished. The live list
  wins again; saved steps still restore cards on reload.
- Loan terms start at 1 year on both sides; a fractional term made a stuck
  slider and, under a month, a zero payment.
- Saved checkmarks that fail to read or write now warn instead of passing
  silently.
- The Agent UI guide says what each card does; a plan doc no longer names
  the removed translator map.
… can do

Nothing in the app said recipes, workouts or calculators come back as cards.

- A "Try" row on the new chat screen fills the box with one example per card
  type, to edit or send.
- The first recipe, workout or how-to card carries a short tip naming its
  hidden controls. Dismissing it, or using one of those controls, hides it on
  every card for good.
…ecipe-artifact-ebf171

# Conflicts:
#	hub/agents/chat/python/CHANGELOG.md
#	src/gaia/apps/webui/src/components/ChatView.tsx
#	src/gaia/apps/webui/src/components/NewChat.tsx
A relayed turn showed no label for the seven new card tools, which the
sidecar contract test requires for every registered tool.
The chip renders "Memory off" one commit before its effect clears the draft,
so a loaded CI runner read the flag in between and failed.
@github-actions github-actions Bot added documentation Documentation changes devops DevOps/infrastructure changes tests Test changes electron Electron app changes agents agent::email Email agent changes labels Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Approve with suggestions

Recipes, workouts, how-tos, checklists, itineraries and the tip and loan calculators now show up as interactive cards. The calculators do the math in the app, so the small model's arithmetic no longer reaches the answer. Each tool checks its arguments and returns an error that says how to fix the call. Cards rebuild from the saved steps after a reload, and a failed call no longer draws an "Invalid payload" card. Both have tests. The PR includes before/after screenshots and names the eval categories to watch on the next RC.

🟡 One changelog line is now wrong for integrators. It says the stdio transport "still never sends" the card type. This PR makes stdio send it too. Stdio uses the same output handler and translator as HTTP, and the handler now attaches the card type itself. Please correct that sentence so integrators on stdio know to expect render.

🔍 Technical details
  • hub/agents/gaia/npm/CHANGELOG.md, "For integrators" bullet: "The stdio transport still never sends it."
  • stdio.py:1396-1439 builds an SSEOutputHandler and feeds its events through CanonicalTranslator.translate. sse_handler.py:985 now sets event["render"] whenever it lifts a card, and sse_translation.py:431 forwards it. So both transports carry render for every tool in _RENDER_TOOL_TO_LANG, not only the new ones.
  • Suggested wording: "It now carries the card type on both the HTTP and stdio transports (guide or calculator for the tools above)."
  • Removing the render_tool_map kwarg is safe. On main only one test passed it, and that test is updated here.

The changelog and the contract said stdio never sends render; the handler now
names the card itself, so stdio carries it as HTTP does.
Main added three audio-generation tools while this branch added seven card
tools; the counts become 105 tools in 28 bundles, and the gaia guide now
counts the audio tools it had missed.
The manifest, registration and capability-matrix test all agree on 105
(98 plus five guide and two calculator tools); the changelog said 102.
@kovtcharov-amd

Copy link
Copy Markdown
Collaborator

Thanks @tylerbea — this is a lovely piece of work. Moving the arithmetic out of the model and into the card is exactly the right call for a small local model, and the live Gemma-4-E4B pass clearly paid off (the "Olive oil (1 tablespoon)" and double-subtracted down payment fixes are the kind of thing only a real run finds).

One gap before this can merge: outside the Agent UI, a recipe or calculator answer now loses its content. The tools tell the model "the card is on the user's screen, don't repeat the list / don't state the results", but the TUI has no guide or calculator renderer, so a TUI user asking for a banana bread recipe gets a one-line tip plus a 12-line truncated JSON dump, and a loan question gets no numbers at all. Today the same question gets a full text answer, so this would land as a regression for TUI users once it's on main.

Either of these would close it: a TUI renderer for the two keys (a readable list for guide, the computed results for calculator), or keeping the content in the reply for clients that can't draw the card. The maintainers may pick that up as a companion PR — no need to rework the Agent UI side, which checks out well.

I merged main in and fixed one changelog line (tool count is 105, not 102). Everything else passes locally: vitest 2151/2151, tsc clean, 3755 pytest passing across the touched agent/UI suites, lint clean. Timing-wise this is queued for right after v0.25.3 ships, for v0.26.0.

🔍 Technical details
  • TUI: tui/internal/ui/cards/cards.go RenderDeduped sends guide/calculator to renderUnsupported, which caps the raw dump at rawDumpLines = 12. The sidecar now sets tool_result.render on both HTTP and stdio, so the TUI receives these keys.
  • Eval watch beyond gaia_tool_selection / tool_selection: gaia_voice/voice_t3_bullets_new_session asks for a camping packing list and fails if the reply isn't a bulleted list, which show_checklist + "do not repeat the list" will likely trip. gaia_core (itinerary and workout turns) is worth a re-run too.
  • Verified: card content renders as React text nodes only; print path escapes & < > " '; calculators take typed numbers and never evaluate expressions; the guided view closes on Escape / X / Finish and uses the shared focus-trap and Escape-layer hooks; invalid payloads fall back to the visible "Invalid … payload" card; ChatView.tsx is 97,276 bytes LF.

…tifact-ebf171

# Conflicts:
#	hub/agents/gaia/npm/CHANGELOG.md
Main added three email tools and an email_threads bundle while this branch
added seven card tools; the counts become 108 tools in 29 bundles.
…tifact-ebf171

# Conflicts:
#	hub/agents/chat/python/CHANGELOG.md
…tifact-ebf171

# Conflicts:
#	hub/agents/chat/python/CHANGELOG.md
…tifact-ebf171

# Conflicts:
#	hub/agents/chat/python/CHANGELOG.md
…tifact-ebf171

# Conflicts:
#	hub/agents/gaia/npm/CHANGELOG.md
Main moved the tool bundles into gaia_agent; the guide and calculator bundles
move with them, and the chat package's changelog no longer claims them.
Main started attaching diff cards in the handler too, so the handler names
every card and the translator's tool-map fallback stays gone. The guide counts
108 tools on a larger model; Gemma keeps the card tools.
…tifact-ebf171

# Conflicts:
#	docs/guides/agent-ui.mdx
#	docs/spec/agent-ui-query-sse-contract.md
#	src/gaia/apps/webui/src/components/ChatView.tsx
#	src/gaia/apps/webui/src/utils/chatStream.ts
tylerbea and others added 7 commits November 3, 2026 19:03
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tifact-ebf171

# Conflicts:
#	src/gaia/apps/webui/src/components/__tests__/PrivateChip.test.tsx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main's new check fails any screen that writes English straight into JSX. The
recipe, workout and calculator card buttons, timers and guided-view labels
now go through useT, with Arabic text for each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tifact-ebf171

# Conflicts:
#	docs/guides/gaia.mdx
#	hub/agents/gaia/CAPABILITY_MATRIX.md
#	hub/agents/gaia/npm/SKILL.md
#	hub/agents/gaia/python/gaia-agent.yaml
#	hub/agents/gaia/python/gaia_agent/__init__.py
#	hub/agents/gaia/python/tests/test_capability_matrix.py
…yout

A recipe or loan answer in the TUI showed a truncated JSON dump, because
the tools tell the model the card carries the content. The TUI now draws
every line of a guide and computes a calculator's results with the same
formulas as the Agent UI; a bad payload shows the invalid-card fallback.

In an Arabic chat, English ingredient and step lines rendered reordered
("cups flour 2") with bullets on the wrong side. Each line now orders its
own text and lines up with the card's side, and an RTL visual test covers
both cards in Arabic and English.
@github-actions github-actions Bot added the tui Go terminal UI (gaia-tui) label Nov 10, 2026
@kovtcharov-amd

Copy link
Copy Markdown
Collaborator

Thanks @tylerbea, this is ready for a final look. I merged main in and closed the TUI gap from the earlier review: a recipe or loan answer in the TUI now draws as a readable card instead of a cut-off JSON dump. I also fixed how the cards read in an Arabic chat, where English ingredient lines came out reordered ("cups flour 2") with bullets on the wrong side.

One thing left for a follow-up: in Arabic, the card's labels (Ingredients, Start cooking, servings, the calculator's field and result names, the first-card tip) still show in English.

🔍 Technical details
  • TUI: new guide and calculator renderers in tui/internal/ui/cards/. A guide draws every line with no 22-row cap, because the tool tells the model not to repeat the list. The calculator recomputes tip and amortization with the same formulas as calculatorDoc.ts. A bad payload shows the existing "Invalid … payload" card. Covered by guide_test.go and calculator_test.go, including escape-sequence and range checks.
  • RTL: guide.css uses logical properties for the bullet, the item padding and the Start button. Model-text lines use unicode-bidi: plaintext with alignment set per direction, which is the pattern the sidebar uses. A new rtl.visual.ts test renders both cards in ar and en and passes the layout audit.
  • Earlier "Test Gaia Agent" failure: the fixed prefix was 15,144 tokens against a 15,000 budget. After merging main it measures 14,883 locally, so CI on the new head should confirm it.
  • Local checks: vitest card suites, tsc --noEmit, the Electron byte-cap suite (63/63), npm run test:rtl (442 + 2 new), PR pytest suites (299), the prefix-stability suite (36), and go test for cards and chat.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent::email Email agent changes agents devops DevOps/infrastructure changes documentation Documentation changes electron Electron app changes tests Test changes tui Go terminal UI (gaia-tui)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(ui): show recipes, workouts, plans and calculators as interactive cards

2 participants