Skip to content

0032 — Studio AI assistant (conversational, context-aware, action-taking)

Status: DRAFT — vision + architecture carve-out; not a full design. This reserves the space and the seams for a large, phased feature; each phase gets its detailed design as it's built. Date: 2026-07-12 Source: ideation 2026-07-12 — a chat interface in the studio where the user talks to their configured provider's AI, and it assists on each section — generating images, scripts, text, and copy as part of the process. Matt: "this should probably be its own spec with how complex it could get, but I want to carve out space for it now." Depends on: the existing chat (internal/chatgen, pkg/studio/chat.go, Chat.svelte), the generation cores (gencmd, internal/takes, social compose), the provider config (providers.chat.provider), GTB pkg/chat (tool-calling capable). Relates: the IA in 0029 §8 (the Chat drawer is this assistant's home), the decomposition 0028 (chat gets a chat.svelte.js session), MCP (0001/0002 §5 — the same command cores exposed as tools).

1. Goal & scope

A conversational assistant embedded in the studio that (a) is context-aware of the current phase and selection, and (b) can both converse and act — draft/critique scripts and text, generate card/cover images, suggest or trigger VO/music, draft per-platform social copy — through the user's configured chat provider (claude | gemini | openai). It is the in-UI counterpart to keryx's external MCP assistant, working the same command cores.

This spec's job is to carve out space, not to finish the design: state the vision, record the honest baseline, fix the architecture and seams so the feature can grow safely, phase it, and enumerate the (many) open questions. Detailed per-phase design follows.

Out of scope (here): the final tool set, the exact agency model, autonomous posting (never — §7), and provider-by-provider tool-calling parity (a Phase-gating question, §8).

2. Baseline — what exists today (honest)

  • internal/chatgen wires GTB's provider-abstracted chat client (chat.ChatClient; provider from providers.chat.provider = claude|gemini|openai, or inferred from the one present API key). GTB's client supports tool-calling (SetTools, tool_use) and streaming (StreamChat).
  • pkg/studio/chat.go exposes a narrow Chatter seam: Propose(ctx, system, user, onToken) → (reel.Storyboard, summary, error). The studio chat endpoint (handlers_chat.go, R-UI-5) takes a message + the current board + source and proposes a new storyboard (structured), streamed. System prompt: "keryx's authoring assistant for short vertical reels."
  • Chat.svelte renders that single flow.

So today the assistant does exactly one thing — draft a storyboard in the Author phase. It can't assist per-section, can't take actions (generate media, edit a single card, draft copy), and isn't a general multi-turn conversation with tools. Everything below generalises this one seam.

3. Vision

The Chat drawer (0029 §8) becomes a phase-aware assistant:

  • Author: draft/refine the storyboard and per-card narration; critique pacing; suggest an accent word; "make card 3 punchier."
  • Produce: "generate an image for card 2 in the theme", "re-roll the VO for line 4", "the music's too busy — regenerate a calmer bed" — proposing/triggering the existing generation cores.
  • Publish: "draft an Instagram caption from this reel", "tighten the LinkedIn copy", "what hashtags fit?" — drafting per-platform copy.

It converses (answers, critiques, suggests) and acts (produces artefacts through keryx's own cores), with the current phase + selection as context, and every paid/side-effecting action gated by explicit user confirmation (§7).

4. Architecture — the seams to reserve now

Five seams. Reserving them now (even before the later phases build out) is the point of this spec.

  1. Provider — reuse as-is. The config-driven GTB chat client (providers.chat.provider). No change; the assistant is provider-neutral by construction.
  2. Assistant seam (generalise Chatter). Replace the board-only Propose with a general Assistant: multi-turn Converse(ctx, turn, context, onEvent) that streams text and tool-call events. The current storyboard proposer becomes one tool (propose_ storyboard) within it — Phase 0 preserved, not discarded.
  3. Action-tool registry. Define studio tools as GTB chat.Tools whose handlers call keryx's existing cores — gencmd.GenerateInto (card/cover/VO/music), takes.Select*, storyboard edits, social compose. These are the same cores keryx's MCP wraps, so the in-UI assistant and the external MCP assistant stay consistent (and post/approve/auth stay excluded — §7). One registry, resolved per phase (Author tools ≠ Publish tools).
  4. Context provider. A seam that assembles the per-turn context — current phase, selected card/line, the board, the source post, the theme, asset/selection state, and quota/cost — injected as system/context so the model answers about this reel, not in the abstract.
  5. Action gating + confirmation. Any paid (generation spend) or side-effecting action is a proposal the user confirms in the UI before keryx executes; cost is disclosed first (reuse the 0016 estimate + QuotaReporter). Posting is never autonomous (§7). Read/draft actions (suggest copy, critique) need no gate.

UI (the 0029 §8 drawer, phase-aware): messages interleave the model's text with action proposals (a confirm button + cost) and results (an image thumbnail, an audio take, a board/copy diff). Tool-call progress surfaces inline ("generating card 2 image…").

5. Phasing (deliberately incremental — this can get complex)

  • Phase 0 (exists): storyboard proposer (Author). Fold into the new Assistant seam as a tool; no regression.
  • Phase 1 — conversational assist (no actions): per-phase, context-aware text answers/critique/suggestions on the current card/copy. Streamed. Lowest risk; immediately useful; establishes the context seam.
  • Phase 2 — proposed + confirmed single actions: the assistant proposes one concrete action (generate this image / re-roll this VO / draft this caption) as a confirmable button; on confirm, keryx runs the existing core and the result lands in the flow. Establishes the tool registry + gating.
  • Phase 3 — tool-calling (light agency): the model chains a few tools to satisfy a multi-step ask ("make card 3 punchier and regenerate its image"), with per-action confirmation for spend/side-effects. Uses GTB's tool_use loop.
  • Phase 4+ — richer: multimodal input (attach a reference image), provider failover, conversation memory/history persistence, "explain what you changed" diffs.

6. Reserve-now (what lands even before Phases 2+)

To avoid a rewrite later, the early work should already: generalise Chatter → Assistant (multi-turn + event stream), stand up the context provider seam, and define the action-tool registry interface (even if Phase 1 registers zero acting tools). The storyboard proposer keeps working throughout. The 0028 decomposition should give chat its own chat.svelte.js session hosting this.

7. Safety, cost, and agency

  • Never autonomous posting. The assistant may draft copy and propose a post, but post/approve/auth are never executable tools — mirroring the MCP exclusion (R-MCP-2). Publishing stays a human action on the CLI/Publish surface.
  • Spend is always confirmed + disclosed. No generation runs without the user OK-ing the estimated cost (reuse 0016 cost + QuotaReporter). Optional per-session spend cap.
  • Actions are reversible / take-based. Generation lands as a candidate take (the existing model), never overwriting a selection without a pick — so an assistant action is as safe as a manual one.
  • Provider keys use the existing config/keychain path; no new secret surface.

8. Open questions (many — reflects the complexity)

  • Q1 — provider tool-calling parity. GTB's Claude client supports tools; verify/gate on Gemini + OpenAI coverage in GTB pkg/chat. Phase 3 may be Claude-first, others text-only until parity.
  • Q2 — agency model. Propose-only (user clicks every action) vs light autonomous chains with per-action confirm. Lean: propose-only through Phase 2, light chains in Phase 3.
  • Q3 — reuse MCP tool definitions directly, or a parallel studio registry? Same cores either way; decide whether the assistant literally consumes the MCP tool manifest or a hand-picked studio subset.
  • Q4 — context/token budget. How much board/source to inject per turn; summarise vs full; cost of a long conversation.
  • Q5 — confirmation UX + streaming tool-calls. How proposals/results render inline; how a mid-stream tool call is surfaced and cancelled.
  • Q6 — per-phase tool sets + when the assistant switches context (follows the active tab? explicit?).
  • Q7 — history/memory. Persist a conversation per reel (stateless-tool tension — 0001 §3.2), or ephemeral per session?
  • Q8 — cost caps + abuse. Per-session/most-per-day spend guard for an over-eager loop.
  • Q9 — how much is keryx-specific vs a GTB primitive? A generalised "studio assistant over tool-wrapped cores" could be a GTB pattern (krites would want it too) — decide the keryx/GTB split, like the local-CA feature.

9. Testing / DoD (per phase)

Failing test → code → green just ci. Fakes for the chat client (no LLM/network) and each tool handler (no spend); assert: context assembly per phase, the Assistant streams text+tool events, confirmation gating blocks unconfirmed paid/side-effecting actions, posting tools are absent, the storyboard-proposer parity holds. A godog scenario per phase (assist → propose → confirm → result via fakes; the real provider path stays INT_TEST=1-gated). Frontend: @testing-library/svelte over the drawer + the 0028 Playwright harness for the assist→confirm→result flow (mocked API). Docs: a studio "AI assistant" concepts/how-to page per phase.