# Talkie Protocol 1 (`talkie/1`) Talkie lets one mobile app talk (text and real-time voice) to **any** backend that runs an AI agent. The backend keeps the agent, the tools, the permissions and the history; the app is a generic client. Talkie is a thin layer on top of existing standards: | Concern | Standard | | --- | --- | | Agent → UI event stream | [AG-UI 1.0](https://docs.ag-ui.com) over HTTP + SSE | | Sign-in | OAuth 2.1 Authorization Code + PKCE (RFC 8252 native apps, RFC 8414 discovery, RFC 9207 `iss`) | | Client registration | OAuth Client ID Metadata Document (the `client_id` is an HTTPS URL) | | Voice transport | WebRTC, signalling by SDP over HTTP POST | Talkie itself only defines: the **manifest**, a few **REST endpoints** (threads, actions, voice sessions), **custom AG-UI events** (`talkie.*`) and the **rules** both sides must follow. The key words MUST, SHOULD and MAY are used as in RFC 2119. --- ## 1. Discovery – the manifest The user enters a server address (`crm.example.com`). The client fetches `GET https:///.well-known/talkie.json`. The user MAY instead enter a full URL of a manifest. ```json { "spec": "talkie/1", "issuer": "https://crm.example.com", "serverId": "crm.example.com", "name": "Example CRM", "description": "Sales assistant of Example CRM", "locale": "cs", "brand": { "logo": "/talkie/logo.png", "accent": "#0b7a75" }, "auth": { "oauth": { "issuer": "https://crm.example.com/oauth", "scopes": ["talkie", "offline_access"] }, "token": { "hint": "Settings → Devices → New token" } }, "endpoints": { "status": "/api/talkie/status", "run": "/api/talkie/run", "cancelRun": "/api/talkie/runs/{runId}/cancel", "threads": "/api/talkie/threads", "thread": "/api/talkie/threads/{threadId}", "action": "/api/talkie/actions/{actionId}", "decision": "/api/talkie/actions/{actionId}/decision", "voiceSessions": "/api/talkie/voice/sessions", "voiceTool": "/api/talkie/voice/sessions/{voiceSessionId}/tools/{name}", "voiceTranscript": "/api/talkie/voice/sessions/{voiceSessionId}/transcript", "voiceUsage": "/api/talkie/voice/sessions/{voiceSessionId}/usage" }, "capabilities": { "voice": { "transports": ["webrtc"], "eventProtocols": ["openai-realtime/1"], "signalingHosts": ["api.openai.com"] }, "actions": { "voiceConfirm": true }, "cards": { "blocks": ["heading", "text", "fields", "list", "link", "file", "badge"] } }, "limits": { "maxMessageChars": 4000 }, "testMode": false } ``` Rules: - `spec` MUST be `talkie/1`. A client MUST refuse other major versions and ignore unknown fields. - `issuer` is the canonical origin (`https://host[:port]`). The manifest MUST be served from that origin. The client MUST NOT follow redirects to another origin while fetching the manifest. - Every endpoint is a path template resolved **against the issuer origin** (not against the manifest URL). Endpoints MUST be same-origin. An absolute URL on another origin is allowed only if the client shows it to the user and the user accepts it when adding the server. `{name}` placeholders are URL-encoded path segments. - `serverId` is stable for the life of the deployment (the client keys tokens and settings by it). - `brand` contains only a logo (same-origin path or `https` URL, PNG/JPEG, ≤ 256 kB) and an accent colour `#rrggbb`. - `auth` MUST offer `oauth` (production). `token` (a pasted bearer token) is for development and MAY be omitted. - Endpoints the server does not support are omitted (for example all `voice*` without voice). - Manifest size ≤ 64 kB. Clients SHOULD cache it (respect `Cache-Control`, default 1 hour) and refresh on 401/404. - `testMode: true` tells the conformance suite that destructive tests (actions, decisions) are safe (fixtures). - HTTPS is REQUIRED, except `http://localhost`, `http://127.0.0.1` and private LAN addresses in development builds. ## 2. Authentication ### 2.1 OAuth (production) - Authorization Server metadata at `/.well-known/oauth-authorization-server` (RFC 8414) or `/.well-known/openid-configuration`. - The Talkie app is a **public client**. Its `client_id` is the URL of its Client ID Metadata Document: `https://talkie.firemni-crm.cz/oauth/client.json`. Servers MUST either fetch and validate that document or pre-register a client with exactly that `client_id` and its redirect URIs. A fork of the app publishes its own document. - Redirect URI: `cz.firemnicrm.talkie:/oauth/callback` (private-use scheme, RFC 8252 §7.1). - Authorization Code + PKCE `S256` is REQUIRED. `state` is REQUIRED and the client binds it to the server (`serverId` + issuer) that started the flow. If the response carries `iss` (RFC 9207) it MUST equal the AS issuer. - Scopes: `talkie` (act as the user through the Talkie endpoints and the tools behind them) and `offline_access` (refresh token). Servers SHOULD rotate refresh tokens and revoke the token family on reuse. - The client sends `Authorization: Bearer ` to every endpoint. On `401` it refreshes once, then asks the user to sign in again. Bearer tokens MUST NOT be sent to any other origin (also not after a redirect). - Sign-out: the client calls the revocation endpoint (RFC 7009) when the AS has one, then forgets the tokens. - Servers SHOULD list signed-in devices to the user and let them revoke each. ### 2.2 Token (development) When `auth.token` is present the client may accept a pasted bearer token. Production builds hide it behind a developer switch. ## 3. Errors Non-2xx responses of Talkie endpoints have the body `{ "error": { "code": "...", "message": "human readable, in the user's language", "retryable": false } }`. | HTTP | code | meaning | | --- | --- | --- | | 400 | `invalid_request` | malformed input | | 401 | `unauthorized` | missing/expired token | | 403 | `forbidden` | not allowed for this user | | 404 | `not_found` | unknown thread/action/session | | 409 | `conflict` / `stale` | decision conflicts with a previous one / `paramsHash` no longer matches | | 410 | `expired` | action or session expired | | 413 | `too_large` | message over `limits.maxMessageChars` | | 429 | `rate_limited` / `quota_exceeded` | slow down / daily quota used up (`retryable` tells which) | | 502/503 | `upstream_unavailable` | model provider down (`retryable: true`) | | 500 | `internal` | anything else | ## 4. Status `GET status` → ```json { "user": { "id": "u1", "name": "Jana Nováková", "email": "jana@example.com" }, "chat": true, "voice": true, "usage": [ { "label": "Tokens today", "used": 12000, "limit": 400000, "unit": "tokens" }, { "label": "Voice today", "used": 6, "limit": 60, "unit": "minutes" } ] } ``` ## 5. Chat run (AG-UI) `POST run` with an AG-UI `RunAgentInput` body, `Accept: text/event-stream`: ```json { "threadId": "t_123", "runId": "r_9f2c", "messages": [ { "id": "m_client_uuid", "role": "user", "content": "Co mám dnes za schůzky?" } ], "tools": [], "context": [], "state": {}, "forwardedProps": { "talkie": { "channel": "chat" } } } ``` - Talkie servers are **stateful**: they keep the thread history. The client sends only the new user message(s). The server MUST ignore a message whose `id` it already stored (idempotent retry). - `threadId` of a new thread is generated by the client. The server MAY assign its own; the `threadId` in `RUN_STARTED` is authoritative and the client adopts it. - `runId` is generated by the client and used for cancelling. - Response: `200 text/event-stream`, each event `data: \n\n`. Servers SHOULD send an SSE comment (`: ping`) every 15 s and disable proxy buffering. Errors before the stream starts use the error format of §3. Event sequence (AG-UI types): | Event | Talkie usage | | --- | --- | | `RUN_STARTED` | first event, `threadId`, `runId` | | `TEXT_MESSAGE_START` / `TEXT_MESSAGE_CONTENT` / `TEXT_MESSAGE_END` | assistant text, Markdown subset (§7) | | `TOOL_CALL_START` | `toolCallId`, `toolCallName`; `metadata.label` = short human status ("Searching contacts…") | | `TOOL_CALL_END` | the tool finished; `metadata.ok` (boolean), `metadata.error` (string) | | `TOOL_CALL_RESULT` | OPTIONAL; clients do not display raw tool output | | `CUSTOM` `talkie.card` | a result card (§6) | | `CUSTOM` `talkie.action` | an action awaiting a human decision (§8) | | `CUSTOM` `talkie.open` | a suggested link `{ url, label }` – the client shows it, opens only on user tap | | `CUSTOM` `talkie.usage` | `{ usage: [...] }` as in status | | `RUN_FINISHED` | last event on success | | `RUN_ERROR` | last event on failure: `message`, `code` (§3 codes), `metadata.retryable` | Clients MUST ignore unknown event types and unknown `CUSTOM` names. **Disconnect is not cancel.** A dropped connection does not stop the run; the server finishes it and stores the result. The client recovers by loading the thread (§9). To stop a run explicitly: `POST cancelRun` → `{ "ok": true }` (also when the run already ended). ## 6. Cards – `talkie.card` ```json { "type": "CUSTOM", "name": "talkie.card", "value": { "id": "c1", "version": 1, "title": "Alfa s.r.o.", "subtitle": "Klient · Brno", "url": "/?record=abc", "blocks": [ { "type": "badge", "text": "Aktivní", "tone": "positive" }, { "type": "fields", "fields": [ { "label": "IČO", "value": "12345678" }, { "label": "Obrat", "value": "1,2 mil. Kč" } ] }, { "type": "list", "items": [ { "title": "Schůzka 10:00", "subtitle": "Novák", "url": "/?record=x" } ], "more": 3 }, { "type": "file", "name": "Nabídka N-2026-014.pdf", "mime": "application/pdf", "size": 48211, "url": "/api/files/123.pdf" } ], "fallbackText": "Alfa s.r.o. – klient v Brně, aktivní", "a11yLabel": "Karta klienta Alfa s.r.o." } } ``` Blocks: `heading {text}`, `text {text}` (Markdown subset), `fields {fields:[{label,value}]}`, `list {items:[{title,subtitle?,url?}], more?}`, `link {label,url}`, `file {name,mime?,size?,url}`, `badge {text,tone?: neutral|positive|warning|negative}`. - `fallbackText` is REQUIRED; a client that cannot render a block shows it instead of the whole card. - `url` values are same-origin paths or `https:` URLs. Same-origin `file` URLs are downloaded with the bearer token; other URLs open in the system browser without the token. Clients MUST NOT render HTML. ## 7. Text format Assistant text is a Markdown subset: paragraphs, `**bold**`, `*italic*`, `` `code` ``, bullet and numbered lists, links `[label](url)` (`https:` or same-origin). No HTML, no images. ## 8. Actions (human in the loop) The agent never performs a risky operation (sending, deleting, paying, …) directly. It **proposes** an action; a person decides. ```json { "type": "CUSTOM", "name": "talkie.action", "value": { "id": "a_77", "version": 1, "threadId": "t_123", "toolCallId": "call_5", "summary": "Odeslat nabídku N-2026-014 na novak@alfa.cz", "riskLevel": "medium", "status": "pending", "paramsHash": "sha256:4f1c…", "createdAt": "2026-10-22T10:00:00Z", "expiresAt": "2026-10-22T10:10:00Z", "confirm": { "methods": ["button", "voice"], "voice": { "keyword": "odeslat" } } } } ``` - `status`: `pending` → (`running` →) `done` | `failed`, or `cancelled` | `expired`. - `GET action` returns the current action (authoritative). - `POST decision` `{ "decision": "confirm" | "cancel", "paramsHash": "…", "method": "button" | "voice" }` → `{ "action": {…}, "events": [ AG-UI events, e.g. talkie.card with the result ] }`. - Repeating the same decision returns `200` with the current state (idempotent). - The opposite decision after one was made → `409 conflict`. A changed `paramsHash` → `409 stale`. - After `expiresAt` → `410 expired`. - A new proposal never silently replaces an older pending one. - **The decision endpoint MUST NOT be reachable by the agent** (not as a tool, not through any internal call made on the agent's behalf). Only the client, on a human gesture or utterance, calls it. - `confirm.methods` is a **UI policy** chosen by the server per action (for example `high` risk → `["button"]`). It is not proof of how the user decided: a client holding the user's token could also press the button. `method` in the decision is recorded for audit only. - Servers SHOULD delay irreversible outbound operations (e-mail, SMS) briefly and allow `cancel` during that time; such an action reports `status: "running"` with `undoUntil`. ## 9. Threads - `GET threads` → `{ "threads": [ { "id", "title", "channel": "chat" | "voice", "createdAt", "updatedAt" } ] }` (newest first, at most 50). - `GET thread` → `{ "thread": {…}, "messages": [ { "id", "role": "user" | "assistant", "content", "createdAt", "cards": [card…], "channel"? } ], "actions": [action…] }`. - `DELETE thread` → `{ "ok": true }` (pending actions are cancelled). ## 10. Voice Voice is optional (`capabilities.voice`). Audio flows **directly** between the device and the speech provider; the server creates the session, locks the model, instructions and tools, and executes the tools. ### 10.1 Session `POST voiceSessions` `{ "threadId"?: "t_123", "locale"?: "cs" }` → ```json { "voiceSessionId": "v_1", "threadId": "t_123", "transport": "webrtc", "eventProtocol": "openai-realtime/1", "signaling": { "type": "sdp-post", "url": "https://api.openai.com/v1/realtime/calls" }, "clientSecret": "ek_…", "expiresAt": "2026-10-22T10:01:00Z", "maxSeconds": 900, "warnSeconds": 60, "locale": "cs", "labels": { "search_records": "Hledám v CRM…" }, "consent": { "confirm": ["ano", "potvrzuji"], "refuse": ["ne", "zruš"] } } ``` - The client sends the SDP offer to `signaling.url` with `Authorization: Bearer ` and `Content-Type: application/sdp`, and receives the SDP answer. `signaling.url` MUST be `https` and its host MUST be listed in `capabilities.voice.signalingHosts`; the client never sends `clientSecret` anywhere else. - The secret only opens the connection (short-lived). The server MUST configure the provider session (model, instructions, tools, limits) so the client cannot change them; clients MUST NOT send `session.update` changing anything except turn detection. - `consent` (OPTIONAL) adds words to the client's built-in confirm/refuse vocabularies. ### 10.2 Tools during voice The provider asks for a function call on the data channel; the client calls `POST voiceTool` `{ "callId": "call_1", "arguments": "{…}" }` → `{ "ok": true, "output": "", "events": [AG-UI events: talkie.card, talkie.action…] }`. - The server binds the call to the voice session and the user, and rejects a reused `callId` (`409`). - `output` is passed to the model unchanged as `function_call_output`. ### 10.3 Transcript and usage - `POST voiceTranscript` `{ "role": "user" | "assistant", "text": "…", "itemId": "…" }` → `201` (stored in the thread). - `POST voiceUsage` `{ "seconds": 125, "final": false }` → `{ "ok": true, "remainingSeconds": 775 }`. The client reports every 30 s and at the end (`final: true`). ### 10.4 Confirming actions by voice Only for actions whose `confirm.methods` includes `voice`. The client: 1. After the tool result with a pending action, disables automatic responses (turn detection `create_response:false`) and lets the model read the proposal. 2. Opens the consent window **after** the assistant finished speaking that proposal (`output_audio_buffer.stopped`). Speech that started earlier never counts. 3. Takes the next **final** user transcript: a short sentence (≤ 6 words) with a confirm word, no negation, no question, no hesitation word, and – if `confirm.voice.keyword` is set – containing the keyword → `confirm`. An explicit refusal → `cancel`. Anything else → normal conversation; after two such utterances the voice window closes and the action waits for the button. 4. Closes the window before calling `decision`, so a duplicated transcript cannot decide twice. Voice confirmation is a convenience allowed by the server's policy, not authentication of the speaker. ## 11. Test mode (conformance) A server whose manifest says `testMode: true` MUST answer these user messages deterministically, without calling a model (they exist so `talkie-conformance` can test any backend). Production servers MUST NOT enable test mode. | Message | Response events (between `RUN_STARTED` and `RUN_FINISHED`) | | --- | --- | | `talkie-test:echo ` | one text message whose content is exactly `` | | `talkie-test:card` | `TOOL_CALL_START`/`TOOL_CALL_END` (`metadata.ok: true`), one `talkie.card` with a `fields` block, a short text | | `talkie-test:action` | one `talkie.action` (`status: pending`, `riskLevel: low`, `confirm.methods: ["button","voice"]`, expires in ≥ 60 s), a short text; confirming it yields `status: done` | | `talkie-test:error` | `RUN_ERROR` with `code: "upstream_unavailable"` and `metadata.retryable: true` | | `talkie-test:slow` | text deltas over ~5 s (for cancel and disconnect tests) | ## 12. Security checklist (servers) - Treat everything the agent returns as data; the client renders no HTML and opens links only on tap. - Keep the decision endpoint unreachable for the agent (§8). - Lock voice sessions server-side (§10.1); bind voice tool calls to the session (§10.2). - Rotate refresh tokens, offer device revocation, rate-limit runs and voice minutes. - Tell users that voice audio goes to the speech provider.