Expressive agent starter

Run your first expressive agent

Start with a complete browser app and local Node server. It validates expression-tagged replies, displays text, drives the face from real speech events, and cancels requests and playback when you press Stop.

Download the starter ZIP or use examples/expressive-agent in the repository. The starter runs a clearly labeled demo without a provider key.

npm install
npm run dev

Open the local address printed in the terminal. Press Send to try an expressive response, then enable browser voice. Reloading clears the conversation.

Connect a real model

Copy .env.example to .env, set your OpenAI API key and a model with structured output support, then restart the server. Keys stay in the server process. The browser receives only validated text and expression segments.

OPENAI_API_KEY=your-api-key
OPENAI_MODEL=your-model-id

The starter is a local development example. Before exposing a paid endpoint, add your app’s authentication, usage limits, and provider budget controls. For Anthropic or Vercel AI SDK, use the same response contract with the provider routes.

Streaming without restarting speech

Token deltas are incomplete text, not speech segments. Accumulate them for the transcript. Validate each completed message and call speech.play once. If your backend streams complete segments, queue them and await playback in order; abort the queue when the user interrupts.

import { AvatarSpeech, normalizeSpeech, type SpeechPerformance } from '@allpersonas/core';

export function connectStream(speech: AvatarSpeech, signal: AbortSignal) {
  let queue = Promise.resolve();
  return (message: SpeechPerformance) => {
    const segments = normalizeSpeech(message.segments);
    queue = queue.then(async () => {
      if (!signal.aborted) await speech.play({ segments }, { signal });
      const error = speech.getSnapshot().error;
      if (error && !signal.aborted) throw new Error(error);
    });
    // Await or catch this promise in your stream consumer.
    return queue;
  };
}

The included starter uses one complete JSON response per request. The adapter above is for an existing streaming transport; it does not implement a token-streaming server.