ALLPERSONAS / DEVELOPERS

Connect your agent

Connect Vercel AI SDK, Anthropic, or OpenAI to your avatar.

The connection

Try the live playground → Enter your own model key, add an optional ElevenLabs voice, and chat with the published core SDK.

Your AI backendReply text + cueCore face + speech

The face stays independent of your model. In your own app, keep provider credentials on your server and pass reply text to AvatarSpeech in the browser.

Install AllPersonas using the installation guide, then choose one server implementation below for your own app. All three accept { message } and return { segments: [{ text, expression }] }, so the browser example stays the same.

Generate typed expression segments through AI Gateway. Use a Gateway key and a provider/model ID with structured output support.

Install the provider in your app

npm install ai@7 @ai-sdk/gateway@4 zod@4 @allpersonas/core

Your app’s .env.local — server only

AI_GATEWAY_API_KEY=your-gateway-key
AI_GATEWAY_MODEL=provider/model-id

These server examples use your app’s environment variables. Replace the placeholders with your key and an available model ID, then restart your server. Keep credentials server-side; do not prefix them with NEXT_PUBLIC_. Running this repository requires no environment file: its playground accepts provider settings in the browser.

app/api/your-agent/route.ts

import { generateText, Output } from 'ai';
import { createGateway } from '@ai-sdk/gateway';
import { z } from 'zod';
import { EXPRESSION_IDS, normalizeSpeech, type SpeechPerformance } from '@allpersonas/core';

const replySchema = z.object({
  segments: z.array(z.object({
    text: z.string().min(1), expression: z.enum(EXPRESSION_IDS),
  })).min(1).max(4),
});

export async function POST(request: Request) {
  // Add your application's authentication and usage limits before public hosting.
  const origin = request.headers.get('origin');
  if (origin && origin !== new URL(request.url).origin) {
    return Response.json({ error: 'Use the same origin.' }, { status: 403 });
  }
  const body = await request.json().catch(() => null);
  if (!body || typeof body.message !== 'string' ||
      !body.message.trim() || body.message.length > 2000) {
    return Response.json({ error: 'Send 1–2,000 characters.' }, { status: 400 });
  }
  const model = process.env.AI_GATEWAY_MODEL;
  if (!model || !process.env.AI_GATEWAY_API_KEY) {
    return Response.json({ error: 'Configure the model and API key on the server.' }, { status: 503 });
  }
  const signal = AbortSignal.any([request.signal, AbortSignal.timeout(45000)]);
  try {
    const instructions = 'Reply in 1–4 short segments, under 2,000 total characters. Pair each text segment with a supported expression matching its meaning. No stage directions. Expressions: ' + EXPRESSION_IDS.join(', ');
    const gateway = createGateway({ apiKey: process.env.AI_GATEWAY_API_KEY });
    const result = await generateText({
      model: gateway(model), system: instructions, prompt: body.message.trim(),
      output: Output.object({ schema: replySchema }),
      maxOutputTokens: 3000, abortSignal: signal,
    });
    const output: unknown = result.output;
    if (!output || typeof output !== 'object' || !('segments' in output)) {
      throw new Error('Invalid model response.');
    }
    const segments = normalizeSpeech(output.segments as SpeechPerformance['segments']);
    return Response.json({ segments }, { headers: { 'Cache-Control': 'no-store' } });
  } catch {
    return Response.json({ error: signal.aborted ? 'Request stopped or timed out.' : 'The model did not return a valid expression response. Please try again.' },
      { status: signal.aborted ? 504 : 502, headers: { 'Cache-Control': 'no-store' } });
  }
}

Based on the official Vercel AI SDK documentation. These are Next.js App Router examples. In another framework, expose the same POST endpoint from your server.

These examples generate one complete reply per request. They do not stream tokens or retain conversation history. Your app controls access to its server endpoint and provider usage.

A complete TypeScript example

Call mountAgent(container) with a DOM container. Use any of the three routes above at /api/your-agent. The example displays the reply, connects speech to the face, and lets Stop cancel the request and playback. Call its returned cleanup function when the view unmounts.

agent.ts

import { AvatarSpeech, createFace, normalizeSpeech, type SpeechPerformance } from '@allpersonas/core';

export function mountAgent(container: HTMLElement) {
  const root = document.createElement('div');
  root.innerHTML = `
    <div data-face style="width:280px;height:340px"></div>
    <p data-reply aria-live="polite"></p>
    <form>
      <input aria-label="Message your agent" maxlength="2000" required />
      <button data-send>Send</button>
    </form>
    <button data-stop type="button">Stop</button>
    <button data-read type="button" disabled>Read reply</button>
    <p data-error role="alert"></p>
  `;
  container.append(root);
  const face = createFace(root.querySelector<HTMLElement>('[data-face]')!, { avatar: 'ruby', expression: 'attentive' });
  const form = root.querySelector('form')!;
  const input = root.querySelector('input')!;
  const sendButton = root.querySelector<HTMLButtonElement>('[data-send]')!;
  const stopButton = root.querySelector<HTMLButtonElement>('[data-stop]')!;
  const readButton = root.querySelector<HTMLButtonElement>('[data-read]')!;
  const replyElement = root.querySelector<HTMLElement>('[data-reply]')!;
  const errorElement = root.querySelector<HTMLElement>('[data-error]')!;
  const speech = new AvatarSpeech();
  let operation: AbortController | null = null;
  let reply: SpeechPerformance | null = null;
  let disposed = false;

  function refresh() {
    sendButton.disabled = operation !== null;
    readButton.disabled = operation !== null || !reply;
  }
  function showError(cause: unknown) {
    if (!disposed) errorElement.textContent = cause instanceof Error
      ? cause.message : 'Your agent could not reply.';
  }
  const unsubscribe = speech.subscribe(() => {
    const state = speech.getSnapshot();
    face.update({
      expression: state.expression,
      speaking: state.speaking,
      boundary: speech.boundary,
    });
    errorElement.textContent = state.error ?? '';
  });

  async function send() {
    const message = input.value.trim();
    if (disposed || operation || !message) return;
    speech.stop();
    speech.clearError();
    const controller = new AbortController();
    operation = controller;
    errorElement.textContent = '';
    face.update({ thinking: true });
    refresh();
    try {
      // Use any of the server routes from this guide.
      const response = await fetch('/api/your-agent', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ message }),
        signal: controller.signal,
      });
      if (!response.ok) throw new Error('Your agent could not reply.');
      const body: unknown = await response.json();
      if (!body || typeof body !== 'object' || !('segments' in body)) {
        throw new Error('Your agent returned an invalid reply.');
      }
      const segments = normalizeSpeech(body.segments as SpeechPerformance['segments']);
      if (disposed || controller.signal.aborted) return;
      reply = { segments };
      replyElement.textContent = segments.map(segment => segment.text).join(' ');
      input.value = '';
      face.update({ thinking: false });
      await speech.play(reply, { signal: controller.signal });
    } catch (cause) {
      if (!controller.signal.aborted) showError(cause);
    } finally {
      if (!disposed && operation === controller) {
        operation = null;
        face.update({ thinking: false });
        refresh();
      }
    }
  }

  function stop() {
    operation?.abort();
    operation = null;
    speech.stop();
    face.update({ thinking: false });
    refresh();
  }
  async function readReply() {
    if (disposed || operation || !reply) return;
    const controller = new AbortController();
    operation = controller;
    refresh();
    try {
      await speech.play(reply, { signal: controller.signal });
    } catch (cause) {
      if (!controller.signal.aborted) showError(cause);
    } finally {
      if (!disposed && operation === controller) {
        operation = null;
        refresh();
      }
    }
  }
  const onSubmit = (event: Event) => {
    event.preventDefault();
    void send();
  };
  const onRead = () => { void readReply(); };
  form.addEventListener('submit', onSubmit);
  stopButton.addEventListener('click', stop);
  readButton.addEventListener('click', onRead);

  // Call when the view unmounts or before mounting a replacement.
  return () => {
    disposed = true;
    operation?.abort();
    operation = null;
    form.removeEventListener('submit', onSubmit);
    stopButton.removeEventListener('click', stop);
    readButton.removeEventListener('click', onRead);
    unsubscribe();
    speech.stop();
    face.destroy();
    root.remove();
  };
}

Browser speech may require a fresh click after a network request. Read reply lets the user replay the text. For a streaming chat, pass each completed, validated performance to speech.play once, rather than restarting it for every token.

Response format

Every server example returns a validated SpeechPerformancewith ordered text and expression segments. The client validates it again with normalizeSpeech, then calls speech.play. Expressions change when their spoken segment actually begins.

Play an expressive response

import { AvatarSpeech, type SpeechPerformance } from '@allpersonas/core';

export function playReply(speech: AvatarSpeech, signal?: AbortSignal) {
  const reply: SpeechPerformance = {
    segments: [
      { text: 'That is an interesting question.', expression: 'curious' },
      { text: 'Let us take it one step at a time.', expression: 'attentive' },
    ],
  };
  return speech.play(reply, { signal });
}

Your app controls the conversation and provider calls. Core supplies the face and browser speech presentation.

The connection

Choose your SDK

A complete TypeScript example

Response format