ALLPERSONAS / DEVELOPERS

Speech & mouth motion

Connect browser speech or your own audio to the face.

Browser speech

AvatarSpeech coordinates expression changes, word events, and interruption. Subscribe to its state and forward it to the face.

Example.tsx

import { AvatarSpeech, createFace } from '@allpersonas/core';

export function mountTalkingAssistant(container: HTMLElement) {
  const root = document.createElement('div');
  root.innerHTML = `
    <div data-face style="width:280px;height:340px"></div>
    <button data-speak type="button">Say hello</button>
    <button data-stop type="button">Stop</button>
    <p data-error role="alert"></p>
  `;
  container.append(root);
  const face = createFace(root.querySelector<HTMLElement>('[data-face]')!);
  const speakButton = root.querySelector<HTMLButtonElement>('[data-speak]')!;
  const stopButton = root.querySelector<HTMLButtonElement>('[data-stop]')!;
  const error = root.querySelector<HTMLElement>('[data-error]')!;
  const speech = new AvatarSpeech();
  let disposed = false;
  const unsubscribe = speech.subscribe(() => {
    const state = speech.getSnapshot();
    face.update({
      expression: state.expression,
      speaking: state.speaking,
      boundary: speech.boundary,
    });
    error.textContent = state.error ?? '';
  });

  const sayHello = () => {
    void speech.speak([
      { expression: 'joyful', text: 'It is good to meet you.' },
      { expression: 'curious', text: 'What is on your mind?' },
    ]).catch((cause: unknown) => {
      if (!disposed) error.textContent = cause instanceof Error
        ? cause.message : 'Speech could not start.';
    });
  };
  speakButton.addEventListener('click', sayHello);
  stopButton.addEventListener('click', speech.stop);

  return () => {
    disposed = true;
    speakButton.removeEventListener('click', sayHello);
    stopButton.removeEventListener('click', speech.stop);
    unsubscribe();
    speech.stop();
    face.destroy();
    root.remove();
  };
}

Start speech from a click or another user gesture. Voices and word-boundary support vary by browser. Without timing events, audio can play while the mouth stays closed.

AvatarSpeech reference

API	Result	Purpose
`speak(input, options?)`	Promise<void>	Play a string or one to four text/expression segments.
`play({ segments }, options?)`	Promise<void>	Play a SpeechPerformance with the same validation as speak.
`stop()`	void	Interrupt playback and clear the word boundary.
`subscribe(listener)`	() => void	Observe state changes; call the returned function to unsubscribe.
`getSnapshot()`	SpeechState	Read status, speaking, expression, and error.
`getServerSnapshot()`	SpeechState	Read the initial state for a server snapshot.
`boundary`	{ current: WordBoundary \| null }	Pass the stable word-timing object to face.update.
`clearError()`	void	Clear the playback error and return an error status to idle.

Status is idle, loading, speaking, or error. Invalid input rejects the promise; playback failures appear in the snapshot’s error. Text is limited to 2,000 combined characters, including spaces between segments. Options accept an AbortSignal and an onSegment callback.

Browser speech is shared across a document. Keep one active speaker at a time. On unmount, unsubscribe, call speech.stop(), and destroy the face.

Your own audio

Use playback events for speaking and a Web Audio analyser for measured RMS amplitude. Call the returned play() from a user gesture and display any rejected playback error in your app. Call dispose() on unmount.

External audio

import { createFace } from '@allpersonas/core';

export function mountAudioAssistant(host: HTMLElement, audioUrl: string) {
  host.style.width = '280px';
  host.style.height = '340px';
  const audio = new Audio();
  audio.crossOrigin = 'anonymous';
  audio.src = audioUrl;
  const context = new AudioContext();
  const analyser = context.createAnalyser();
  analyser.fftSize = 256;
  const source = context.createMediaElementSource(audio);
  source.connect(analyser);
  analyser.connect(context.destination);
  const samples = new Float32Array(analyser.fftSize);
  const face = createFace(host, {
    avatar: 'leo',
    getLevel: () => {
      analyser.getFloatTimeDomainData(samples);
      let sum = 0;
      for (const value of samples) sum += value * value;
      return Math.sqrt(sum / samples.length);
    },
  });
  let disposed = false;
  const onPlaying = () => face.update({ speaking: true });
  const onQuiet = () => face.update({ speaking: false });
  const quietEvents = ['pause', 'waiting', 'ended', 'error'] as const;
  audio.addEventListener('playing', onPlaying);
  quietEvents.forEach(event => audio.addEventListener(event, onQuiet));

  return {
    // Call from a user gesture; catch playback errors in your UI.
    async play() {
      if (disposed) return;
      await context.resume();
      if (disposed) return;
      try {
        await audio.play();
        if (disposed) audio.pause();
      } catch (cause) {
        if (!disposed) throw cause;
      }
    },
    stop() { audio.pause(); },
    async dispose() {
      if (disposed) return;
      disposed = true;
      audio.pause();
      audio.removeEventListener('playing', onPlaying);
      quietEvents.forEach(event => audio.removeEventListener(event, onQuiet));
      source.disconnect();
      analyser.disconnect();
      face.destroy();
      await context.close();
    },
  };
}

Use a same-origin audio URL or a server that allows cross-origin audio analysis. For your provider’s own timing, update mouthOpen with a value from 0 to 1 instead. The mouth closes whenever speaking is false. This is event-driven mouth motion, not phoneme-level lip sync.