Audio8 preview

Experimental engine, assets not distributed

Audio8 0.1B is exposed through @optimalai/react-native as an experimental engine, but OptimalAI does not distribute the graphs, tokenizer or reference prompts. The original iPhone spike still lives in apps/expo-tts-demo. Treat everything on this page as a moving target.

What it is

A second on-device engine that trades Kokoro's fixed voice list for zero-shot voice cloning. Three ONNX graphs run through the same onnxruntime-react-native session API the Beta already depends on:

Graph	Precision	Size
Slow AR	INT8, FP32 recurrent state	131.89 MiB
Fast AR	INT8	35.51 MiB
Codec decoder	FP16 weights, FP32 PCM	249.23 MiB
Tokenizer, manifests, reference codes	—	5.59 MiB
Online synthesis total		422.21 MiB
Codec encoder (optional, for cloning)	FP16	396.12 MiB

The codec decoder, not the 170M-parameter backbone, dominates the footprint.

What works today

• Editable Chinese and English text, up to 150 code points per synthesis.
• 31 selectable reference profiles — the upstream default plus 15 Chinese and 15 English official demo recordings. Recordings may share speakers; these are not 31 verified identities.
• On-device voice cloning: record 1–15 s of mono 44.1 kHz Float32 PCM, confirm the transcript, encode locally, and save a reusable voice template under Documents/audio8-voices. Nothing is uploaded.
• 44.1 kHz 16-bit mono WAV written to the app cache, played back in the demo.
• Greedy decoding to EOS, capped at 512 codec frames. Hitting the cap is reported as truncated rather than silently cut.

Use it from the SDK

Pass an Audio8Model descriptor to the same createTts call:

import { createTts } from "@optimalai/react-native";

const tts = createTts({
  model: {
    engine: "audio8",
    modelId: "audio8-0.1b",
    modelVersion: "1",
    assets: {
      slow: require("./models/slow_ar_int8.mobile.onnx"),
      fast: require("./models/fast_ar_int8.mobile.onnx"),
      decoder: require("./models/codec_decoder_fp16.mobile.onnx"),
      encoder: require("./models/codec_encoder.mobile.onnx"), // Optional: cloning.
      tokenizer: require("./models/tokenizer.json"),
      voices: { default: require("./voices/default.prompt.bin") },
    },
    manifest: {
      schemaVersion: 1,
      engine: "audio8",
      modelId: "audio8-0.1b",
      modelVersion: "1",
      sampleRate: 44100,
      maxTextCodePoints: 150,
      maxNewTokens: 512,
      contextLength: 2048,
      voices: [{ id: "default", label: "Default", language: "zh" }],
    },
  },
  voiceId: "default",
  numThreads: 4,
});

await tts.load();
const result = await tts.synthesizeToFile({ text: "你好，Hello world." });
console.log(result.frames, result.truncated, result.fileUri);

Audio8 rejects speed; use maxNewTokens to cap generation. Kokoro rejects maxNewTokens. The SDK ships the engine only, not the model kit.

What it costs

Audio8 is autoregressive at 44.1 kHz with a hop of 2048, which is about 21.53 frames per audio second. Each frame runs one Slow AR step plus ten Fast AR steps, so one second of audio is roughly 237 session calls before the sliding-window codec decoder even starts. The SDK keeps the whole loop in JavaScript.

The only measured number so far is a Mac four-thread smoke test: 32 frames, 1.486 s of audio, 7.251 s of synthesis, RTF 4.879. That is a warning sign, not an iPhone benchmark — physical-device RTF and peak memory are still open.

Expectation setting

Do not plan a product around the preview until a physical iPhone actually sustains RTF below 1 and the memory probe is clean.

Run it from source

pnpm install
pnpm audio8:prepare        # 3 online graphs, hash-verified, ~422 MiB
pnpm audio8:voices         # +30 official demo references
pnpm audio8:clone-prepare  # optional codec encoder for cloning, ~396 MiB
pnpm typecheck
pnpm test
pnpm demo:ios -- --device "your iPhone"

Then select Audio8 at the top of the demo, pick a reference (or clone your own), edit the text, tap Load model, and generate. The first development load pushes roughly 417 MiB of assets from Metro into the app cache, so keep the phone connected and the Mac awake.

Generated assets are intentionally ignored by Git. pnpm audio8:prepare pins Hugging Face commit 317c12d4e0da83847b594fcf8bd74bf2c76615ec and verifies every upstream hash.

Limitations

• iPhone 17 Pro prototype only. There is no Android evidence, no simulator story, and no supported-device list.
• File output only. The upstream sliding-window streaming path has not been ported; the SDK has no streaming or cancellation API.
• No speed control. Audio8 exposes temperature / top-p / top-k / seed, not a Kokoro-style speed scalar. Do not pretend the semantics match.
• Memory pressure is unmeasured. Three graphs plus roughly 54 MiB of FP32 recurrent state, all driven from JS, is the main crash risk.

Licensing gate

This is why OptimalAI does not ship Audio8 assets:

• The 0.1B base model ships under the Audio8 Community License v1.0, which is free for commercial use only below USD 2M annual revenue.
• The 0.1B ONNX INT8 model card separately claims Apache-2.0. Audio8 has not published how the two licenses relate.
• The reference voice recordings have unverified commercial redistribution rights.

Until Audio8 provides written clarification covering the converted ONNX graphs, the tokenizer, and the reference codes — and until voice rights are cleared — do not bundle these assets into a distributed build.

See AUDIO8_TTS_RESEARCH.md and AUDIO8_IPHONE_PROTOTYPE.md in the repository for the full evidence and go/no-go rule.