AI・テクノロジー
Sakana AI's "KAME" — A Voice AI That Thinks While It Speaks
機械翻訳 / Machine-translated
A
AI Friends
2026年5月6日・22分で読める
機械翻訳 / Machine-translated
@aifriends
AI Friends（https://aifriends.jp）のクロスポスト公式アカウント。AIツールの紹介・使い方・できることを、中学生でもわかるやさしい日本語で届けます。
Have you ever felt that a conversation with an AI feels awkward because it needs time to think before responding? This stems from the fundamental dilemma facing voice AI: answer fast, or answer intelligently. On May 3, 2026, Sakana AI — a Tokyo-based AI research lab — released "KAME" to solve this dilemma.
KAME (Knowledge-Access Model Extension) means "turtle" in Japanese.
Despite its name, it operates remarkably fast.
KAME, released by Sakana AI on May 3, 2026, is a real-time conversational voice AI. It was developed based on a paper (arXiv:2510.02327) accepted at ICASSP 2026, a top international conference in the field of speech and signal processing.
Traditional voice AI operated on a "think first, then speak" model. KAME is different. It adopts a new approach: "keep thinking in parallel while speaking."
This mechanism enables deep, knowledge-rich responses without sacrificing response speed.
Sakana AI is a research-driven AI startup based in Tokyo.
Founded by researchers from major companies including Google, it has released a succession of industry-surprising research, including the Japanese-specialized model "Namazu" and "Evolutionary Model Merge."
KAME is a further achievement in that tradition, bringing a new paradigm to the field of voice AI.
Voice AI can be broadly divided into two approaches.
The first is the "cascade" type. It processes user speech in three sequential steps: convert speech to text (STT) → reason with an LLM → convert text back to speech (TTS).
While cascade systems can deliver high-accuracy responses, they often require more than 2 seconds before responding, which disrupts the conversational tempo. A leading example, Unmute, has a median response wait time of 2.1 seconds.
The second is the "end-to-end" type. Because it processes audio directly as audio, responses are extremely fast.
However, end-to-end systems struggle to leverage the rich knowledge of large language models and tend to perform poorly on questions requiring deep reasoning. The representative model Moshi has an MT-Bench score of just 2.05, revealing clear challenges with knowledge quality.
Sakana AI describes this trade-off as "Think then speak" vs. "Speak while thinking."
What KAME aimed for was to maintain the speed of end-to-end systems while approaching the knowledge quality of cascade systems.
The answer is the tandem architecture.
KAME operates with two components running in parallel.
A "frontend S2S (speech-to-speech) model" and a "backend text LLM" operate asynchronously, combining the strengths of each.
The frontend is based on Moshi's architecture and processes audio instantly as the user speaks. Response latency is near zero.
For the backend, any preferred large-scale LLM can be connected — including GPT-4.1, Claude Opus 4.1, or Gemini 2.5 Flash. The backend performs deep reasoning asynchronously and continuously feeds results to the frontend as an "oracle."
Moshi originally had three streams (data flows): input audio, internal thought text, and output audio.
KAME adds a fourth stream called the "oracle stream."
Here's how it works: as the user speaks, an STT component continuously generates partial text transcriptions and sends them to the backend LLM. From these fragmentary inputs, the LLM infers "this is probably a question about X" and returns candidate responses (oracles) to the frontend.
The frontend receives the oracle and adjusts its audio output to align with it. Each time the backend updates, the direction of the audio output is refined accordingly.
Making this work requires special training so the frontend model can effectively use "oracles sent mid-stream."
Sakana AI developed a technique called "Simulated Oracle Augmentation," training the model on 56,582 synthetic dialogue examples.
The training data was created by converting well-known benchmark datasets — MMLU-Pro, GSM8K, and HSSBench — into conversational format and synthesizing them into audio using TTS.
One of the key metrics for evaluating KAME's performance is "MT-Bench" — a benchmark that rates reasoning, STEM, and humanities quality in multi-turn conversations on a scale of 0 to 10.
The baseline Moshi's MT-Bench score was 2.05. KAME raised this to 6.43 — an improvement of more than threefold.
Breaking it down: reasoning 6.48, STEM 8.34, humanities 8.56, for an overall average of 7.79 (with a GPT-4.1 backend). The comparable system Unmute scores 7.70, meaning KAME reaches nearly equivalent knowledge quality.
What's remarkable is not just the score, but achieving it while maintaining speed.
KAME's median response latency is near zero — essentially the same level as Moshi.
Meanwhile, Unmute's median response wait time is 2.1 seconds. KAME achieves knowledge quality close to Unmute's at a dramatically lower latency.
| Model | MT-Bench Score | Response Latency |
|---|---|---|
| Moshi (baseline) | 2.05 | Near zero |
| KAME (GPT-4.1 backend) | 6.43 | Near zero |
| KAME (Claude Opus 4.1 backend) | 6.23 | Near zero |
| Unmute (cascade type) | 7.70 | 2.1 seconds |
Another feature of KAME is the ability to swap the backend LLM without retraining.
The frontend is trained with gpt-4.1-nano, but at inference time it can connect to any LLM — GPT-4.1, Claude Opus 4.1, Gemini 2.5 Flash, and more.
The flexibility to switch between accuracy-focused models and lightweight cost-efficient models depending on the use case is a significant advantage for business applications.
KAME is fully released for free under the MIT License.
The inference code is available on GitHub as "SakanaAI/kame," the fine-tuning code as "SakanaAI/kame_finetune," and model weights can be downloaded from Hugging Face at "SakanaAI/kame."
It is provided as a Python package — after installation, simply run server_oracle.py to try oracle-guided dialogue.
The fine-tuning pipeline using Simulated Oracle Augmentation is also published, allowing you to train new frontend models on your own data.
Sakana AI, the developer of KAME, is a Japanese AI research lab based in Tokyo.
Acceptance at ICASSP 2026 — a top international conference in speech and signal processing — means this research has been recognized as a globally significant achievement.
Japan has a high demand for accurate voice AI in fields such as real-time translation, service robots, and medical interview AI. KAME's architecture has strong potential for direct application in these areas.
Let's imagine some real-world business applications.
A call center operator wants to introduce an AI-powered automated response system. With traditional cascade AI, a silence of over 2 seconds — "Please wait a moment" — would drag on and lower customer satisfaction. KAME's tandem approach can start a conversation instantly while the backend LLM simultaneously delivers accurate information.
In customer service AI handling restaurant reservations, changes, and cancellations, it becomes possible to "respond while checking availability" at a natural conversational pace.
With health consultation AI for elderly users, a slow response can be mistaken for a malfunction. KAME's near-zero latency characteristic helps drive adoption among users less familiar with technology.
OpenAI's GPT-4o also features "Advanced Voice Mode" — a high-quality voice conversation function.
GPT-4o Voice is high quality, but it is a closed system dependent on OpenAI's services. KAME is an open-source, open architecture where you can freely choose your backend LLM.
From a privacy perspective, running KAME on your own infrastructure means conversational data never leaves your environment. This makes it well-suited for adoption in industries handling sensitive information, such as healthcare, finance, and legal services.
A. The KAME frontend model currently available supports training data that is English-based.
However, it is technically possible to connect a Japanese-compatible model as the backend LLM. Given Sakana AI's track record in Japanese AI research, there is significant anticipation for a future Japanese-language version.
A. The frontend S2S model is based on Moshi and is relatively lightweight, so it can run on consumer-grade GPUs.
If you call the backend LLM via API (e.g., GPT-4.1 or Claude Opus 4.1), no GPU is needed — just an API key. Running your own LLM locally will require a suitable GPU.
A. KAME's code is released under the MIT License and commercial use is permitted.
You will need to separately check the terms of service for any backend LLM you connect (e.g., GPT-4.1). Using a model you have fine-tuned yourself as the backend allows you to build a fully controlled configuration.
A. Moshi is the real-time S2S model itself. KAME is an extended framework that adds an "oracle stream" to Moshi, enabling the injection of knowledge from a backend LLM.
The MT-Bench score improved more than threefold from 2.05 (Moshi) to 6.43 (KAME) — KAME is essentially a mechanism that makes Moshi smarter.
Sakana AI's KAME is a groundbreaking achievement that resolves the apparent contradiction of "fast and intelligent voice AI" through technology. Your first step is to visit GitHub (SakanaAI/kame) and run it in your own environment.
This article is a cross-post from AI Friends.
