Running a 125B-Class AI on 12GB GPU | Introducing Strata
機械翻訳 / Machine-translated
A
AI Friends
5日前・18分で読める
機械翻訳 / Machine-translated
@aifriends
AI Friends（https://aifriends.jp）のクロスポスト公式アカウント。AIツールの紹介・使い方・できることを、中学生でもわかるやさしい日本語で届けます。
A massive AI that should require hundreds of gigabytes of memory — running on a gaming PC. When you hear that, doesn't it make you wonder whether it's really true?
This article walks through the mechanics, speed, requirements, and caveats of "Strata," released under the MIT license.
Strata is an inference engine that developer Niko1221 published on GitHub. An inference engine is software that takes a finished AI model and actually runs it to produce answers.
The target model is Alibaba's "Qwen3.8-Flash-Next." Because it is under the MIT license, you are free to use and modify it. Note, however, that the model itself has a separate license.
When launched, a local API at 127.0.0.1:8080 starts up on your machine. It supports both OpenAI-compatible and Anthropic-compatible interfaces, so you can connect existing AI coding tools. There is also a browser-based chat interface and image input.
Your data never leaves your own computer. The fact that text you type or internal code is never sent to an external server is one of the major advantages of running AI locally.
According to the model page on Hugging Face, the language model has 125 billion parameters. However, when generating a single token, only about 6 billion parameters are actually used.
This is due to an architecture called MoE (Mixture of Experts). MoE works by preparing many small specialists with different areas of expertise and calling only the necessary few for each token.
This model has 48 layers, each containing 512 experts — 24,576 in total. When generating a single token, only 10 of them plus one shared expert are used.
The context length is 262,144 tokens, extendable to 1 million tokens. The license is listed as "qwen-community-1.0." If you intend to use it commercially, please verify the terms yourself.
Normally, a 125-billion-parameter model cannot fit into GPU memory. Strata addresses this by distributing the experts across three locations.
According to the official documentation explaining the mechanism, the division of responsibilities is as follows.
The GPU holds the parts that are always used for every token, along with the most frequently used experts. System RAM holds all 24,576 experts.
When an expert not residing on the GPU is needed, the CPU takes over. This computation runs in parallel with the GPU's computation, which helps minimize wait times.
A large lookup table (an n-gram table) of approximately 28.8GB resides on the SSD. Since only a few rows are read per token, the load on the SSD remains small.
To use a library analogy: frequently borrowed books sit on the desk within arm's reach, the full collection is in the stacks, and a massive index is kept in an annex.
Another technique is speculative decoding. A small assistant built into the model predicts the next several tokens in advance. The larger model then verifies those predictions all at once in a single pass.
The final answer is still determined by the large model, so output quality does not change. The developer states that this results in a 1.6–1.8× speed improvement.
In official benchmarks, using an RTX 5070 (12GB VRAM) and 64GB of system RAM, the lightest quantization "Q2_0" generates text at 94 tokens per second. A token is the unit AI models use to process text.
The prefill (reading a long prompt) speed is 2,650 tokens per second. With the recommended "IQ2_XS," generation speed is 79 tokens per second and prefill is 2,090 tokens per second.
The AMD RX 9070 XT (16GB VRAM) also achieves 60 tokens per second on Q2_0. On an RTX 3090 (24GB VRAM), the developer estimates 100–140 tokens per second.
The required environment is as follows:
Model sizes are: Q2_0 at 37.6GB, IQ2_XS at 39.2GB, IQ3_XXS at 47GB, and IQ3_S at 54.8GB. Larger numbers mean higher accuracy but lower speed; IQ3_S runs at 53 tokens per second.
Setup is straightforward — on Windows, according to PC Watch, you simply extract the ZIP and double-click "START-HERE.bat." The Python environment setup and model download proceed automatically.
The idea of offloading experts to RAM is not new to Strata. Here is a quick comparison.
First, llama.cpp. As described in this guide, options like --cpu-moe and --n-cpu-moe allow experts to be offloaded to the CPU. It supports many models and is easy to try.
Next, KTransformers. A group at Tsinghua University published a paper at the international conference SOSP '25. Using a dedicated mechanism to accelerate CPU-side computation, it is reported to achieve 1.25–4.09× the generation speed of existing approaches.
Strata's distinguishing feature is that it is highly optimized for a specific model. It learns which experts are frequently used and places them on the GPU, combining that with speculative decoding and an SSD-based lookup table. The trade-off is that it only supports a limited set of models.
PC Watch covered Strata on October 2, 2026, and interest among Japanese users is likely to grow.
Consider an engineer at a company where code cannot be sent outside the organization. A locally running AI for coding would let them work while keeping confidential information protected. The Anthropic-compatible API also makes it easy to connect to existing tools.
For individual developers, you can run as many inference iterations as you like without worrying about monthly fees. Electricity costs still apply, but usage limits are no longer a concern.
Academic research labs could benefit as well. Large-model experiments become possible on a local gaming PC without renting expensive servers.
That said, output quality in Japanese was not verified in this investigation. Please test it for your own use case before committing to it.
The numbers are appealing, but there are caveats worth noting.
First, the speed figures are self-reported by the developer. The PC Watch article explicitly notes that these numbers were not independently verified. Figures for the RTX 3090 and 40-series cards are listed as estimates.
Second, there are limitations pointed out by AlphaSignal. Only one request can be processed at a time, output is limited to greedy decoding (always selecting the highest-probability token), and prefill slows down for prompts longer than 128K tokens.
Third, the hardware requirements are demanding. 12GB of VRAM, 64GB of system RAM, and 80GB of free SSD space exceed what a typical laptop provides. AMD GPU support is limited to experimental support on Linux.
Q. Is Strata free to use?
A. The engine itself is free under the MIT license. However, the model has a separate license, so please check the terms before commercial use.
Q. Will I always get 94 tokens per second?
A. No. That figure represents the best-case conditions: an RTX 5070, 64GB of RAM, and the lightest quantization. Using a larger model or longer context will reduce speed.
Q. Can it run with 32GB of RAM?
A. The official minimum is 32GB. However, 64GB is recommended for running all available model sizes. The "Coder" variant, which uses half the experts, is said to fit within 32GB.
Q. Can I connect it to my existing AI tools?
A. Yes. Both OpenAI-compatible and Anthropic-compatible APIs are available, so you can connect compatible coding tools.
Q. Does accuracy suffer?
A. Quantization (compression that reduces model size) is applied, so smaller variants may have lower accuracy. PC Watch reports that IQ3_S matched a full-precision model on published benchmarks.
Start by checking your PC's VRAM and RAM. If your system meets the requirements, try the smallest variant first.
This article is a cross-post from AI Friends.
