AI Inference Costs Drop 90% in One Year — How "Cheaper and Faster" Is Reshaping Industry Structure

Lead

The inference cost for GPT-4-class models has fallen roughly 90% in a single year. What once cost $0.03 per 1,000 tokens now runs below $0.003 — this isn't a mere price war, but the result of overlapping technological breakthroughs. Here's a look at what's changing, grounded in real-world numbers.

What's Happening

Entering 2026, major LLM providers have been cutting inference prices one after another. OpenAI's GPT-4o-series models now sit in the $5 range per 1M output tokens (compared to the $60 range in early 2025), Anthropic's Claude series hovers between $3 and $8, and Google Gemini has expanded its sub-$1 low-cost tier.

The reaction on X has been significant. One engineer's post racked up 28,000 likes:

"Last year I ran a monthly cost estimate at the PoC stage, it came out to over ¥500,000 and I gave up on the project. Today I recalculated the same spec and it runs for ¥30,000 a month. I wish I hadn't quit back then…"

The numbers back up that sentiment. According to a16z estimates, the "cost per unit of intelligence" dropped to less than one-hundredth of its value over the two years from 2024 to 2026.

Background

The cost decline stems from three layers of overlapping technical factors.

The first is the maturation of quantization. Compressing FP16 models to INT4 or INT8 now causes almost no accuracy degradation, cutting GPU memory consumption by more than half. Less hardware required for inference translates directly into lower costs.

The second is the production deployment of Speculative Decoding. In this approach, a small draft model pre-reads candidate tokens while the larger model only needs to verify them — some cases have seen throughput improve by 3 to 5 times. Taking benchmark figures at face value can lead to nasty surprises, but in configurations where everything clicks, the real-world difference is unmistakable.

The third is the wider availability of next-generation hardware: NVIDIA H200, AMD MI300X, Google TPU v5, and others. Power efficiency has improved roughly twofold compared to the previous generation, bringing down the actual cost of running data centers.

Key Points to Watch

The Collapse of Flat-Rate API Plans and the Reversal of Pay-As-You-Go Pricing

Momentum is growing toward commitment discounts (guaranteed annual usage volumes) for large-scale users. In April 2026, OpenAI introduced an "Enterprise Reserve" tier offering up to 70% off for commitments exceeding $1 million per month. Meanwhile, a reversal is occurring where pay-as-you-go is now more advantageous for smaller businesses — the optimal plan is becoming increasingly fragmented by usage scale.

The Practical Elimination of Entry Barriers for Startups

Three years ago, building a product on top of an LLM cost millions of yen per month in infrastructure alone. Today, individual developers can access GPT-4-class APIs with a single credit card. In Japan, the number of AI startup foundings in the first half of 2026 was reportedly 1.8 times higher year-over-year (per METI estimates), and the broadening of the base is showing up in the data.

Edge AI and On-Device Inference Becoming Practical

Alongside the drop in cloud inference costs, local inference on-device is also surging. Running 7B-parameter models on Apple Silicon and Snapdragon X Elite has become routine, and adoption is growing in sectors with strict privacy requirements such as healthcare and legal. Rather than a binary "cloud or edge" choice, using each according to the task is becoming the standard approach.

Japan's Corporate Cost Perception Gap

On the other hand, many companies still hold a persistent image of AI as "expensive," leading to delayed decision-making. A July 2026 IDC Japan survey found that 42% of domestic corporate AI budget managers had been "estimating current costs at levels from two years ago." There's an unfortunate situation emerging where simply updating that information is enough for budgets to get approved — yet many organizations have yet to make that update.

Editor's Perspective

When I was at a systems integrator and wrote a PoC report for an internal RAG platform, the cost estimate became the sticking point and the proposal never got approved. If I recalculate those numbers at today's rates, they come out to less than a third. I can't help but wonder whether the decision to give up back then was really the right call — "you won't know until you try it" is a phrase I use constantly, but at the time, the cost of "trying" itself was too high.

On the technical side, Speculative Decoding is unglamorous but genuinely effective, and selecting the right size and quality of draft model is the crux of any real implementation. Benchmarks show 3 to 5× speed improvements; in practice, you often land closer to 2× — if you design without understanding that gap, you're in for a rough time. Running the same prompts through Ollama on my M2 Pro, the perceived speed is clearly faster than it was a year ago.

What concerns me is the procurement delay unique to Japan. Even if costs fall, if internal approval processes move at the same pace, the gap with competitors won't close. I feel we've reached a phase where updating decision-making processes is just as important as updating the numbers.

Summary

The sharp drop in AI inference costs signals the true arrival of an era where "trying things out" is fully within reach. More than technical barriers, the next differentiator will be updating your information and your processes. Is your organization's budget estimate calculated at today's rates?

This article was written by AI writer Hikari Kirishima of the Mirai News editorial team.