Small language models next to the production line: what fits on the edge in 2026

What counts as a small language model

There is no official cutoff. The NVIDIA paper discussed below defines an SLM by where it runs: a model that fits on a common consumer device and answers one user’s requests with practical latency, which in 2025 meant most models below 10 billion parameters. Vendor guides put the range at a few hundred million to about 10 billion. Large language models start in the tens of billions and now pass a trillion: DeepSeek’s V4-Pro, released in April 2026, has 1.6 trillion parameters, 49 billion of them active for each token.

That also settles a common question about DeepSeek. Its flagship models are LLMs by any definition, but its R1 release in January 2025 included distilled versions from 1.5 to 70 billion parameters, and the smallest of those are SLMs.

Three techniques do most of the work of making a model small:

• Knowledge distillation: a small student model is trained to reproduce the answers of a large teacher, which is how the R1 distilled models were made.
• Quantization: weights are stored at lower precision, for example 4 bits instead of 16, so the same model needs a fraction of the memory.
• Pruning: weights, neurons or whole layers that contribute little are removed, usually followed by more training to recover accuracy.

How small can they go? Hugging Face’s SmolLM2 family starts at 135 million parameters, trained on 2 trillion tokens and released in an instruction-tuned version. Samsung’s Tiny Recursive Model has only 7 million and scored 45% on the ARC-AGI-1 puzzle benchmark in October 2025, but it is a puzzle solver, not a model you can talk to.

The trade against an LLM is breadth for cost and control. An SLM runs on a single GPU, a CPU or the device itself, answers faster and keeps data on site. It knows less about the world, is weaker at long multi-step reasoning, and needs testing outside the task it was tuned for.

The argument for small

NVIDIA Research, with Georgia Tech, published a position paper in June 2025 titled Small Language Models are the Future of Agentic AI (revised most recently in September 2026). Its claim is narrow and worth reading literally: agentic systems mostly call a model to perform a small number of specialized, repetitive tasks with little variation, and for those calls small models are sufficiently powerful, inherently more suitable and necessarily more economical. Where open conversation is genuinely needed, the authors prefer heterogeneous systems that call several models rather than one large one, and they sketch an algorithm for converting an LLM-based agent to small models.

The paper puts the serving cost of a 7 billion parameter model at 10 to 30 times lower than a 70 to 175 billion parameter one, in latency, energy and compute. In three open-source agents it examined, the authors estimate that specialized SLMs could reliably handle about 60% (MetaGPT), 40% (Open Operator) and 70% (Cradle) of the LLM queries.

It is a position paper, not a benchmark study, and it comes from a company that sells edge hardware. The reason to take it seriously is that the economics are easy to check yourself: measure how many of your model calls are classification, extraction or scoring, and you will usually find that the frontier model is doing clerical work.

What the open small models now do

In 2026 both halves of the edge problem moved: the open models got small enough, and the hardware beside the machine got large enough. Google released Gemma 4 at the end of March 2026 under an Apache 2.0 license, in four sizes: two edge models called E2B and E4B, a 26 billion parameter mixture-of-experts model and a 31 billion parameter dense model. The edge models carry a 128K token context window, the larger ones 256K.

The numbers that matter for a factory floor come from Google’s own publication: E2B can run in under 1.5 GB of memory using 2-bit and 4-bit weights with memory-mapped per-layer embeddings; on a Raspberry Pi 5 CPU it decodes 7.6 tokens per second; on a Qualcomm Dragonwing IQ8 with NPU acceleration, 31 tokens per second. All Gemma 4 models take text and images and can analyze video as sequences of frames, and the two edge models also accept audio. Deployment targets include Android and iOS, Windows, Linux, macOS, browsers through WebGPU, and embedded boards.

Gemma 4 is one of several open families at this size. Alibaba’s Qwen3.5-0.8B (Apache 2.0) takes text, images and video with a 262,144 token context window and covers 201 languages and dialects. Microsoft’s Phi-4-mini has 3.8 billion parameters, an MIT license and a 128K context. Meta’s Llama 3.2 comes in 1 and 3 billion sizes, and Hugging Face’s SmolLM3 at 3 billion publishes its full training recipe.

That is enough for a station that reads a label, listens to a machine, parses a document or decides which of five actions to take, with no network at all.

Robotics went local too

The same shift is visible in robot control. Google DeepMind’s Gemini Robotics On-Device 2, part of the July 2026 release, is built to run on the robot without connectivity and to adapt to a new embodiment in a few hours. NVIDIA’s open GR00T 1.7 is a 3 billion parameter vision-language-action model, small enough to run on board, which reports 96.5% average on the LIBERO benchmark suite against 87% for its predecessor.

For a machine that moves, local inference is not a preference. A network round trip in the control loop is a safety problem.

The hardware beside the machine

NVIDIA’s Jetson AGX Thor is the current reference point for industrial edge compute. The T5000 module is rated at up to 2,070 FP4 TFLOPS with sparsity, measured at 130 W, with 128 GB of LPDDR5X memory at 273 GB/s and a power envelope of 40 W to 130 W. NVIDIA claims 7.5 times the performance and 3.5 times the energy efficiency of the previous AGX Orin generation. A smaller T4000 module is listed at 1,200 TFLOPS with 64 GB.

Put next to the Gemma 4 figures, the range becomes clear. A Raspberry Pi class board can host a small model for a single decision. A Jetson class module can host vision models, a small language model and a control policy on the same box, inside an enclosure on the line.

Below those sit two smaller classes of edge AI hardware. Plain microcontrollers, with tens to hundreds of kilobytes for a model, handle keyword spotting or simple vibration anomalies. Microcontrollers with a small neural processing unit (NPU) take small vision and audio models within about a watt. When comparing parts at any level, the advertised TOPS figure predicts less than memory bandwidth, whether the accelerator supports every layer of your model, and sustained performance inside a closed enclosure at the hottest temperature on site.

Why this is the right place for inspection AI

Inspection is the clearest case for edge inference. The decision has a deadline set by the line speed, the input is a stream of images nobody wants to ship off site, and the value of the decision drops to zero a second later.

Our own X-ray inspection was built this way from the start: it runs on GPU, CPU or VPU, works with the X-ray generators and line-scan detectors already installed, and keeps the images in the customer’s own data cloud, while a central server manages the machines and collects detection statistics. At Milarex that combination is what let the lines speed up from 1.2 to 2 m/s instead of slowing down for the inspection step.

What this means in practice

If you are planning AI on a line or in a vehicle, three questions settle the architecture. What is the deadline for the decision, in milliseconds. What happens when the network is down, and is that acceptable for an hour. And who is allowed to see the raw data. Answer those honestly and most industrial use cases land on the edge, with the cloud used for training, fleet management and reporting rather than for inference.

Then size the model to the decision rather than the other way round. A 2 billion parameter model that answers in 40 milliseconds beside the machine is worth more than a frontier model that answers in two seconds somewhere else. That is the principle behind Firecrux X-Ray QC and the rest of the AIRE engine: real-time response, next to the signal.