I Ran a 135M-Parameter LLM in Verilog

I made SmolLM-135M continue a prompt using a SystemVerilog program. The tokenizer, 30 decoder layers, KV cache, and text decoder all live in that source.

The tested prompt Hi produced ,. A second check, The capital of France is, produced Paris. Those are small outputs, but they went through the full 135-million-parameter model and agreed with an independent Torch reference for the tested next tokens.

Here is the crucial boundary: this is behavioral SystemVerilog running in the Verilator simulator. It uses real arithmetic, strings, and file I/O. It is not synthesizable FPGA or ASIC logic. The experiment asks whether the entire model path can be expressed and executed in SystemVerilog, not whether the current source is a deployable chip design.

A language model needs more than matrix multiplication. It needs a tokenizer to turn text into IDs, a way to read checkpoint weights, position-aware attention with a cache, a vocabulary projection, a token choice, and a decoder to turn that choice back into text. Moving any one of those steps into a helper script would make the claim less interesting.

The source file smollm.sv owns that path. It reads prepared lookup tables with $readmemh, opens the published model.safetensors checkpoint, decodes F32 weights into Verilog real values, and runs the transformer. Verilator compiles the SystemVerilog into a local executable. The optional Prompt Studio dashboard has a Node server that starts the executable and relays progress; Node does not tokenize text or choose tokens.

The checkpoint is 538,090,408 bytes. Its configuration has a 576-wide hidden state, 30 decoder layers, nine query heads, three key/value heads, a 49,152-token vocabulary, and a 2,048-token context limit. That is a lot of arithmetic to ask of a simulator designed for a different kind of work.

The tokenizer handles Unicode/ByteLevel splitting and byte-pair merges before inference starts. In each piece, the code searches adjacent token pairs for the best ranked merge, replaces that pair with its merged token, and repeats. It also recognizes the checkpoint's special tokens. A +tokenize_only mode prints IDs without opening the model checkpoint, which is useful when checking the input path separately.

For each prompt token, the simulator runs a full forward pass to fill the KV cache. This is prefill. Only after the last prompt token does it have a first next-token prediction to print. That explains a moment that looked like a hang in the original run: a six-token prompt can take roughly two minutes before any generated text appears. The current executable prints PROGRESS: prompt token 1/6 and subsequent steps so the wait is visible.

The timeline below uses the measured rough rate of 20 seconds per prompt token to show why prompt length matters. It is a guide to the observed wait, not a timing guarantee.

The public Git repository contains code and lookup tables, while the 538 MB checkpoint is downloaded separately. A static table of 272 tensor offsets points at the exact checkpoint revision; the README gives its SHA-256 digest because a different file could make those offsets read the wrong bytes.

Keeping every matrix expanded in memory would make the simulation awkward. The program keeps the tied 49,152 × 576 embedding resident, then loads other matrices as each layer needs them. A layer has nine tensor slots: two norm vectors; query, key, value, and output projections; and the three feed-forward projections. The source computes b = 1 + layer * 9 to find that layer's slots.

The diagram lets you inspect those slots. The loaded data is still the original checkpoint's F32 values, converted to real for the behavioral calculation. This is not quantized inference.

Inside each layer, the model normalizes the hidden state, projects queries, keys, and values, and rotates query/key pairs with RoPE. It then stores the current key and value at [layer][position][dimension]. Future tokens reuse those entries instead of recomputing every earlier token's projections.

SmolLM-135M uses grouped-query attention: nine query heads share three key/value heads. The mapping in the code is integer division, (query_head / 3). Click a query head below to see which cache it reads. Each head scores previous positions up to the current one, applies a stable softmax, and forms a weighted sum of cached values. After the attention output projection and residual addition, a SiLU-gated MLP updates the hidden state.

The final RMSNorm feeds a tied vocabulary projection: the same embedding matrix used at input produces 49,152 logits at output. A loop keeps the largest logit and emits its token bytes. Generation is greedy, with no temperature or sampling. On the tested Mac, the short-prompt one-token run took roughly 21 seconds. Longer prompts add roughly one model pass per prompt token.

verilator --binary --timing -O3 -Wno-fatal -j 4 smollm.sv
./obj_dir/Vsmollm '+prompt=Hi' +max_new=1

The second command printed OUTPUT: ,. The independent Torch checks covered that next token and The capital of France is → Paris. The dashboard was also tested end to end with Hi → ,, including its local process bridge and browser output.

The build proves a prompt can enter a SystemVerilog implementation and come back as decoded model text. It does not establish parity across a large prompt suite or measure output quality. The base SmolLM checkpoint continues text rather than following chat instructions, and greedy decoding makes its choices deterministic.

The current program also cannot be synthesized as written. Turning it into hardware would require replacing simulation file I/O, dynamic strings, and real arithmetic with explicit memories, fixed-point or floating-point units, and a schedule for moving weights through them. That would be a different engineering project, with separate area, timing, and numerical-accuracy tests.

The interesting surprise is where the work went. The matrix loops are direct. Preserving the tokenizer, checkpoint layout, cache, and byte output inside the language took more care. A language model is a whole path from characters back to characters; moving only the multiply-adds would have left most of the experiment outside the frame.