I Ran a 135M-Parameter LLM Inside SQLite

I made SQLite generate text with SmolLM-135M. The tokenizer, transformer, KV cache, and next-token loop are SQL.

The original rule was deliberately unreasonable: the entire inference engine had to be SQL, with no Python or JavaScript quietly doing the math offstage. The result is a working SQLite database that loads the published model weights, processes a prompt, and emits decoded text. Starting from Hello, a tested run printed , and then I.

There is a cost to the joke. After one optimization, the first-token run took about 44 seconds on the build machine. The same test had taken about 90 seconds before. This is a language model you can inspect with SELECT; it is not a language model you would choose to chat with all afternoon.

SQLite is both the storage engine and the executor. The authored inference path consists of tables, views, recursive queries, and triggers. The sqlite3 program submits statements and prints rows; it does not implement tokenization or transformer operations for the model. SQLPage provides a local dashboard, but the page is only a front end to the same database.

The published safetensors file and tokenizer are inputs. A one-time SQL import reads the weight file, parses 272 tensors, and materializes 134,515,008 numeric weights into indexed SQLite rows. The resulting accelerated database is about 4.2 GB, compared with roughly 528 MB for the upstream weight file. I traded compactness for a representation that SQL can multiply without calling a host-language function.

Once imported, a prompt goes through Unicode pretokenization and byte-level BPE. Token IDs become embeddings, then pass through 30 transformer layers. The final projection scores a vocabulary of 49,152 tokens; the largest logit wins. There is no sampling temperature here: generation is greedy.

Tokenization is the first place a “SQL-only model” can cheat without anyone noticing. A transformer accepts IDs, not characters. If a script outside the database turns the prompt into IDs, the hard part of the input path has simply moved out of view.

Here the tokenizer lives in SQL too. Text is split into pieces; bytes map to tokenizer symbols; adjacent symbols are repeatedly merged according to BPE ranks. In the implementation, inserting a piece into bpe_piece initializes rows in bpe_symbol. A trigger finds the best ranked adjacent pair, updates the left row, and deletes the right one. It keeps doing that until no ranked pair remains. A final join with tokenizer_vocab yields token IDs.

The small figure below uses a toy vocabulary to show the operation. Its merge ranks are illustrative; they are not the model's actual tokenizer table.

The hidden state is a table. Each row holds a dimension and a value. Inserting a token into inference_input loads its 576-dimensional embedding and inserts a job for layer zero. The layer trigger runs RMSNorm, query/key/value projections, rotary position encoding, grouped-query attention, the output projection, and a SwiGLU feed-forward block. It then inserts the next layer job. After layer 29, a final job computes logits.

The KV cache is also ordinary SQL data: rows keyed by layer, token position, head, and dimension. When attention scores the current token, its query joins cached keys from positions up to the current one. A windowed softmax converts scores to probabilities; another join weights the cached values. That is real causal attention, expressed in a form a query planner can execute. It is also a lot of rows.

The core projection is a matrix-vector multiply. Each output row is a dot product between a weight row and the current input vector. The interactive example uses three rows and three input values; the model uses matrices with hundreds or thousands of columns.

I first grouped all weight products by output dimension. SQLite's query plan built a temporary GROUP BY tree for each projection. The faster trigger instead enumerates output rows and performs a bounded weight-range scan for each one. The SQL is still a dot product; the planner now has an easier path to it.

SELECT r,
  (SELECT sum(w.value * i.value)
   FROM weight_real AS w
   JOIN matrix_input AS i
     ON i.dim = w.linear_index - r * spec.cols
   WHERE w.tensor_id = spec.tensor_id
     AND w.linear_index >= r * spec.cols
     AND w.linear_index < (r + 1) * spec.cols)
FROM rows CROSS JOIN spec;

In one local Hello first-token test, that change took runtime from roughly 90 to 44 seconds, with the same output. That is about a 2× speedup, not a general benchmark across machines or prompts. The database still visits roughly 135 million numeric weight rows per generated token. Weight layout is the next obvious bottleneck.

The first working CLI example returned a token ID. That proved the model was choosing something, but it was a poor interface for a text generator. The streaming script now selects only the newly decoded text after each inference step. A small cursor records how many output bytes have already been printed, so each SELECT emits the delta.

sqlite3 outputs/smollm.db < outputs/stream.sql

With the script's Hello prompt, the verified first two chunks were:

,
 I

Those are chunks, not necessarily words. Subword tokenization can produce punctuation, a leading-space word, or a fragment. The dashboard displays the same decoded stream beside the prompt and lets a local user request one to eight output tokens. Its pages are SQLPage files, while model inference remains in SQLite SQL.

The engine has been exercised through all 30 layers in a two-token Hello run, including its KV cache. Import checks found the expected 272 tensors and 134,515,008 weights; SQLite's integrity check returned ok. This establishes a working path, not broad language-model quality testing. I have not benchmarked a representative prompt suite, long contexts, or generation on other machines.

The model supports its 2,048-token context and greedy decoding. The accelerated database takes gigabytes of disk, and every new token is slow enough to notice. That is the trade: SQL makes the state inspectable down to individual weights and cached values, but the row-oriented representation is expensive for dense arithmetic.

I am interested in what happens if the weights become packed blocks instead of 135 million separate numeric rows. That would make SQLite do less bookkeeping per multiplication, though it would need a new correctness check against the current engine. For now, the odd thing works: you can ask a database for the next word, and the database has actually done the transformer math.