Notes on .NET, XAF, and building things
Twenty years of XPO, application frameworks, and lately a lot of applied AI. 350 articles and counting.
Browse by topic
All topics →Latest articles
View all →
What Are Embeddings? King − Man + Woman, Tested for Real
An embedding turns a word or a sentence into a list of numbers where similar meanings land close together. The famous party trick is king − man + woman = queen. I tested it on real vectors: classic GloVe gets queen, and so does a modern LLM's embedding table, but each fails somewhere interesting. Then the useful part: a sentence about rain in English, Spanish, Russian and Chinese lands in almost the same place, even though the tokenizer cut each one into completely different pieces.
What Is Attention? How a Word Decides Which Other Words Matter
Attention is how a language model lets each word look back at the others and decide which ones matter. I read the real attention weights out of a small open model for 'The animal didn't cross the street because it was too tired' and the same sentence ending in 'wide'. In one head, 'tired' puts 73% of its attention on 'animal'. Change one word, and 34 of the 336 heads switch toward 'street'. Plus the surprise: 61% of all attention goes to the very first word.
What Is a Perceptron? The Smallest Piece of Every AI Model
A perceptron is the smallest decision-maker in AI: multiply the inputs by weights, add a bias, and say yes or no. Frank Rosenblatt described it in 1958. One perceptron can learn AND and OR in a few steps, but it can never learn XOR, and that limit shaped the history of AI. Add one hidden layer and XOR falls. Here's the experiment, run on my Mac, and a playground where you can train one yourself.
Putting It Together: Watch a Model Pick Its Next Word, Through Its Weights
Tensors, tokenizers, weights: this is where they meet. I recorded what a real model (Qwen3.5-2B, on my Mac) does for every word of one answer: the tokens it reads, the embedding row it looks up, what it would say after each of its 24 layers, and the 248,320 probabilities it chooses from. Press play and watch it decide, including the exact moment it went wrong, and the two layers where it was thinking in Chinese.
What Are Open Model Weights? Open Weights vs Open Source, and How to Use Them
When a company 'releases a model', what it usually gives you is the weights: a big file of numbers. That's not the same as open source: the training data and code usually stay private, and every model comes with its own licence. I opened a real weights file with ten lines of plain Python, then ran the model two ways on my Mac. Here's what open weights are, what 'open' really means, and how to use them.
More topics
All topics →
A 2B Model Invents El Salvador, Until You Hand It the Page
Asked about El Salvador with no material, a 2B model invents the answer to 84–94% of 4th-grade civics questions: a school project with a made-up name, a Lenca language it thinks is an endangered species. Hand it the right page and the same model gets 98–100% right, and says 'no está en mi material' when the page doesn't have the answer. Knowledge belongs in the library, not in the model. Part 3 of Narrow Intelligence.
Small Models Give Away the Answer
I asked four small models to tutor El Salvador's 4th-grade math, and every one of them gave away the answer: 49–100% of replies stated the final result before the student had tried. Adding one line to the prompt, 'don't give the final answer', fixed it for the 4B and the Gemma E2B (0–2%). The 2B still leaked 56–60% of the time, while telling the student to find the answer themselves. At the size that fits a 4 GB tablet, the teaching behaviour has to be trained. Part 2 of Narrow Intelligence.
Can a Laptop With No Internet Teach El Salvador's 4th Grade?
El Salvador's public-school students take a government laptop or tablet home, but only 37% of Salvadorans have internet at home, 13% in rural areas. So I'm testing a question: can a small AI model, running offline on that device, tutor the official 4th-grade program? Not one model that knows everything, but three parts: a small model for the behaviour, an on-device library for the facts, and checkers for what's right. This is part 1 of Narrow Intelligence: the idea, and how I built test sets that grade themselves.
What Is Tiny Corp, and Is tinygrad Open Source? I Ran It on My Mac
Tiny Corp is George Hotz's company, and tinygrad is its deep-learning framework: a PyTorch-like library small enough to read, with its own compiler. Yes, it's open source, under the MIT licence. I installed it on my Mac Studio: it picked the GPU on its own, fused a 'giant' matrix product into one kernel running at 3.5 TFLOPS, and trained MNIST to 98% in about 11 seconds. Then I benchmarked it against MLX and PyTorch on the same training run: MLX is still fastest, but with its kernel search on, tinygrad's own generated code matched Apple's. Here's what tinygrad is, how the company makes money, and who it's for.
What Is a Tokenizer? Why the Same Sentence Costs More in Russian Than in English
Before a model sees a single tensor, a tokenizer chops your text into pieces. Which pieces depends on the language. I ran the same three sentences in English, Spanish, Russian and Chinese through four real tokenizers: GPT-2 needs 5.4× more tokens for Russian than for English, while Qwen3.5 needs fewer tokens for Chinese than for English. Here's how tokenizers work, why languages pay different prices, and an interactive explainer to see it yourself.