A raw, timestamped record of progress, dead ends, and small ideas. Written the same day, edited only for spelling. If you want the polished version, read the essays.
Rewrote the eval loop so it runs against a live vector index instead of a static snapshot. Numbers moved: p50 latency from 780ms → 460ms, hallucination rate on the internal QA set from 8.1% to 3.2%. Interesting failure mode: it now confidently refuses questions it used to hallucinate — feels like progress but I want to measure the refusal quality separately.
You can feel it before you see it in the numbers. Loss keeps dropping, val loss looks fine, but the model's answers start getting too specific — quoting phrasings from training data verbatim, losing the ability to paraphrase. Three signals I now look for: (1) drop in n-gram diversity, (2) rising log-likelihood on train-set paraphrases vs. novel ones, (3) qualitative — does it start sounding like a librarian who's memorised the index?
340M params, 12B tokens of Tamil poetry + prose. First readable output at step 42k. It rhymed. Badly. But it rhymed. Saving the checkpoint. Tomorrow I'll try a smaller LR and longer context.
Tamil compound verbs are getting split into 4-6 subwords each, which is wrecking the loss curve on longer phrases. Considering a morphologically-aware BPE or just curating a merges file by hand. Neither sounds fun.
Norman's mapping principle keeps rearranging how I think about API design. If a user has to remember the shape of your function signature, you've already failed. Next up: 'Thinking in Systems' by Meadows.