First, I apologise to the abstractions or gods that hold me accountable to weeknotes, as I am writing this on Monday morning (“the morning after”) on account of a Sunday riveted to another digital undertaking, about which more soon. I took a breather from Rheo development this week to participate in the second week of an online interpretability intensive at the Digital Theory Lab run by Tyler Shoemaker. I have been collaborating with Tyler for a year or so now on research that is nearing publication readiness, and it was great to see him in his element: namely, a Jupyter notebook.
I also got set up with OBS so that I can livestream some research work later this month. I played around with some livestreams this time last year, and despite their resplendent inconspicuousness in the world of views and likes, I nevertheless found the exercise very helpful. (The phrasing ‘even though… I nevertheless’ seems to be something of a pattern in evaluations of my publicly posted research.) So I will take to the binary skies again in the coming weeks, likely with passengers like Rheo and an agential harness in tow.
Tyler’s summer school was designed to bootstrap the technical knowledge necessary for interpretability work for language models. Practically speaking, this means a whirlwind tour of Python and an exhibition of how one can use it in a Jupyter environment to do the archaeology of language models that is now generally called interpretability research.
Tyler’s course is a practical introduction to thinking differently about large language models. The Digital Theory Lab was founded by Leif Weatherby, who wrote about the conceptual stakes of the ‘science’ of language models in his 2025 book, Language Machines: Cultural AI and the End of Remainder Humanism. (For a gentler and less thoughtfully argued introduction to the Lab’s position, I gave a short presentation (with slides) earlier this year problematizing the rhetorics of objectivity in interpretability research. Though there are aspects of my argument that could be more precise, it paints an accurate picture of the thrust of my early thinking about the misunderstandings that traffic in the interpretability mainstream.)
As Director of the Lab, Tyler now leads the small but serious research effort we have going towards the different kind of language model interpretation that Leif calls for in his book. I thought I would listen in to remind myself of what I’m supposed to already know, but—as often happens in ‘introductions’—I learned I need to firm up my sense of a whole lot of things in the space. I went back and started working through the sessions in Tyler’s 2024 course, as even though this 2026 one was supposed to be standalone, I personally find that it works much better as a follow-on from that earlier edition. The 2024 course gives shape to language models as an abstraction by first introducing an n-gram model that doesn’t involve any machine learning, just statistical sampling; whereas the 2026 edition throws you right in the deep end of transformers and tokenization.
I felt like I first got a real grasp on machine learning after working through the fast.ai course Practical Deep Learning back in 2019 or so. Two of the tidbits that have stuck with me from this course are:
As I listened to Tyler fluently talking through various statistical terms and operations, I realized quite quickly that I would need to port these and other aspects of the fast.ai pedagogy to really understand the notebooks in the course. To that end, after struggling to follow through the first few seminars, I set up a local dev environment with all the data in marimo notebooks, as I feel that they have both better UX and functionality than Jupyter notebooks. I have now started rewriting some of these tutorials in that environment, using more descriptive variables (as I have never been comfortable with the Pythonic proclivity for C-like variable abbreviations; I always forget what they’re supposed to mean) and adding markdown cells to explain the operational motive better where I struggled to follow.
I’m now considering streaming some of the content from this ‘literate programming’ translation of the combined 2024 and 2026 courses. But at the very least I intend to make the notebooks available as a supplement for those who similarly have difficulty following mathematical formulae and stylishly terse Python.
A big DTL section this week; as I also did some more work on the OxCaml/Bonsai viewer for the set of experiments in the aforementioned paper. Because it deals with gigabytes of data on disk, I’ve set up a backend server for it as well, and will deploy it this week to a Digital Ocean droplet. This gives me a nice opportunity to try out some of the patterns for NixOS in Production, which has been on my want-to-dos for a while.
I built out the basic structure of the syllabus for this course (described in last week’s weeknotes). It is split between several weeks on the LLM engine component of an agential harness, which comprises the hardware-intensive substrate of running next-token generation with an LLM somewhere, and the harness architecture, which covers the agentic loop, tool use, session context, and the other aspects that make next-token generation practicable as a software development assistant in tools like Claude Code. I found this post a helpful jumping-off point, and also discovered the pi coding agent through my research.
Pi is a fantastically minimal yet extensible coding harness, which reads with respect to Claude Code or Codex as (Neo)vim does to Visual Studio Code. As of yesterday, I’ve now switched over to pi for all my agential programming. Critically, pi’s minimalism makes it practicable for me to swap out Anthropic’s engine for local models running on ollama. (I recognize that there are likely ways to optimize a local LLM engine, but ollama with ROCm has been sufficient for me to get started seriously using open weights models locally.) I’ve been playing around with qwen3.6:35b, a model that generates tokens remarkably promptly on my Framework Desktop. I haven’t leaned in on it yet, but I expect that Qwen 3.6′s performance and capability will be sufficient to work on features and bugs filed by more capable models, a plan-implement workflow that I currently manage with beads and jj.
Open models are an increasingly exciting space, both for interpretability and agential harness research. I highly recommend the latest Oxide and Friends podcast with Simon Willison on open weights models and the recent Anthropic/Hugging Face incident. If you’re not already jazzed about open weights, I expect you will be after listening.
I went on a bike ride and a few runs this week, but the August heat is really starting to dig in its heels. In an attempt to quarrel with it, I’ve been experimenting with artificially creating various airflows around my apartment by positioning fans near windows and our air conditioner, which unfortunately combats the 30-degree-plus norm only in its direct proximity, rather than across all of our home. (Our roof is old and doesn’t insulate as well as it should.)
I also renamed my main blog to ‘one long token soup’ after reading this scrumptious phrase in an interesting new paper from ICML. I apologise if this refreshes or otherwise upsets your RSS reader!