Tyler’s summer school was designed to bootstrap the technical knowledge necessary for interpretability work for language models. Practically speaking, this means a whirlwind tour of Python and an exhibition of how one can use it in a Jupyter environment to do the archaeology of language models that is now generally called interpretability research.
Tyler’s course is a practical introduction to thinking differently about large language models. The Digital Theory Lab was founded by Leif Weatherby, who wrote about the conceptual stakes of the ‘science’ of language models in his 2025 book, Language Machines: Cultural AI and the End of Remainder Humanism. (For a gentler and less thoughtfully argued introduction to the Lab’s position, I gave a short presentation (with slides) earlier this year problematizing the rhetorics of objectivity in interpretability research. Though there are aspects of my argument that could be more precise, it paints an accurate picture of the thrust of my early thinking about the misunderstandings that traffic in the interpretability mainstream.)
As Director of the Lab, Tyler now leads the small but serious research effort we have going towards the different kind of language model interpretation that Leif calls for in his book. I thought I would listen in to remind myself of what I’m supposed to already know, but—as often happens in ‘introductions’—I learned I need to firm up my sense of a whole lot of things in the space. I went back and started working through the sessions in Tyler’s 2024 course, as even though this 2026 one was supposed to be standalone, I personally find that it works much better as a follow-on from that earlier edition. The 2024 course gives shape to language models as an abstraction by first introducing an n-gram model that doesn’t involve any machine learning, just statistical sampling; whereas the 2026 edition throws you right in the deep end of transformers and tokenization.
I felt like I first got a real grasp on machine learning after working through the fast.ai course Practical Deep Learning back in 2019 or so. Two of the tidbits that have stuck with me from this course are:
As I listened to Tyler fluently talking through various statistical terms and operations, I realized quite quickly that I would need to port these and other aspects of the fast.ai pedagogy to really understand the notebooks in the course. To that end, after struggling to follow through the first few seminars, I set up a local dev environment with all the data in marimo notebooks, as I feel that they have both better UX and functionality than Jupyter notebooks. I have now started rewriting some of these tutorials in that environment, using more descriptive variables (as I have never been comfortable with the Pythonic proclivity for C-like variable abbreviations; I always forget what they’re supposed to mean) and adding markdown cells to explain the operational motive better where I struggled to follow.
I’m now considering streaming some of the content from this ‘literate programming’ translation of the combined 2024 and 2026 courses. But at the very least I intend to make the notebooks available as a supplement for those who similarly have difficulty following mathematical formulae and stylishly terse Python.
A big DTL section this week; as I also did some more work on the OxCaml/Bonsai viewer for the set of experiments in the aforementioned paper. Because it deals with gigabytes of data on disk, I’ve set up a backend server for it as well, and will deploy it this week to a Digital Ocean droplet. This gives me a nice opportunity to try out some of the patterns for NixOS in Production, which has been on my want-to-dos for a while.