Last but not least, I started a new project this week in the distributed systems space. I’ve decided to wait and see what comes of working on it for a few weeks before I say too much more, but I’m quite excited about it. My last serious job before grad school was at Halter working out the best way to build a feature store as an application running on Apache Flink, and throughout my PhD I’ve cast around in the streaming execution engine and vector database space for something that would stick. Part of the problem I had in this search, I realize now in retrospect, is that Flink and similar distributed systems are typically built out for the purpose of large-scale, industrial data processing, where data is processed as soon as possible after ingested (in ‘real time’).
Karaji—the name I’m giving to this new system—is something I’ve started building to address a personal use case, not a hypothetical business one. I started writing Rheo for the same reason—to do something with my computers that I hadn’t worked out how to do with existing software in a way that worked for me. It seems obvious to say (and this advice is widely circulated in engineering communities): but personal software is the most inspiring software to write, as there’s no better motivation for making something work well than wanting to use it yourself.