I’ve written about what I call Tooling for Tooling, basically meta-tools that help people build better developer tools faster and easier. Arguably the most famous and ubiquitous meta-tool is not actually a tool—it’s the Language Server Protocol (LSP). By defining a protocol for language implementations to communicate with editors, the LSP lets language authors write a single LSP server and let their users bring any editor that implements an LSP client. Similarly, editor authors only need to write a single LSP client and let their users bring any language server of their choice.
This is really fantastic! Previously if you used a niche editor or language, it was highly unlikely that you’d get good language integrations. At best you’d get syntax highlighting and some primitive tagging system. Now it’s very common to expect solid go-to definition, semantic syntax highlighting, and other niceities previously restricted to IDEs on mainstream languages.
However, the LSP does have its limits. It’s fundamentally a tool for
editors, so the abstraction is a persistent server that needs to be
started and queried. This makes sense for an editor, but it doesn’t
scale well into other domains such as CLIs or web services. This also
doesn’t necessarily play well with how tools are implemented under the
hood. A lot of tools are originally conceived and implemented as
one-shot systems, i.e. you run
clang and wait until the
process ends. Usually this process takes a few seconds to run. Whereas
a language server is queried repeatedly as the code changes, and is
expected to return an answer within a few hundred milliseconds (or
risk providing a poor user experience). This usually necessitates some
amount of caching or incremental computation, which is often awkwardly
tacked on post-facto.
There are attempts to extend the protocol with persistence (LSIF and SCIP). I’m pretty excited about these as a way to offer better static tooling, i.e. a supercharged GitHub code navigation. But persistence is only part of the story.
What I would really love is a way to extend this protocol beyond editors. Specifically, I’d love for a protocol that handles the other ways in which we invoke tools. There are a few ways that I’ve encountered: as a watcher, as a CI process, and as part of a build graph. Each of these ways are essentially front-ends to how we use a tool. And yet each of them are either implemented in an ad-hoc, limited fashion, or are completely separate to the tool and only interact with it at a very high level.
Let’s start with the watcher. In a lot of tools we use watch mode, basically a mode where the tool watches a set of files and re-runs when they change. This could be a dev server, a linter, a compiler, really anything that processes your code.
This may seem like a relatively simple process. But it’s really
not. For one, consistent, efficient file watching is surprisingly
tricky. At Vercel, we did a lot of work to get our file watching code
to behave the same across operating systems. Turns out there’s a lot
of subtle differences in how OS’s implement file system events. It’s
also very easy to do a simple action, like move a directory or delete
node_modules and end up with a cascade of events that destroy your
app’s performance, or worse, cause an out-of-memory error.
Second, when we talk about file watching, we usually mean something a
little more sophisticated than just watching a list of files and
returning changed or unchanged. Usually you want to have globbing to
declare the set of files that are being watched, as well as which
files to exclude. You probably want to filter using the
files, unless you want users to accidentally get terrible performance
out of the box because you ended up watching
dist. And you likely want some way of surfacing which files have
changed, so the tool can make an intelligent decision about what to
do. Maybe it can only rebuild those files. Or maybe it can rebuild a
single module instead of multiple modules.
With CI, there are similar but different requirements. Instead of file watching you probably want some degree of change detection. What files have changed in this pull request or commit? That may influence which tools are even being run. Which in turn may influence what your build environment may be. Do you really want to install Node.js if you don’t need to run ESlint?
Likewise, you want some way of surfacing errors. For most CI systems, they just show you stdout and stderr. I never understood how that could be a good user experience. Errors could be so much richer and dynamic. You could provide a link to the relevant code in GitHub or wherever it’s stored. You could have an automatic fix button—or heck, if the tool indicates that all of the errors are fixable, it could enter a self-healing loop. You could keep a database of errors and how they were solved previously. You could link and share them to your coworkers. You could link to the error code index for the language. Errors are not just text, they’re deeply hyperlinked structures.
With Turborepo, we’re executing different tools like ESLint or Next in the build graph. In some cases, we’d like the ability to integrate with a tool at a deeper level. For instance, with a testing library, it would be nice to surface UI prompts to the user for interaction. Or maybe we want errors to appear in our Vercel Spaces UI. Or going even deeper, maybe we want the ability for a tool to provide data that another tool takes in. All of this is theoretically feasible with our current methodology of forwarding stdin and stdout, but it could be a lot nicer with a proper protocol.
In each of these cases, we need a way to interact with a tool that is more sophisticated than stdin, stdout, stderr. Basically, we need a protocol around tooling inputs and outputs.
This protocol would help tools receive inputs in smarter ways by letting them subscribe to specific changes, watching files, and even integrating with version control. Once the tool has received these inputs, the protocol would also help with outputs, say by event sourcing errors in a consistent schema, so they could be consumed by a web UI, by a terminal, and by an error database.
In a really ambitious case, all of these protocol messages could be backed by a content addressed store, which could allow you to replay a tooling execution, either remotely or locally.
I’d love to build this. If you’re interested too, send me an email at email@example.com.