Introducing Toolchain!

Throughout our careers in software engineering, we’ve each worked on a variety of projects, using various programming languages and frameworks, at several different companies. Yet there was always one experience in common: Builds. Just. SUCKED.

In fact, the common bond that unites every developer seems to be that builds are slow, flaky, and fragile, using tools that are opaque, convoluted, and ad-hoc.

The #1 programmer excuse for legitimately slacking off: “My code’s compiling.”
https://imgs.xkcd.com/comics/compiling.png

But wait, this is ridiculous! The tools we use to build software are themselves made of software. Unlike almost all other professions, we have the unique ability to create and improve our own tools. So why don’t we?

Well, now we are! Toolchain is our response to this sad state of affairs: a build system that is fast, stable, straightforward, and uniform.

Before telling you more about Toolchain, it’s worth stepping back to talk a little about builds, and how they reached their sorry state.

What is a build?

The process of transforming the code you write into software that runs correctly involves iterating over many linked steps, such as resolving dependencies, generating code, compiling, type checking, testing, finding errors, bundling, and more.

These processes form the build. The tools that perform them are build systems.

No matter what languages and frameworks we use, builds are all too often:

  • slow, due to resource constraints
  • flaky, due to sensitivity to local conditions on each machine
  • fragile, due to increasing complexity

How did we get here?

Increasing Complexity

Builds are complex processes: they have many moving parts and require careful coordination.

And in recent years, several converging trends have greatly increased that complexity. New programming languages and frameworks, containerization, microservices, the growth of open-source software, and the democratization of software development, have all created a huge need for better build tools.

Build tools have not kept up

Unfortunately, build tools have not kept up with these trends. So the build experience is a far cry from where it should be.

  • Builds are slow, interrupting flow and forcing developers to context-switch.
  • Builds can be inconsistent from one run to the next, especially across machines.
  • Results are hard to interpret, and errors hard to isolate
  • Build tool user interfaces are text-based and outdated.
  • Tooling isn’t uniform across languages and frameworks.
  • Builds depend on changing external state, such as other open-source projects.
  • Build tools don’t expose their data or provide analytics.

Introducing Toolchain

We’re a team of experienced tech industry professionals with a passion for the craft of software development. We’ve worked on build systems for years, so we’re well aware of the deficiencies of existing paradigms (to be honest, we’re responsible for some of those deficiencies…). We’ve created Toolchain to address these challenges.

Toolchain combines remote, concurrent execution and caching with a modern user experience to implement builds that are fast, stable, hermetic, transparent, and a joy to use.

How does it work?

Toolchain leverages the Pants build system’s brand-new "v2" execution engine, written in Rust, which models build steps as standard Python 3 async coroutines with no side-effects.  This leads to consistent, deterministic builds, and is a precondition for reproducible builds. This addresses the flakiness and fragility concerns that plague builds today.

But this execution paradigm also allows build work to escape the constraints of your laptop: work can be executed concurrently and remotely, on our servers, and then cached so that the results are shared across your team. This combination of fine-grained invalidation, caching, concurrency and remote execution yields massive performance benefits, even at scale.

Toolchain provides many useful build rules out of the box, but like many engineering teams, you probably have your own custom build steps. So we've made it easy to write your own build rules, and the invalidation, caching, concurrency and remoting semantics fall out of the design - without you having to think about them!

Running builds and viewing results on the command line is traditional, but limiting. It’s quite literally 1970s technology! So Toolchain also provides a modern browser-based interface that is far better at visualizing complex, concurrent builds and code dependencies than a terminal ever could be. It also provides insight into builds over time, with search, history, and analytics, across the entire organization.

With Toolchain, your builds will be fast, stable, transparent, extensible, and easy to use, keeping you focused on the task at hand.

What’s the current status of Toolchain?

We’ve raised a seed round from a group of investors including Harrison Metal, Homebrew, and prominent angel investors from the engineering community, in order to build out the team and the infrastructure necessary to realize this vision.

Toolchain is now in private beta for Python builds, with support for JavaScript, Java, Scala, and other languages to follow soon. And of course you can always use Toolchain as a platform for your own custom build rules. If you’re interested in more information, please reach out to us at beta@toolchain.com.

Build on!

~ The Toolchain Team

PS Who are we?

  • Benjy Weinberger: A tech industry leader with over 20 years of experience at Google, Twitter, Foursquare and elsewhere. A long-time contributor to the Pants OSS build system.
  • John Sirois: A tech industry veteran with over 20 years of experience at VMWare, Google, Twitter and elsewhere. Created the Pants OSS build system project.
  • Tansy Arron-Walker: A Pants contributor since 2016, who believes that programming is a powerful tool for effecting change.
  • Alex Schmitt: A full-stack engineer who is passionate about lowering the barriers to entry for software development through effective tools.
  • Asher Foa: Has seen it all through having worked on everything from command-and-control systems to web services to build systems.
  • Greg Shuflin: A Rust enthusiast with experience in a wide range of languages and frameworks. Passionate about technologies that assist programmers in writing software correctly and efficiently.
  • Eric Arellano: Worked on build systems at Foursquare and Twitter and led the Pants OSS migration to Python 3. Captivated by change at scale.

See more about us here: https://www.toolchain.com/about

Show Comments