wallzero

Dependency-Driven Auto-Configuration

a

An approach for building shareable tooling configuration that automatically adapts to installed packages.

Inspiration

During the early days of Webpack (shortly before the release of create-react-app), I was working across projects that were all using webpack, babel, typescript, eslint, jest, and more. Over time these configurations became a challenge to update, a challenge to improve, a challenge to test, and changes in one configuration could break another. The repetition of maintaining configurations across several projects became a job itself.

graph TD
  Project1@{shape: doc, label: "Project 1"}
  Configs1@{shape: processes, label: "Webpack+ESLint"}
  Project2@{shape: doc, label: "Project 2"}
  Configs2@{shape: processes, label: "TypeScript"}
  Project3@{shape: doc, label: "Project 3"}
  Configs3@{shape: processes, label: "Webpack+TypeScript+Babel+ESLint+Jest+Nightwatch"}
  Project4@{shape: doc, label: "Project 4"}
  Configs4@{shape: processes, label: "Webpack+Babel+ESLint"}
  Project5@{shape: doc, label: "Project 5"}
  Configs5@{shape: processes, label: "ESLint+Jest"}

  Configs1 --> Project1
  Configs2 --> Project2
  Configs3 --> Project3
  Configs4 --> Project4
  Configs5 --> Project5

My first attempt to address the issue was creating a shared package called webpack-digest. The goal of my shared package was to save time configuring and updating webpack and its ecosystem. In theory dozens of projects could remain updated with ease.

graph TD
  Project1@{shape: doc, label: "Project 1"}
  Project2@{shape: doc, label: "Project 2"}
  Project3@{shape: doc, label: "Project 3"}
  Project4@{shape: doc, label: "Project 4"}
  Project5@{shape: doc, label: "Project 5"}
  Configs[/webpack-digest/]

  Webpack
  Babel
  TypeScript
  ESLint
  Jest
  Nightwatch

  Webpack --> Configs
  Babel --> Configs
  TypeScript --> Configs
  ESLint --> Configs
  Jest --> Configs
  Nightwatch --> Configs

  Configs --> Project1
  Configs --> Project2
  Configs --> Project3
  Configs --> Project4
  Configs --> Project5

However, as the configuration grew, it began to include more features and dependencies. Projects using the shared configuration were pulling dependencies for features they were not using. Around that same time, create-react-app was released.

I investigated how create-react-app was managing webpack configuration among a myriad of other dependencies. It used a monorepo to share several aligned packages including webpack and babel configurations. While create-react-app was a great shared configuration, many projects still had to eject to extend that configuration or integrate missing features. Ejecting would abandon the shared config and commit a project to again maintaining complex configurations.

Drawing inspiration from create-react-app, I decided to similarly split the shared webpack-digest configurations into separate packages in a monorepo. My design goals for this shared configuration were:

  1. Downstream projects should pull only the dependencies they need
  2. Configuration with sensible defaults
  3. Auto-configuration based on installed dependencies
  4. Methods for full configuration customization
    • Shared configuration through a shared file
    • Priortity for local configuration files (webpack.config.js, eslint.config.js)
    • Supplementary configuration via environment variables (12 factor guidelines)

And so I created the @digest repository and npm scope.

Dependencies and Their Intermediaries

To begin, a separate package would be needed for each tool. For example, webpack would use @digest/webpack and babel would use @digest/babel. This pattern enables two things:

But what about optional dependencies? For example, webpack can use babel. However, some projects may use webpack without babel. If @digest/webpack depended on @digest/babel, some downstream projects would be pulling more dependencies than they needed (breaking goal #1). webpack also cannot use babel without babel-loader. But then which package is expected to include babel-loader: @digest/webpack or @digest/babel?

Instead, intermediary packages are used to share dependencies that neither package needs alone, such as babel-loader. For babel-loader, we use @digest/webpack-babel. The intermediary package @digest/webpack-babel also depends on both @digest/webpack and @digest/babel.

The pattern of intermediary packages really comes in handy. A few more examples just for webpack:

Intermediary packages can also simplify downstream dependency management. In the example above, a project using both webpack and babel could depend only on @digest/webpack-babel instead of including both @digest/webpack and @digest/babel.

graph TD
  digest-webpack@{shape: hex, label: "@digest/webpack"}
  digest-webpack-eslint@{shape: hex, label: "@digest/webpack-eslint"}
  digest-webpack-react@{shape: hex, label: "@digest/webpack-react"}
  digest-webpack-babel@{shape: hex, label: "@digest/webpack-babel"}
  digest-babel@{shape: hex, label: "@digest/babel"}
  digest-eslint@{shape: hex, label: "@digest/eslint"}
  digest-eslint-babel@{shape: hex, label: "@digest/eslint-babel"}
  Project1@{shape: doc, label: "Project 1"}

  digest-eslint --> digest-webpack-eslint
  digest-webpack --> digest-webpack-babel
  digest-webpack --> digest-webpack-eslint
  digest-babel --> digest-webpack-babel
  digest-babel --> digest-eslint-babel
  digest-eslint --> digest-eslint-babel
  digest-webpack-babel --> digest-webpack-react
  digest-webpack-react --> Project1
  digest-webpack-eslint --> Project1
  digest-eslint-babel --> Project1
  digest-webpack-eslint -. check -.-> digest-eslint-babel

Dependency Awareness and Automatic Configuration

Packages can bring in shared configurations and we want those shared configurations to be aware of any other installed @digest packages and automatically configure accordingly (Goal #3). Following the webpack and babel examples above, @digest/webpack would share webpack.config.js and @digest/babel would share babel.config.js. If @digest/babel is installed, we want the shared webpack.config.js to automatically configure itself to use babel.

To accomplish dependency awareness and automatic configuration, we need:

A shared package for configuration and package detection was needed and @digest/scripts was created for this role. Every @digest package depends on it. @digest/scripts checks whether every @digest package is installed. The package detection is simple and runs something like:

const webpackBabel = Boolean(require.resolve('@digest/webpack-babel'));

Now a shared configuration in another package can import @digest/scripts to know whether @digest/* packages are installed. @digest packages and shared configurations detect when a package is installed and enable related features by default. A project using @digest packages will have shared configurations that automatically configure themselves.

For example, @digest/webpack's shared webpack.config.js configuration can use @digest/scripts to check whether @digest/webpack-babel is installed. It can then automatically enable babel features and import the shared babel.config.js configuration.

sequenceDiagram
  participant Project
  participant WebpackConfig as @digest/webpack
  participant Scripts as @digest/scripts
  participant BabelConfig as babel.config.js

  Project->>WebpackConfig: Run webpack
  WebpackConfig->>Scripts: Is @digest/webpack-babel installed?
  Scripts-->>WebpackConfig: true
  WebpackConfig->>BabelConfig: Enable Babel and import either local or shared config
  WebpackConfig-->>Project: Return merged webpack config

Extending Configuration

Now that @digest packages' configuration files are aware of @digest dependencies, they can enable configuration features automatically. For example, after installing @digest/webpack-eslint, @digest/webpack will work with eslint. Installing @digest/webpack-typescript makes @digest/webpack work with Babel and TypeScript.

But remember, ESLint will still look for an eslint.config.js in a project root. It is still important that provided default configurations remain extensible (Goal #4). For example, the webpack command looks for the default webpack.config.js file. In most cases, just import and export the shared configuration:

// Project root / webpack.config.js

const webpackDigest = require('@digest/webpack');

module.exports = webpackDigest;

This is also how to extend the shared configurations when needed! It is always possible that a project using @digest packages will need to make changes, and it should be easy to extend the configuration.

But this raises another issue: how will @digest packages be aware of extended configuration files if they are only looking at packages for the shared configurations? If @digest/webpack is using @digest/babel, but babel.config.js is in the project root extending the shared configuration from @digest/babel, the @digest/webpack webpack.config.js will not find babel.config.js.

We need to track installed @digest packages and configuration paths.

Configuration Paths

To track configuration paths, @digest/scripts can be used again. Now it tracks installed @digest packages and configuration files. For example, it can automatically check a project's root folder for a babel.config.js and share that path. Now other @digest packages needing babel.config.js, such as @digest/webpack's webpack.config.js, can import it successfully.

Now consider whether configuration paths could also be customized. If @digest/scripts had its own configuration file, it could include settings for configuration paths. Those settings could override project-root and @digest package paths.

A central configuration could also override more than just configuration paths. There are many settings tools have in common. It would be helpful to have shared configurations load some default settings from a central configuration...

Central Configuration

The central configuration file would become digest.config.json. See settings in the wiki or the settings script for an exhaustive list. @digest/scripts would have its own default settings and import digest.config.js from a project's root for overrides.

For example:

// digest.config.json
{
  babel: ./node_modules/@digest/babel/dist/babel.config.js,
  cssModules: false,
  parallel: false,
  reactNative: false,
  reactNativeWeb: false,
  env: {
    test: 'test'
  }
}

Introducing another configuration layer when already orchestrating webpack, babel, typescript, eslint, and others adds further complexity. For that reason, configuration for @digest should stay limited in scope and be restricted to the following:

graph TD
  Digest[digest.config.js]
  Paths[Configuration paths]
  Packages[Package paths and feature toggles]
  Common[Common settings]
  PathValues[source, target, cache, reports]
  ToggleValues[babel: false or /path/to/config]
  CommonValues[minification, source maps, ports]

  Digest --> Paths
  Digest --> Packages
  Digest --> Common
  Paths --> PathValues
  Packages --> ToggleValues
  Common --> CommonValues

Caveats

While I do like the design of dependency-driven auto-configuration, it is not without its issues:

Conclusions

I think do think dependency-driven auto-configuration is a good approach to orchestrating disparate tools in a cohesive manner. Spring Boot Auto-Configuration also uses a dependency driven approach - I think the difference is being Spring Boot owns most of there stack. node shared configurations like create-react-app or @digest are coordinating tool configurations which they do not control.

One lesson I took from working on @digest is that while dependency-driven defaults are powerful, the override footprint must remain small and intuitive. The more tools a shared configuration tries to coordinate, the more important it becomes to keep the central configuration file small. Focus on paths, toggles, and a small number of common settings. Otherwise, allow respective configurations to be extended and contain their complexity.

The @digest project has evolved quite a bit over time. A few package trees have become less relevant. Nightwatch, for example, evolved quite a bit, and a shared configuration is no longer feasible; it seems better to integrate it in a separate project under a monorepo. I am also debating whether to remove the @digest/eslint* tree of packages since eslint-config-canonical is already quite extensible. Jest has also been a challenge to maintain, and Vitest may be a simpler configuration alternative.

Finally, @digest was born during a period of complex configurations. It saved time sharing a webpack config across many projects. But are shared configurations as necessary today? The churn from that era (ES6, Babel, TypeScript) has slowed. Tools that are easier to set up and run, like Vite or One, likely meet most needs. For webpack, create-react-app can still serve as a focused configuration. And for complex projects with many toosl, perhaps LLMs can also help make quick adjustments to common configurations.

@digest has been a fun project to update and maintain. But I'm always looking to reduce the burden of configuration complexity in my projects. While it is sad to think of moving on from a project that I poured so much thought into, perhaps it means projects no longer need the machinery of @digest to manage that complexity.