Sovereign Tech Fund invests in Cabal as critical Haskell infrastructure

We are delighted to announce that the Sovereign Tech Fund is investing in the development and maintenance of Cabal, the Haskell build system, following a proposal Well-Typed submitted to the “Improve FOSS Developer Tooling” challenge.

The Sovereign Tech Fund is funded by the German government and tasked with “strengthening digital infrastructure”. There are a variety of investments which they support. Firstly, a general maintenance investment for under-maintained but critcial technologies, but also challenge funding for specific investment into particular challenges which have been identified as of critical importance for the future of open-source software projects.

Cabal is a central and critically important technology for all Haskell projects, but it relies heavily on volunteer efforts and does not have significant commercial sponsorship. This makes it challenging to find the resources to make Cabal more maintainable and deal with legacy aspects of its architecture that threaten its long-term sustainability.

The project we proposed includes two key aspects:

• Investigate alternatives to the Custom build-type for Cabal packages. This work is aimed at improving the overall resilience and maintainability of the Haskell ecosystem by providing a more future-proof mechanism for augmenting the building of Haskell packages.

• Maintainer time for tasks such as making releases, keeping CI up and running, reviewing and landing volunteer contributions, and assisting users of Cabal and cabal-install. This is especially welcome as Cabal has been lacking maintainer funding for some time.

In the rest of this post we will explore the first part in more detail.

Motivation

When the original Cabal specification was created, there were only a handful of Haskell packages, and at the time each of them had its own build system. Thus the design fundamentally assumed that each package would have its own build system. In practice, however, this fundamental assumption turned out to be wrong: almost all Haskell packages use the build system provided by the Cabal library, perhaps with minor customisations.

Today, each package can provide a Setup.hs script defining how it should be built. The Cabal specification dictates that a Setup.hs script must obey a specific interface. In principle, one can build an individual package by first compiling Setup.hs and then invoking:

./Setup configure
./Setup build

However, this design makes it difficult to robustly implement cross-cutting build system features, such as:

• building individual components independently,

• loading multiple packages into a single GHCi or HLS session, or

• speeding up builds through more effective use of parallelism.

While these features work well in many common cases, they cannot handle some more complex situations. Ideally, rather than each and every package having its own build system, the build tool (e.g. cabal-install or HLS) should define a build system for all the packages at once.

Existing build types

Packages indicate how much customisation of the build process they require by declaring which build-type they use. Currently there are four options:

Type	Description
Simple	Use the standard Cabal build system. Most packages use this option.
Configure	Run a ./configure script to discover system configuration, then build using the standard build system.
Make	Invoke make to build the package using its own build system (obsolete).
Custom	Compile and run a custom Setup.hs script defining the package’s build system.

The most flexible option used in practice is build-type: Custom. In this case, the package can define an arbitrary Setup.hs script which implements the command-line interface defined by the Cabal specification. That is, the program must support being executed as ./Setup configure, ./Setup build, and so on, but it can implement whatever logic it wants.

In practice, almost all custom Setup.hs scripts depend on the Cabal library, which allows for customisation of its build system by providing a value of the UserHooks datatype. However, this type is itself not ideal, and the Cabal project has been seeking to move away from the Custom build-type for a long time, as its documentation makes clear:

This hooks type is widely agreed to not be the right solution. … At some point it will have to be replaced.

Why do packages use the Custom build-type?

There are various reasons why packages may use the Custom build-type, such as:

• detecting system configuration;

• generating source code for modules during a build (e.g. to make information about the build environment available to the final executable);

• adding build system features which are not natively supported by Cabal, such as doctests;

• working around bugs in build tools; or

• executing additional steps when a package is installed (e.g. the Agda compiler is a normal Haskell package that needs to compile some Agda libraries when it is installed).

Over time, Cabal has gradually incorporated more features to allow some of these use cases to be subsumed, typically by adding more declarative information to the .cabal file format. For example, pkgconfig-depends reduces the need for packages to have custom configuration logic. Similarly, build-type: Configure can be used to implement configuration logic in a more targeted way than build-type: Custom.

What is wrong with Custom, and what can we do instead?

As discussed earlier, the primary issue with the Custom build-type is that the current Cabal specification allows Setup.hs to completely replace whole phases of the build process. This is too flexible, as when a package uses Custom then any tool has to behave very pessimistically under the assumption that it might do anything.

This means that where a package in the build plan uses Custom, build tools such as cabal-install must fall back on legacy code paths to compile it. This causes various problems and requires hacky workarounds:

• The Setup.hs interface works only with whole packages, and cannot easily be changed. Modern cabal-install versions support building individual components independently (e.g. compiling a library and one selected test-suite without compiling other test-suites in the same package).

• Features such as multi-repl need to work across multiple packages simultaneously, and this may or may not make sense depending on what Setup.hs does.

• We want to compile multiple packages in parallel using -jsem, but with a Custom build-type the -jsem flag could in principle be completely ignored. We would much rather make those decisions at the cabal-install level than delegate responsibilty for these choices to each package.

• There are challenges supporting Setup.hs files in HLS.

We want to tackle this problem from two angles:

• As much as possible, we should encourage package authors to replace uses of the Custom build-type with declarative features.
• However, sometimes it is not feasible to remove the need for custom build logic entirely. Thus we plan to introduce a successor to the Custom build-type, which will allow a package author to augment (but not replace) build phases.

Vision

In the first phase of the project, we have been surveying existing libraries which use the Custom build type, ascertaining the most important uses of the current mechanism. We can then focus our design on supporting those cases, and identify opportunities to use declarative features instead.

The second phase is to design and implement a new build type: Hooks in the Cabal library as a replacement for Custom. The goal is for Hooks to be expressive enough to perform all the tasks implemented by the Custom scripts we have analysed (and that cannot reasonably use Simple), while being limited enough to resolve the main problems with Custom and the UserHooks type. As part of this process, we are writing a detailed design document which explains the motivations and considerations in more detail, and asking the community for feedback.

The third phase is to begin to migrate packages away from build-type: Custom to Simple or Hooks, thereby establishing they are a sufficient replacement. Of course, Cabal will need to retain support for build-type: Custom for some time to allow gradual migration, but eventually we hope it will be feasible to deprecate it and eventually remove support entirely.

A crucial goal of this work is making Cabal easier to maintain and develop over the long term. However, in the short term it is going to require additional effort from Cabal maintainers to review the design and implementation. To compensate for this and and make the project feasible we have allocated part of the budget for general Cabal maintenance work, which is greatly welcome due to a recent lack in funding in this area.

Future work

The Sovereign Tech Fund has so far committed to fund four months of work, which we anticipate will allow us to carry out the plan above, focusing on the Cabal library. After this initial phase there will be the opportunity to apply for a second round of funding.

If we are able to raise further investment, we would like to modify cabal-install to take advantage of the new opportunities offered by the Hooks build type. In particular, it should be possible to construct a more fine-grained build graph which is controlled by cabal-install rather than GHC. If we can express a build-graph on a per-module level then this can greatly increase the amount of parallelism available. A similiar process in Hadrian, GHC’s build system, led to a 25% reduction in wall-clock build time. This would also lead to much more accurate progress and error reporting as cabal-install would gain much more responsibility for these aspects of a build.

Conclusion

We have started drafting a design document and welcome feedback on Cabal issue #9292. We intend to open a Haskell Foundation Tech Proposal in due course to seek feedback from a wide range of stakeholders, including Cabal developers, authors of other Haskell developer tools, and users of build-type: Custom.

The investment provided by the Sovereign Tech Fund is invaluable in being able to tackle these fundamental issues which are critically important to the Haskell software ecosystem but also difficult for an open-source project to tackle without funding.

There is still a need for more resources to be invested in GHC, Cabal, HLS and other core Haskell infrastructure, and plenty of opportunities for improvements that will make a real difference to the Haskell developer experience. Well-Typed actively works on these issues thanks to funding from various sponsors. If your company might be able to contribute to this work, sponsor maintenance efforts, or fund the implementation of other features, please read about how you can help or get in touch.