We are delighted to announce that the Sovereign Tech Fund is investing in the development and maintenance of Cabal, the Haskell build system, following a proposal Well-Typed submitted to the “Improve FOSS Developer Tooling” challenge.
The Sovereign Tech Fund is funded by the German government and tasked with “strengthening digital infrastructure.” There are a variety of investments which they support. Firstly, a general maintenance investment for under-maintained but critcial technologies, but also challenge funding for specific investment into particular challenges which have been identified as of critical importance for the future of open-source software projects.
Cabal is a central and critically important technology for all Haskell projects, but it relies heavily on volunteer efforts and does not have significant commercial sponsorship. This makes it challenging to find the resources to make Cabal more maintainable and deal with legacy aspects of its architecture that threaten its long-term sustainability.
The project we proposed includes two key aspects:
Investigate alternatives to the
Custombuild-type for Cabal packages. This work is aimed at improving the overall resilience and maintainability of the Haskell ecosystem by providing a more future-proof mechanism for augmenting the building of Haskell packages.
Maintainer time for tasks such as making releases, keeping CI up and running, reviewing and landing volunteer contributions, and assisting users of Cabal and
cabal-install. This is especially welcome as Cabal has been lacking maintainer funding for some time.
In the rest of this post we will explore the first part in more detail.
When the original Cabal specification was created, there were only a handful of Haskell packages, and at the time each of them had its own build system. Thus the design fundamentally assumed that each package would have its own build system. In practice, however, this fundamental assumption turned out to be wrong: almost all Haskell packages use the build system provided by the
Cabal library, perhaps with minor customisations.
Today, each package can provide a
Setup.hs script defining how it should be built. The Cabal specification dictates that a
Setup.hs script must obey a specific interface. In principle, one can build an individual package by first compiling
Setup.hs and then invoking:
However, this design makes it difficult to robustly implement cross-cutting build system features, such as:
building individual components independently,
While these features work well in many common cases, they cannot handle some more complex situations. Ideally, rather than each and every package having its own build system, the build tool (e.g.
cabal-install or HLS) should define a build system for all the packages at once.
Existing build types
Packages indicate how much customisation of the build process they require by declaring which
build-type they use. Currently there are four options:
|Use the standard Cabal build system. Most packages use this option.
./configure script to discover system configuration, then build using the standard build system.
make to build the package using its own build system (obsolete).
|Compile and run a custom
Setup.hs script defining the package’s build system.
The most flexible option used in practice is
build-type: Custom. In this case, the package can define an arbitrary
Setup.hs script which implements the command-line interface defined by the Cabal specification. That is, the program must support being executed as
./Setup build, and so on, but it can implement whatever logic it wants.
In practice, almost all custom
Setup.hs scripts depend on the
Cabal library, which allows for customisation of its build system by providing a value of the
UserHooks datatype. However, this type is itself not ideal, and the Cabal project has been seeking to move away from the
Custom build-type for a long time, as its documentation makes clear:
This hooks type is widely agreed to not be the right solution. … At some point it will have to be replaced.
Why do packages use the
There are various reasons why packages may use the
Custom build-type, such as:
detecting system configuration;
generating source code for modules during a build (e.g. to make information about the build environment available to the final executable);
adding build system features which are not natively supported by Cabal, such as doctests;
working around bugs in build tools; or
executing additional steps when a package is installed (e.g. the Agda compiler is a normal Haskell package that needs to compile some Agda libraries when it is installed).
Over time, Cabal has gradually incorporated more features to allow some of these use cases to be subsumed, typically by adding more declarative information to the
.cabal file format. For example,
pkgconfig-depends reduces the need for packages to have custom configuration logic. Similarly,
build-type: Configure can be used to implement configuration logic in a more targeted way than
What is wrong with
Custom, and what can we do instead?
As discussed earlier, the primary issue with the
Custom build-type is that the current Cabal specification allows
Setup.hs to completely replace whole phases of the build process. This is too flexible, as when a package uses
Custom then any tool has to behave very pessimistically under the assumption that it might do anything.
This means that where a package in the build plan uses
Custom, build tools such as
cabal-install must fall back on legacy code paths to compile it. This causes various problems and requires hacky workarounds:
Setup.hsinterface works only with whole packages, and cannot easily be changed. Modern
cabal-installversions support building individual components independently (e.g. compiling a library and one selected test-suite without compiling other test-suites in the same package).
Features such as multi-repl need to work across multiple packages simultaneously, and this may or may not make sense depending on what
We want to compile multiple packages in parallel using
-jsem, but with a
-jsemflag could in principle be completely ignored. We would much rather make those decisions at the
cabal-installlevel than delegate responsibilty for these choices to each package.
There are challenges supporting
Setup.hsfiles in HLS.
We want to tackle this problem from two angles:
- As much as possible, we should encourage package authors to replace uses of the
Custombuild-type with declarative features.
- However, sometimes it is not feasible to remove the need for custom build logic entirely. Thus we plan to introduce a successor to the
Custombuild-type, which will allow a package author to augment (but not replace) build phases.
In the first phase of the project, we have been surveying existing libraries which use the
Custom build type, ascertaining the most important uses of the current mechanism. We can then focus our design on supporting those cases, and identify opportunities to use declarative features instead.
The second phase is to design and implement a new
build type: Hooks in the
Cabal library as a replacement for
Custom. The goal is for
Hooks to be expressive enough to perform all the tasks implemented by the
Custom scripts we have analysed (and that cannot reasonably use
Simple), while being limited enough to resolve the main problems with
Custom and the
UserHooks type. As part of this process, we are writing a detailed design document which explains the motivations and considerations in more detail, and asking the community for feedback.
The third phase is to begin to migrate packages away from
build-type: Custom to
Hooks, thereby establishing they are a sufficient replacement. Of course, Cabal will need to retain support for
build-type: Custom for some time to allow gradual migration, but eventually we hope it will be feasible to deprecate it and eventually remove support entirely.
A crucial goal of this work is making Cabal easier to maintain and develop over the long term. However, in the short term it is going to require additional effort from Cabal maintainers to review the design and implementation. To compensate for this and and make the project feasible we have allocated part of the budget for general Cabal maintenance work, which is greatly welcome due to a recent lack in funding in this area.
The Sovereign Tech Fund has so far committed to fund four months of work, which we anticipate will allow us to carry out the plan above, focusing on the
Cabal library. After this initial phase there will be the opportunity to apply for a second round of funding.
If we are able to raise further investment, we would like to modify
cabal-install to take advantage of the new opportunities offered by the
Hooks build type. In particular, it should be possible to construct a more fine-grained build graph which is controlled by
cabal-install rather than GHC. If we can express a build-graph on a per-module level then this can greatly increase the amount of parallelism available. A similiar process in Hadrian, GHC’s build system, led to a 25% reduction in wall-clock build time. This would also lead to much more accurate progress and error reporting as
cabal-install would gain much more responsibility for these aspects of a build.
We have started drafting a design document and welcome feedback on Cabal issue #9292. We intend to open a Haskell Foundation Tech Proposal in due course to seek feedback from a wide range of stakeholders, including Cabal developers, authors of other Haskell developer tools, and users of
The investment provided by the Sovereign Tech Fund is invaluable in being able to tackle these fundamental issues which are critically important to the Haskell software ecosystem but also difficult for an open-source project to tackle without funding.
There is still a need for more resources to be invested in GHC, Cabal, HLS and other core Haskell infrastructure, and plenty of opportunities for improvements that will make a real difference to the Haskell developer experience. Well-Typed actively works on these issues thanks to funding from various sponsors. If your company might be able to contribute to this work, sponsor maintenance efforts, or fund the implementation of other features, please read about how you can help or get in touch.