New Cabal and cabal-install releases

Tuesday, 17 June 2008, by Duncan Coutts.
Filed under coding.

Cabal-1.4 and cabal-install-0.5 are now released!

If you were already using a pre-release of cabal-install then you can just:

$ cabal update
$ cabal install cabal-install

The Cabal library

There are no huge new features in the Cabal library itself. The main new feature is that is supports a significantly improved version of cabal-install. Cabal-1.4 does have a number of incremental improvements and lots of bug fixes. You can see the full list in the changelog.

So the most exciting thing is the release of cabal-install...

New features in cabal-install

Command line improvements

The most immediately noticeable thing is that the command line interface now has all the commands that runhaskell Setup.hs has. Of course it still has the features to download and install packages from hackage. It also gained an upload command. So it now provides a command line interface to the whole Cabal/Hackage system. There's no need to use runhaskell Setup.hs ever again. There is also bash command line completion support included which I find is a great time saver.

Installing and upgrading

The next big thing is that it includes a new package dependency resolution system that finds correct and sensible solutions more of the time and has better default behaviour. The new behaviour should be similar to other package managers that people are used to.

For example, suppose you've got xmonad-0.5 installed and version 0.7 is the latest on hackage, then

$ cabal install xmonad

will upgrade to xmonad-0.7.

The behaviour of install is to upgrade as little as possible to satisfy your request, but sometimes you want to upgrade all the dependencies too. Supposing now that we have xmonad-0.7 installed, but we're still using X11-1.4.1 and the latest version on hackage is X11-1.4.2, then

$ cabal upgrade xmonad

will install X11-1.4.2 and re-install xmonad-0.7, this time built against the newer X11-1.4.2.

So in general, the install command will install the latest version of things but will try and use any existing installed versions of dependencies while the upgrade command will also try to use the latest versions of all dependencies. As a special case, cabal upgrade on its own will try to upgrade all the packages that you have installed.

For both command there is a --dry-run flag so you can see what would be installed without actually doing it.

Hugs

Yes, it even works with hugs. That is, cabal-install built by ghc can manage the installation of packages for hugs. In principle cabal-install should be able to be run by hugs but currently the zlib binding is using a function that hugs does not support.

Note that for hugs, Cabal does not know what packages are already installed because there is no equivalent of the package database that ghc has. So that means cabal-install cannot do very sensible installation planning. It should work ok so long as all the dependencies are already installed.

Windows

Yes, it even works on Windows. The one caveat is that cabal-install cannot currently upgrade itself because Windows makes it hard for a process to overwrite its own executable file. It needs more complex trickery with the Win32 API. In the meantime the workaround is to rename the cabal.exe file first eg to cabal-foo.exe, then say cabal-foo install cabal-install.

Build reporting

One feature that made it into this release is build reporting. cabal-install keeps logs of all packages that you install (at least packages from hackage, not local ones). It records a bit of information about each package, in particular whether the outcome was successful or not. You can see these build reports in ~/.cabal/packages/$server/build-reports.log.

The plan in the longer term is to let people upload these build reports to hackage so we can get a wider range of testing data about the packages on hackage.


Information plumbing

Thursday, 08 May 2008, by Duncan Coutts.
Filed under coding.

If you look at real programs you'll find there is often a lot of code spent just on information plumbing. Plumbing is code that doesn't do any "real work" but is just concerned with getting the right information from one bit of a program to another, often with some rearranging or impedance matching. It's not as cool as code with real algorithmic content. So how can we stop plumbing code taking over and obscuring the code that is doing real work?

A nice example is some code in cabal-install for installing a package. Well, I say installing a package but really the actual installing is done by the Cabal library. This code just gathers information from various sources and uses that to decide how to configure a package. There is a little bit of real work mixed in, like unpacking tarballs, but it is mostly plumbing. For example there is the global configure options that apply to each package, per-package flag assignments and dependencies. Then there are various options for controlling how we compile and run the Setup.hs scripts for each package, including what compiler to use and what version of the Cabal library to run them with. There's also an option to use some kind of sudo style wrapper for the install phase.

The code followed a common pattern of a series of layers, each one calling the layer below. In the cabal install example we have executeInstallPlan that calls installConfiguredPackage, which calls installAvailablePackage which calls installUnpackedPackage which calls setupWrapper (which as the name suggests is a layer on top of something else). It's a lot of layers. Lets pick out just two examples:

installConfiguredPackage verbosity
    scriptOptions miscOptions configFlags
    (ConfiguredPackage pkg flags deps)
  = installAvailablePackage verbosity
    scriptOptions miscOptions configFlags' pkg
  where
    configFlags' = configFlags {
      configConfigurationsFlags = flags,
      configConstraints = deps
    }

installAvailablePackage verbosity
    scriptOptions miscOptions configFlags
    (AvailablePackage _ pkg LocalPackage)
  = installUnpackedPackage verbosity
      scriptOptions miscOptions configFlags
      pkg Nothing

As you can see, the problem with this style is that each layer gets cluttered with passing all the information that the lower layers need. Bundling related bits of information into tuples or records helps to some extent. In this example you can see it's been done already; scriptOptions, miscOptions and configFlags are bundles of other parameters. (The name miscOptions is a sure sign that we're bundling arbitrary things together just to try and reduce the number of parameters we're passing.)

What is worst though is that we end up with a tightly coupled system. To add a new feature at the bottom layer requires changing all the layers to pipe that information all the way down.

So one nice trick is instead of having each layer directly call the next one down, we parametrise each layer by all the layers below it. For example:

installConfiguredPackage configFlags
    (ConfiguredPackage pkg flags deps)
    installPkg
  = installPkg configFlags' pkg
  where
    configFlags' = configFlags {
      configConfigurationsFlags = flags,
      configConstraints = deps
    }

installAvailablePackage
    (AvailablePackage _ pkg LocalPackage)
    installPkg
  = installPkg pkg Nothing

So we take the next layer as an installPkg parameter, do whatever transform the layer was supposed to be do and call installPkg as the next layer. So all the extra parameters that this layer did not care about are gone. They can be passed directly to installPkg by the caller.

So in the end we stick all the layers together:

executeInstallPlan installPlan $ \cpkg ->
  installConfiguredPackage configFlags cpkg $
  \configFlags' apkg ->
    installAvailablePackage verbosity apkg $
      installUnpackedPackage verbosity
          scriptOptions miscOptions configFlags'

Unlike before, we can now see all the layers at once rather than just the top layer. So we can pass those extra parameters directly down to the bottom layer. As you can see from the installConfiguredPackage layer, we can modify the values on the way down but we only have to pay in plumbing for the ones we're using at each layer rather than in every layer.

After the refactoring the code can become more general which has advantages for testing and reuse. For example installConfiguredPackage turns out to be pure and executeInstallPlan can work in any monad where as previously it was tied down to being in the IO monad because the bottom layer was in the IO monad. Not being in IO makes it much easier to use a testing system like QuickCheck and being able to pass a dummy makes it possible to test each layer in isolation.

So obviously the message must be that higher order plumbing is cool!

Well perhaps not, but it can make it a little less interconnected and easier to manage.


The dreaded diamond dependency problem

Thursday, 24 April 2008, by Duncan Coutts.
Filed under coding, cabal.

One issue that came up several times at the hackathon-especially when talking to the Yi hackers-is the dreaded diamond dependency problem. This happens when you have 4 packages in a diamond shaped dependency graph:
Diamond dependency graph
The problem arises when we have package B and C already installed but built against different versions of D and then we try to use both packages B and C together in package A:
Diamond dependency problem
This can work ok but only if packages B and C do not expose types defined in D in their interfaces. If they do then package A will not be able to use functions from B and C together because they will not be working with the same type. That is you'll get a type error.

To pick a concrete example, suppose package D is bytestring and we have both bytestring-0.9.0.1 and 0.9.0.4 installed. Lets say B is utf8-string and C is regex-base. Lets say that package A is the Yi editor program. So the point is, at some place in the code in Yi we want to pass a bytestring produced as the result of UTF-8 decoding as input to one of the regex functions. But this does not work because the functions in the utf8-string package are using the ByteString type from bytestring-0.9.0.1 while the regex functions in the regex package are using the ByteString type from bytestring-0.9.0.4. So we get a type error when we try to compile Yi:

Couldn't match expected type `bytestring-0.9.0.4:Data.ByteString.ByteString'
against inferred type `bytestring-0.9.0.1:Data.ByteString.ByteString'

As far as GHC knows, these two types are totally unrelated!

This is obviously extremely annoying. There is also no easy solution. In this example we're assuming that packages B and C have already been built, so there is actually no way to sensibly use the two packages together without rebuilding on of them against a different version of package D. In this case the obvious solution is to rebuild B to use the D-1.1 rather than D-1.0. The problem with rebuilding a package of course is it breaks all other packages that have already been built against it. It isn't clear that you want a package manager to go automatically rebuilding lots of apparently unrelated packages.

In the longer term the best solution would seem to be to do what Nix does. In the above example, instead of replacing package B built against D-1.0 with B built against D-1.1, Nix would add another instance of B built against D-1.1. So the original instance of B would remain unchanged and nothing would break. It's the functional approach: we never mutate values (installed packages) we just create new ones and garbage collect old one when they are no longer needed.

In practice it means we have to identify installed packages using some hash of the package and the hashes of all dependent packages. jhc already does this and there are moves afoot to do something similar for GHC, though aimed more at tracking API/ABI changes. For sane source based package management, I think it's the right direction to take.

I should note that this is not a new problem. You've been able to construct this problem ever since ghc started to allow multiple versions of the same package to be installed at once. We are just noticing it a lot more frequently now because we split up the base package and allow those split-out packages to be upgraded.

The current state of play is that Cabal warns of this problem but doesn't really help you solve it. For the above example we'd get:

$ cabal configure
Configuring A-1.0...
Warning: This package indirectly depends on multiple versions of
the same package. This is highly likely to cause a compile failure.
package B-1.0 requires D-1.0
package C-1.0 requires D-1.1

GSoC project for Cabal make-like dependency framework

Monday, 21 April 2008, by Duncan Coutts.
Filed under community.

I am very pleased to be mentoring Andrea Vezzosi (aka Saizan) on a Google Summer of Code project to develop a make-like dependency framework for use in Cabal. I'm really looking forward to working with Saizan on this. It should be a fun and useful project.

Congratulations to Saizan and good luck with the project!

The project itself is to start prototyping one of the big features we need for Cabal to become a really excellent build system. At the moment Cabal calls out to ghc --make to do the hard work of figuring out what needs to be rebuilt. This only works for .hs files of course while we need it to work for all preprocessors. More generally there are a lot of things that Cabal does that would benefit from being done in a dependency style - not just limited to running external programs that update files. In addition everyone has multi-core CPUs now and we'd like to do parallel builds.


Real World Haskell book available for pre-order

Sunday, 20 April 2008, by Duncan Coutts.
Filed under community.

The Real World Haskell book is now available for pre-order!

I've just ordered my copy from Amazon UK for £30.99 and it's available from Amazon in the US for $49.99.

I'd like to be able to recommend a canonical "learning Haskell" bookshelf. The slot for an introductory book has been filled by the excellent Programming in Haskell (you can read my review). Judging by the material the Real World Haskell book aims to cover, it looks like it should be a pretty good complement, mostly picking up where Programming in Haskell leaves off and covering the language techniques and tools you need to get started writing serious programs.


Cabal and Hackage at the hackathon

Friday, 18 April 2008, by Duncan Coutts.
Filed under community.

We ended up with quite a big group of people interested in various aspects of Cabal, cabal-install and Hackage. So I ended up spending much more time talking than hacking. I'm sure that's a good thing though. Hackathons should be about sharing ideas, brainstorming and experimenting.

I arrived at the hackathon with ideas about three main areas I wanted to get people to work on:

In fact these are more or less exactly the topics for proposed Google Summer of Code projects, but there's no harm in starting early and exploring possible solutions.

I expounded a bit on the dependency framework issue though we didn't end up working on that in the end. It's actually an interesting research problem. I'll write about it another time.

Josef Svenningsson got interested in the cabal-install dependency resolver issue. He spent much of the hackathon prototyping a BDD-style approach. Hopefully Josef will post his current ideas to the cabal-devel mailing list in the next week or so.

Thomas Schilling (aka nominolo) worked on the last remaining feature we wanted to get into Cabal-1.4. When configuring a package want to be able to specify additional constraints on the versions of dependent packages. As a developer we might want this just to be able to test building a package against an old rather than the latest version of a dependency. The reason we really want it for Cabal-1.4 though is so that we can use it in cabal-install. In cabal-install we want to work out up front exactly how we will configure a whole set of packages. We need a way to get Cabal to configure a package exactly how we wanted it in our global plan, rather than making some local decision. Thomas got this finished and I'll be integrating the patches some time soon.

Lennart Kolmodin (with some help from David Waern and myself) spent quite a while hacking on build reporting in cabal-install. The idea here is that we want to get cabal-install to report build success and failure back to the hackage server so we can discover what packages build in what environments. On the client side most of the work involves information plumbing - gathering information from various places within Cabal to include into a build report. We realised that most of the information for a build report is discovered once we've decided how we're going to configure the package. So we spent most time hacking on some code I'd written previously for representing installation plans and extending it to collect information on the outcome of installation. We managed to get it integrated though it still needs more work because our new InstallPlan type has a much more demanding notion of validity than the abstraction it replaces, so the current dependency resolver needs to be updated to produce valid InstallPlans.


Hac4: Thank you!

Wednesday, 16 April 2008, by Björn Bringert.
Filed under community.

Hac4 is now over, everyone has arrived home safely and all the equipment has been returned. 27 people attended, though not everyone could come every day. 10 attendees travelled from several European countries and from the US. People hacked on lots of projects, including Yi, Cabal, ByteString, a new web application interface, QuickCheck, CGI, first-order logic reasoning, Allegro bindings, Haddock, the community server, HAppS-HSP, replacing libgmp in GHC, Squiggle, restricted monads, and more.

We also decided to set up a Haskell Hackathon Steering Committee which will help find hosts for future hackathons and aid the local organizers.

Thanks to sponsorship from Credit Suisse and Galois, we could provide food for everyone during the whole hackathon. This made it possible to keep the group together and keep hacking (some stayed well into the night). The Department of Computer Science and Engineering at Chalmers University of Technology and University of Gothenburg provided space and networking.

A big thank you to everybody who made this happen by sponsoring, attending and helping out with the organization!


Hac4: Last day

Sunday, 13 April 2008, by Björn Bringert.
Filed under community.

Pictures from the third and last day of Hac4:

Top: leather, mahogny, Lemmih, krasimir, kosmikus, kolmodin, tibbe, jyp, dons, dcoutts, bringert, Igloo. Bottom: thorkilnaur, ralejs, nominolo, Heffalump
Top: leather, mahogny, Lemmih, krasimir, kosmikus, kolmodin, tibbe, jyp, dons, dcoutts, bringert, Igloo. Bottom: thorkilnaur, ralejs, nominolo, Heffalump


Hac4: Second day

Saturday, 12 April 2008, by Björn Bringert.
Filed under community.

Some pictures from the second day of Hac4:

ralejs and koen
ralejs and koen
thorkilnaur, dcoutts and kolmodin
thorkilnaur, dcoutts and kolmodin

dcoutts, augustss, dons, kosmikus, Igloo, tibbe and Heffalump
dcoutts, augustss, dons, kosmikus, Igloo, tibbe and Heffalump
kosmikus and Heffalump
kosmikus and Heffalump


Hac4: Haskell Hackathon in Gothenburg

Friday, 11 April 2008, by Björn Bringert.
Filed under community.

The 4th Haskell Hackathon has started, and all of Well-Typed is here.

Some pictures from the first day:

leather, Heffalump and Igloo
leather, Heffalump and Igloo
kosmikus and Lemmih
kosmikus and Lemmih

jyp and krasimir
jyp and krasimir
mahogny and Lillann
mahogny and Lillann

tibbe and dons
tibbe and dons
kolmodin
kolmodin


Previous entries

Next entries