#80 introduced some interop with Java rules: if a Java provider is present, we add its jar to CLASSPATH during compilation. The right thing to do there however is to add transitive jars. This is blocked on an upstream bazel ticket (bazelbuild/bazel#4384).

Until then, the user has to manually insert all the jars they require even if they are already present somewhere down the Java compilation chain.

Bazel has a notion of "toolchain", which are typically specified by giving a URL via one of new_http_archive, new_git_repository etc. It would be great if we could furthermore specify a toolchain as an attribute in Nixpkgs. Reasons:

• It is wasteful to download a new toolchain, when this toolchain is likely already in the Nix store and can be shared across many Bazel repos.
• There does not exist GHC bindists that, when unpacked, work on NixOS, which has ELF loaders and system libraries in non-standard locations. On NixOS, only the GHC provided by Nixpkgs works.
• Nixpkgs can provide the strong reproducibility guarantees that Bazel requires. Given a Nixpkgs Git commit identifier, and an attribute name, the toolchain will always be byte-for-byte identical if it is downloaded now or in the future.

The problem is, I don't see how to achieve this with Bazel's current rule set. AFAIU, there is only a fixed set of primitive rules that are legal in a WORKSPACE file. What we'd want to define is a macro

new_nixpkgs_package(
  name = "somepkg",
  sha256 = "...",
  build_file = "BUILD.somepkg",
  nixpkgs_version = "...", # optional
)

and have that expand to some invocation of new_local_repository(path = somepkg_path, ...). But we can't hardcode such a path, which we don't know a priori. We can derive it after invoking some Nix CLI tools, however.

So what we'd need from Bazel is the ability to run arbitrary shell commands in a WORKSPACE file and allow fields of say new_local_repository to depend on the output of these commands. @jin any advice as to whether this is possible, or would make sense to add to the Bazel roadmap?

We should have support for .hs-boot files (https://wiki.haskell.org/Mutually_recursive_modules). In current state this likely involves no real new work from us: just allowing these files in srcs should be enough. This ticket should only be closed after adding tests with .hs-boot files such that any future changes that are likely to break them (such as incremental builds) are caught and we don't lose the ability to build with these.

Ideally rules_haskell should produce binaries that are relocateable. This is a discussion ticket. Most basic things to consider:

Relocateable where:

• elsewhere on build host
• onto a different host

There are multiple things to solve

• non-absolute dynamic libraries: this is all .so objects we depend on that come from bazel itself for example.
• absolute dynamic libraries: if we use something like rules_nixpkgs toolchain for system libs, we can only move the binary onto hosts with those libraries present
• runtime files: bazel has some support for files needed at binary runtime: think of something like Cabal's getDataDir. These have to be moved with the binary.

Possibly we can use packaging rules to help somehow. I'd be interested to see what pkg_tar results in: perhaps it will dutifully copy both dependencies and runtime files with the binary out. I don't think we can do much about absolute dynamic binaries so if we want to cross host boundaries we'll have to do something magic. Ideally we probably don't want to rely on absolute paths at all. May require GHC work(?).

I think non-targets of this ticket are movable haskell_library and haskell_haddock artifacts: both may be desired though: haskell_library for general use and haskell_haddock if we want to host the docs somewhere for example.

Create a haskell_haddock test like

haskell_haddock(
  deps = [":haskell-dep", ":non-haskell-dep"],
)

that verifies docs are still generated correctly. It should be Trivial™ as long as non-haskell-dep is marked as test-only.

Created from #47.

What

A command-line tool to generate BUILD files from Cabal package descriptions.

Would behave similarly to the gazelle tool provided by https://github.com/bazelbuild/rules_go

Why

Supporting dependencies

• the vast majority of existing Haskell code uses Cabal
• to use these projects as dependencies of a Bazel project requires they each have BUILD files
• writing those files by hand is tedious & error-prone

Migrating to Bazel

• People who want to start using Bazel will have a much easier time of it if there is a tool to help them migrate

Maintaining Haskell Bazel projects

• Bazel works best with fine-grained dependencies—more opportunity for parallelism, less to rebuild when things change
• It's very common to change the dependencies of a module when writing code
• It would be great if you could just run a tool to update the BUILD file when this happens

Complicated dependencies

• A lot of Haskell code depends on C libraries, which could theoretically be built in Bazel
• Rather than many human beings re-learning how to do this, how about we encode the knowledge in a computer program?

The exact details & design of this tool need to be fleshed out. I would suggest closing this ticket once there's a minimal tool that handles the common 80% use case and filing follow-ups for the rest.

A very useful feature would be cross-compilation support. Bazel already has the ability to specify if a target should be built on host or target. Cross-compiling Haskell is currently difficult/black magic and this feature would be a big selling point.

Related to #29. https://github.com/bazelbuild/bazel/wiki/Building-with-a-custom-toolchain might be helpful.

Currently haskell_haddock outputs all Haddock outputs that are in its direct deps results. This means that if we have

haskell_library(X)
haskell_library(Y, deps = X)
haskell_haddock(deps = Y)

outputs shown to the user are only those files from Y. Possibly we might need files from X too: if they aren't included then cross-package links might not work for example. Check if this is the case. This might be the same as #50.

Created from #47.

This isn’t really an issue, just a way to kick off communications. (Do you have a mailing list I should be using instead?)

Very excited to discover this repo. I spent a chunk of time making https://github.com/jml/rules_haskell, but had to pause 3 months ago due to workload.

When I paused, I was working on generating rules files from cabal files, in a fashion not dissimilar to gazelle (see rules_go). I’m not sure if that’s pushed anywhere public yet.

• I’m going to deprecate my repo in favour of this one
• I’d appreciate it if you take a look at my repo, see if there are any ideas worth stealing, and steal them
• If you don’t, I’ll try to find anything worth submitting
• Do you feel that something to generate BUILD files (ala gazelle) is in scope for this project? If so,

We should have a haskell_test rule just like java_test. This will get us some common attributes and behaviour (https://docs.bazel.build/versions/master/be/common-definitions.html#common-attributes-tests). In the future this could also be used to integrate directly with tasty &c.

Since 0bba4b8 I have a feeling my linking has been slower. Perhaps we really do want to clean these up before passing them to GHC: if library is mentioned multiple times, do we pay multiple times for it?

However I may be imagining it so this ticket is about measuring the impact on linking itself.

Currently if you have

A, deps = []
B, deps = [A]
exe, deps = [B]

you can import modules from A in exe.

Support builds with documentation.

As we may want to build all documentation, #29 might be related here. Possibly due to the way haddock works, bazel aspects are actually relevant here.

Recently merged f580110 has broken a usecase like this:

haskell-project-1:
  WORKSPACE
  BUILD
  …
haskell-project-2:
  WORKSPACE
  …

haskell-project-2.WORKSPACE:
  local_repository(name = "haskell_project_1", path = "/path/to/haskell-project-1")

haskell-project-2.BUILD:

haskell_library(
  name = "project-2",
  deps = [ "@haskell_project_1" ],

	File "/home/shana/.cache/bazel/_bazel_shana/73d4161e7228d1e130e8631a000a62c8/external/io_tweag_rules_haskell/haskell/path_utils.bzl", line 26, in path_to_module_path
		paths.relativize(hs_file.short_path, pkg_dir)
	File "/home/shana/.cache/bazel/_bazel_shana/73d4161e7228d1e130e8631a000a62c8/external/bazel_skylib/lib/paths.bzl", line 180, in paths.relativize
		fail("Cannot relativize paths above t...")
Cannot relativize paths above the current (unknown) directory

It works when path is used.

In #27 it was pointed out that bazel has toolchain support. We should use it and provide some default toolchains. This could also solve things like #24 for us as version information could be included out of the box.

We uses depset(foo).to_list() in many places to get unique elements of foo. Skylib offers this in a much faster way: build a dictionary with None values then use .keys(): apparently depsets do not support fast membership check and to_list() is definitely bad.

Minor ticket but simple.

We have a problem in that to build anything non-trivial, we need prebuilt_dependencies: this is a list of packages that we expect to be available to GHC. It is up to the user to make sure a toolchain where they are present is used.

Sadly this causes a big inconvenience: the user has to both list everything in prebuilt_dependencies and then manually aggregate all these lists for all packages that toolchain will build back to toolchain definition. This is extremely tedious and goes out of date very quickly. We would like to only list packages in a single place: the dependencies for the package.

There is a way to achieve this and that is to stop using prebuilt_dependencies and instead pass honest bazel targets to deps. The question is where to get these targets. For purpose of this ticket, I will assume nix is still the answer. Note however there is no dependency on rules_nixpkgs from rules_haskell: in theory the user will be able to use the same mechanism with a different backend. It's up to the user.

Instead of generating a toolchain using ghcWithPackages (p: [ every-package-we-need-ever ]), generate package sets. A good candidate for this are LTS packages (but note user is not limited to these: they can cabal2nix/stackage2nix anything they want and use it just the same). The user defines:

http_archive(
  name = "lts",
  repository = …,
  strip_prefix = "lts-collection/lts-10.0",
)

Inside lts-10.0 there's a BUILD file with one bazel target per one package in the snapshot. These targets are provided by lts-10.0/WORKSPACE and may look a little like

nixpkgs_package(
  name = "aeson",
  attribute = "haskellPackages.aeson",
  build_file_content = """
filegroup(
  name = "haskell-outputs",
  …
  srcs = ..., # package db, package conf, libraries, everything needed.
)
"""
)

then inside build file we just have

something_todo(
  name = "aeson",
  target = "@aeson//:haskell-outputs",
)

Finally the user can now say

haskell_library(
  name = "my_lib",
  …,
  deps = [
    "@lts//:aeson"
  ],
)

Where rules_haskell comes in is to make sure it can deal with these dependencies properly: notably that it can find, use and include all necessary files. This for most part will involve -package-db and -package flags into outputs of those packages. Possibly something_todo above can just output a Provider that directly gives us all this information regardless of backend: we can provide default implementation for nix packages somewhere.

We now no longer specify Haskell packages in a toolchain: we unify these on bazel side. This allows us to only specify dependencies where they are needed and removes the tedium.

The reason this works is that bazel targets are lazy: nix (or other backend) will not build the targets until we try to use them. We can therefore safely generate full sets of LTS packages and just pick and choose without inconveniencing users who just need a couple of packages.

What we need is a tool to generate WORKSPACE/BUILD files for LTS (should be simple by re-using stackage2nix), consider if we want to host nix-backend pregenerated package sets in a repository somewhere (I would say yes: requiring that user runs stackage2nix on full LTS is too much: we can run it for every LTS and update periodically + provide instructions how to DYI if users want updates faster; automation of this is possible; having said that there's a new nightly every day so maybe it's not a good idea after all…).

We also need a way on rules_haskell side to unify these packages. We need to implement something_todo for nix backend which provide needed files.

In general I don't think there's that much work though I can imagine quite a few hurdles along the way. For the start we can of course just generate one LTS and use that as POC.

This means adding a test with libs

X
Y deps X
Z deps Y

haddock_library(deps = [Z])

and checking that any docs generated for X are reachable through Y. Should be no problem and work out of the box.

Created from #47.

I'm trying to use https://hackage.haskell.org/package/vector, and I got this BUILD file:

load("@io_tweag_rules_haskell//haskell:haskell.bzl", "haskell_cc_import", "haskell_library")

cc_library(
    name = "empty",
    hdrs = ["include/vector.h"],
)

haskell_cc_import(
    name = "cbits",
    hdrs = [
        "include/vector.h",
        "internal/unbox-tuple-instances",
    ],
    shared_library = ":empty",
)

haskell_library(
    name = "vector",
    srcs = glob(["Data/**/*.*hs"]),
    prebuilt_dependencies = [
        "base",
    ],
    visibility = ["//visibility:public"],
    deps = [
        ":cbits",
    ],
)

This fails, because haskell_cc_import requires ".h", and anyway it's wrong, because vector.h is included in .hs files, it has nothing to do with C, but I can't list .h files as srcs for haskell_library, since it only accepts .hs and .hsc. unbox-tuple-instances is also a haskell file that's #included in .hs files. How can I get this to work?

Currently haskell_haddock provides its own flags to GHC: this is not a complete behaviour. If a package requires certain flags to compile, they have to be passed in through Haddock as well: after all, Haddock basically compiles the sources. It's not viable and it's very error prone to try to reimplement whole build logic we already have inside haskell_haddock. Instead we should capture any relevant flags during the build of the package itself and store the flags for later inspection. This way haskell_haddock should be able to just use these as well without having to re-implement the whole logic that went into generating them. How this should be done is an open question.

Created from #47. Also see comments there.

Different Haddock versions behave differently: in modern Haddock, haddock-utils.js is not created for example. This means that haskell_haddock can break easily if an unexpected version is used.

We should be able to specify/discover Haddock version then perform conditional tasks in haskell_haddock based on it.

Created from #47.

Currently we use the stock io_tweag_rules_haskell//haskell:toolchain. This only checks for presence of ghc, ghc-pkg and hsc2hs. This is fine but is not guaranteed to have Haddock present. Adding a check for haddock there is not correct either: if the user is not using haskell_haddock, they shouldn't have to provide it.

What we should do instead is use toolchain constraints: we could have a constraint for haddock (and ghc/ghc-pkg/hsc2hs too) and let bazel pick a toolchain that meets the constraints and inform the user.

This will involve adding the constraint to our existing toolchain too: it does provide Haddock after all.

Created from #47.

Not all source files can be handed over to as-is to GHC. hsc2hs is one such preprocessor that needs to be called by hand. Cabal in fact defines a number of such hard-coded preprocessors: happy and alex among them. I'm not too worried about those, which are easy for the user to special-case using a genrule. But hsc2hs is a tad tricky.

It sometimes creates multiple output files, sometimes not. At most, these output files are (given an input file A.hsc): A.hs, A_hsc.h, A_hsc.c. Other complications include the fact that the same -I and possibly even -L and -l compiler switches than for ghc need to be passed to hsc2hs as well.

We need a rule of the following form:

haskell_hsc(
  name = "foo",
  src = "Foo.hsc",
)

The user supplies a version number to haskell_toolchain. The user also provides a label locating a GHC install. We should check (at runtime) that the supplied GHC matches the expected version. For more user-friendly errors in case of a mismatch.

We have logic generating flags in c_compile.bzl which is also implemented in toolchain.bzl: for example -package flags. Split out common parts from compilation_defaults and re-use: re-implementing this everywhere is unfun.

Note this is unlike #57 as we're not trying to recover any flags after the fact, we just want to have a common place where they are generated and use that.

We're decent at documenting what the arguments to functions mean, but we nearly never document their return value when not None. This is flagged as a warning by Skylint.

Like in #26, where we moved .hsc files to the srcs, we should also move .c files there. Same reasoning: these files are source files like any other, passed as arguments to the same compiler (GHC).

HaskellPackageInfo contains far too much information and a lot of it is only needed during build: if a user was to inspect this provider for a target, they would find all information clumped together without being any wiser about the target itself.

gather_dependency_info should use a dedicated structure for build information and HaskellPackageInfo should only carry the information that it needs for itself + any other info that's absolutely needed where transitive dependencies are concerned.

There is a linter for Skylark: https://docs.bazel.build/versions/master/skylark/skylint.html.

We should enable it in CI. It will return a non-zero exit code if it found any warning.

As suggested on #85 we should have tests that tests that certain configurations fail. For example that users has to explicitly state direct dependencies &c.

Dyn libs in dynamic-library-dirs section of ghc package files need to have -ghcVERSION name suffix. This is a problem because we don't know the version and we can't generate it dynamically which means we need to provide it as input to the rule. For now I'm just doing a ghcVersion attribute with default value but we need to figure out how to define and use toolchain info instead.

#27 allows users to pull in their own tools to use during build. This lets them use GHC with prebuilt packages being available &c. The default there however uses rules_nixpkgs: in general we do not want to force nix on user.

We should provide basic targets for various GHC versions that will use official binary distribution. This will be plenty for users which only need vanilla GHC tooling to start building their project.

Likely related to #24

Getting a lot of warnings like the following in tests:

hsc-lib-1.0.0: Warning: dynamic-library-dirs: /home/mboes/.cache/bazel/_bazel_mboes/822aa44857e17f47d08bb198503f3032/bazel-sandbox/3199633755278556121/execroot/io_tweag_rules_haskell/bazel-out/k8-fastbuild/bin/tests/hsc/dynlib-hsc-lib-1.0.0 doesn't exist or isn't a directory

And these:

WARNING: cache does not exist: bazel-out/k8-fastbuild/bin/tests/library-deps/sublib/sublib-1.0.0/package.cache
ghc will fail to read this package db. Use 'ghc-pkg recache' to fix.

These need to be addressed.

Most of development was focused with haskell_library in mind and haskell_binary is very poor right now. Factor out code there so that we get various phases for free. ccphs/hsc are example of such code which only works in haskell_library and doesn't fire in haskell_binary

Currently in #47 we rely on documentation for various packages to be neighbours: the links are made to point to ../some-package. This happens to work but it seems to rely on an implementation detail: I don't know if this is guaranteed to be the case and won't just randomly break in some scenarios.

Figure out if there's' a better heuristic we could use. Absolute paths are not really an option as bazel moves things around. I think the way it is now might be OK.

Cabal and stack have a ghci subcommand (e.g. stack ghci) which loads the appropriate environment (compiler, dependencies, runs within docker or nix when appropriate) as well as the files for the project.

This is used by most IDEs (intero, dante, ghcid) to provide feedback to the programmer (type errors, type informations, completion options, etc…).

In order to provide compatibility with these IDEs, there should be an equivalent invocation of ghci with Bazel and rules_haskell, which uses the WORKSPACE and BUILD.

Minimal repro:

import Language.Haskell.TH

main = return ()

with following BUILD file:

haskell_binary(
  name = "binary-with-prebuilt",
  srcs = ["Main.hs"],
  prebuilt_dependencies = ["base"],
)

Expected a build failure, because "template-haskell" not in deps, but the build succeeds.

We may want some benchmark support. Some ideas:

• benchmarks that produce HTML/json/whatever: think criterion outputs
• it could just be a haskell_binary with some bazel tag we pick that identifies it as a benchmark: at the very least the user can ask bazel to list/run all rules with the tag then.

I think we definitely should just start with haskell_binary alias with a rule tag.

Take the following example BUILD file from https://github.com/jin/bazel_haskell_example:

haskell_binary(
    name = "main",
    srcs = ["Main.hs"],
    deps = [
	":a",
	":b",
    ],
)

haskell_library(
    name = "a",
    srcs = ["A.hs"],
)


haskell_library(
    name = "b",
    srcs = ["B.hs"],
)

This throws an error:

→ bazel build :main
ERROR: /Users/jin/Code/rules_haskell_conflict_error/BUILD:18:1: generated file 'package.cache' in rule 'b' conflicts with existing generated file from rule 'a', defined at /Users/jin/Code/rules_haskell_conflict_error/BUILD:12:1.
ERROR: error loading package '': Package '' contains errors.
INFO: Elapsed time: 0.051s

If a Java dependency is provided, add its (transitive?) jar(s?) to CLASSPATH during compilation. This should allow much more natural implementation of things like tweag/sparkle#133

Some source files need preprocessing before being handed over to GHC. One way or another, they are still source files and the good news is that which preprocessor to run is always clear from the name of the source file alone. So we can list them in srcs along with the rest and add an extra action for those files that end in .hsc. Easier for the users: sources in srcs, dependencies in deps, etc.

Frequently libraries live under src or similar. As we are guessing module names from file paths, we can only translate <project-root>A/B/C.hs right now. The way this is solved in cabal is simple: hs-source-dirs. Instead of using srcs as a flat list of files, we should take (directory, files) pairs instead. This way we can drop the directory prefix when needed.

Some alternative UI choices are possible.

We should have support for HPC: https://wiki.haskell.org/Haskell_program_coverage .

Existing tools are very bad at providing this. It's unclear what exactly this feature would do: a first approximation might be enabling hpc in libraries and running a binary with hpc. Possibly part of #29.

We now use ctx.host_fragments.cpp.ar_executable to get ar. We may want to consider making the user pass ar as part of toolchain instead: this way we always ensure it's available and that it's the version user expects.

However ar is usually available/doesn't change so it's low priority.

We need this attribute to know what library name GHC will be trying to look for during linking. Currently the user has to specify it and it's error prone: even on master, this is set to 8.2.2 while CI uses GHC 8.0.2: this will give the user non-obvious error messages the moment they want to use shared library dependencies.

We should be able to generate sources and source links, preferably hyperlinked.

This shouldn't be too hard: we do need to replicate source file hierarchy but we already are able to do this. We may need extra tool in toolchain (#55) from what I remember.

We need support for compilation modes. An easy example is profiling build. I have previously evaluated aspects and they are not suited. I have since found out that Go rules support this (https://github.com/bazelbuild/rules_go/blob/master/go/modes.rst#id1) so we may want to do something similar.

We may want to support prebuilt_dependencies of packages in haskell_haddock. This involves:

• finding out paths of prebuilt_dependencies
• adding them to --read-interface

We may be able to get these paths from ghc-pkg. To feed these paths into haddock itself we may want to use run_shell and interpolate or generate a response file with all the args prepopulated (but still needs run_shell most likely). These things can be a bit more granular by abusing files: for every prebuilt dependency, run ghc-pkg and produce a file with the path needed. Then when invoking haddock use $(cat path_of_prebuilt_dep_FOO) $(cat path_of_prebuilt_dep_BAR). This allows us to only invoke ghc-pkg once per every prebuilt dependency across all rules rather than call ghc-pkg for every single prebuilt dep for every single dep in the package which would likely be rather slow.

Before anything, think about whether we actually want this as a feature. prebuilt_dependencies is somewhat of a hack to begin with and maybe we should be encouraging to package more things and use prebuilt_dependencies less rather than encouraging its use.

Created from #47.

These paths are used in shared object libraries. We could get the needed libs from bazel in case of something like inline-java but it's unusable: best we can do is a relative path to external and during builds, the shared library moves around: the relative path becomes useless.

I don't know how to sanely solve this. For now I have something like

  external_libraries = {
    "jvm": "/nix/store/f4vdrkqqh71wffq3zz10ymywg3x28acw-openjdk-8u152b16/lib/openjdk/jre/lib/amd64/server",
…
}

but it's not really a solution…

ghc accepts literate Haskell files. So should these rules. I can't build MissingH because of it.