mietek / halcyon Goto Github PK

View Code? Open in Web Editor NEW

168.0 7.0 20.0 1.14 MB

System for installing Haskell apps

License: BSD 3-Clause "New" or "Revised" License

Shell 100.00%

halcyon's Introduction

This project is no longer maintained.

Halcyon

Halcyon is a system for installing Haskell apps and development tools, including GHC and Cabal.

Follow the Halcyon tutorial to get started.

Usage

Halcyon provides the halcyon install command, which can be used to install Haskell apps:

$ halcyon install https://github.com/mietek/halcyon-tutorial

Installation

Halcyon can be installed by cloning the Halcyon source repository:

$ git clone https://github.com/mietek/halcyon

Alternatively, you can run the Halcyon setup script, which also installs the necessary OS packages and sets up the environment:

$ eval "$( curl -sL https://github.com/mietek/halcyon/raw/master/setup.sh )"

Documentation

Start with the Halcyon tutorial to learn how to deploy a simple Haskell web app using Halcyon.
See the Halcyon reference for a complete list of available commands and options.
Read the Haskell on Heroku tutorial to learn how to deploy Haskell web apps to Heroku.

Internals

Halcyon is written in GNU bash, using the bashmenot library.

Read the Halcyon source code to understand how it works.

About

Made by Miëtek Bak. Published under the BSD license.

halcyon's People

Contributors

Stargazers

Watchers

Forkers

circuithub timjb pbrisbin aherrmann rjmac yanhick nashera pharpend divarvel gavinwhyte bssstudio nkondratyev tolysz xtendo-org jgallen merlinnot doppioslash mdmm2 akilegaspi

halcyon's Issues

Improve Cabal version and remote-repo handling

Halcyon was originally built with the assumption that the version number of cabal-install and the contents of its package database don’t need to be treated as app dependencies.

Unfortunately, build directories created with cabal-install 1.20.0.3 are not usable with cabal-install 1.22.0.0 (haskell/cabal#2320). Additionally, recent Hackage issues (haskell/hackage-server#305) and the slow adoption of constraints files put the original assumption further into question. Therefore, I propose to:

Ensure that sandbox directories, build directories, and install directories are built with the right version of cabal-install, and that changes to Cabal magic files are reflected in the magic hash.
Allow setting HALCYON_CABAL_VERSION with a .halcyon/cabal-version magic file.
Allow setting HALCYON_CABAL_REMOTE_REPO with a .halcyon/cabal-remote-repo magic file.
Avoid needless warnings and cabal update runs when using Stackage as the Cabal remote-repo.

This will have the interesting benefit of making it practical for install archives of popular Haskell apps to be included in Halcyon public storage, built using Stackage and not explicit version constraints, and for Halcyon to be suggested as a simple installation method for these apps.

The work is already done and awaits further testing. The remaining questions are:

Should changing the Cabal remote-repo trigger a sandbox rebuild, just like changing the Cabal version? If yes, all Halcyon public archives would have to be built using the default Halcyon Cabal remote-repo.
Therefore, should Stackage LTS be the default Cabal repository for Halcyon?

CC @begriffs @dbp @jspahrsummers @ozataman @pbrisbin @rehno-lindeque @snoyberg

Add option to not remove generated Haddock/documentation in sandbox

We use the documentation generated in the sandbox.

Support keeping build products in source directory

By default, Halcyon performs builds in a temporary build directory.

Some established use-cases can depend on build products being available in the source directory after a build is performed.

In other cases, copying the source directory can take too much time to be performed on every build (#59 (comment)).

Currently, one workaround is to use Halcyon for building the sandbox with the HALCYON_DEPENDENCIES_ONLY option, and to point cabal-install to the Halcyon sandbox. Improving the related documentation is tracked as mietek/halcyon-website#2.

This is related to #11.

Add curl as dependency

I'm in the linux-ubuntu-14 case (in setup.sh), which installs

build-essential
git
pigz
zlib1g-dev

I'm on a clean system and it would be nice to add explicitely curl in the above list. (Halcyon simply fails to download anything, but it was unclear at first that curl was missing.)

Thanks a lot for working on this!

Support Windows

Halcyon depends on bashmenot, which depends primarily on GNU bash. Both projects already support Linux and BSD, so maybe supporting Windows with a Cygwin-like compatibility layer (Gow, perhaps?) isn’t too far-fetched.

This depends on mietek/bashmenot#7.

Improve Stackage support

Using a particular Stackage snapshot appears to be a good alternative to explicitly specifying version constraints.

This is related to #40.

Support static linking

Hi,

I was wondering if supporting building statically linked executables might be a stated goal of Halcyon.

Thanks!

Support OS X fully

Most of the work is already done. Everything is expected to work on OS X, except installing native OS packages declared with HALCYON_SANDBOX_EXTRA_OS_PACKAGES or HALCYON_EXTRA_OS_PACKAGES.

Improve the setup process for GNU bash 3, as it doesn’t recognise the source <( … ) syntax.

$ curl -sL https://github.com/mietek/halcyon/raw/master/setup.sh >/tmp/halcyon-setup.sh && . /tmp/halcyon-setup.sh
$ /app/halcyon/halcyon paths >/tmp/halcyon-paths.sh && . /tmp/halcyon-paths.sh

Provide public storage archives for OS X 10.9 and 10.10.
Speed up stripping GHC on OS X (#43)
Prevent the OS X linker from complaining about non-existent directories passed to cabal install.
```
ld: warning: directory not found for option '-L/app/sandbox/usr/lib'
```

Prevent the OS X linker from complaining about non-existent directories in LD_LIBRARY_PATH.

ld: warning: directory not found for option '-L/app/lib'
ld: warning: directory not found for option '-L/app/usr/lib'
ld: warning: directory not found for option '-L/app/ghc/usr/lib'
ld: warning: directory not found for option '-L/app/sandbox/usr/lib'

Ensure the dependency on GNU bash 4, GNU date, and GNU sort is documented, or removed.
```
$ brew install bash coreutils
```
Wrap around brew to support installing native OS packages.

Depends on mietek/bashmenot#2.

Unexpected constraints difference

Using Cabal to freeze constraints before installing dependencies may sometimes give a different result than freezing constraints after dependencies are installed, even with all constraints explicitly declared in a cabal.config file.

This does not adversely affect the operation of Halcyon or Haskell on Heroku. The underlying Cabal bug is tracked as haskell/cabal#1896.

This page is intended to collect and display live bug specimens. If you reproduce the issue, please add a comment. To help me keep a handle on things, please include:

Project name

GHC version
Cabal version
Link to a gist of the entire build log
Link to the commit exhibiting the issue

Warning message, including the diff, inline

Improve and document cross-platform support

Platforms

Tier 1

Tested automatically.

Use of HALCYON_EXTRA_OS_PACKAGES and HALCYON_SANDBOX_EXTRA_OS_PACKAGES is supported.

Name	Platform identifier	32-bit	64-bit
Amazon Linux 2014.09	`linux-amzn-2014.09`		✔︎
CentOS 6	`linux-centos-6`	✔︎¹	✔︎
CentOS 7	`linux-centos-7`	✔︎	✔︎
Debian 6	`linux-debian-6`	✔︎¹	✔︎¹
Debian 7	`linux-debian-7`	✔︎	✔︎
Debian 8	`linux-debian-8`	✔︎	✔︎
Fedora 20	`linux-fedora-20`	✔︎	✔︎
Fedora 21	`linux-fedora-21`		✔︎
Red Hat Enterprise Linux 6	`linux-rhel-6`		✔︎
Red Hat Enterprise Linux 7	`linux-rhel-7`		✔︎
Ubuntu 10.04 LTS	`linux-ubuntu-10.04`	✔︎¹	✔︎¹
Ubuntu 12.04 LTS	`linux-ubuntu-12.04`	✔︎	✔︎
Ubuntu 14.04 LTS	`linux-ubuntu-14.04`	✔︎	✔︎
Ubuntu 14.10	`linux-ubuntu-14.10`	✔︎	✔︎
Ubuntu 15.04	`linux-ubuntu-15.04`	✔︎	✔︎

Tier 2

Tested automatically.

Name	Platform identifier	64-bit	Details
Arch Linux	`linux-arch`	✔︎	#26
openSUSE 13.2	`linux-opensuse-13.2`	✔︎
SUSE Linux Enterprise Server 12	`linux-sles-12`	✔︎

Tier 3

Tested manually.

Name	Platform identifier	32-bit	64-bit	Details
Gentoo Linux	`linux-gentoo`		✔︎
OS X 10.8	`osx-10.8`	✔︎²	✔︎	#16
OS X 10.9	`osx-10.9`	✔︎²	✔︎	#16
OS X 10.10	`osx-10.10`	✔︎²	✔︎	#16
Slackware 14.1	`linux-slackware-14.1`		✔︎

Notes

GHC 7.10.1 is not supported on 32-bit CentOS 6, Debian 6, or Ubuntu 10.04 LTS yet.
32-bit GHC 7.8 and newer is not supported on OS X, and probably won’t be.

No details for HTTP 400 Bad Request

I am trying to get Amazon S3 private storage working with Haskell on Heroku.
I have created a user with full access to S3, and configured variables (bucket is located in Frankfurt):

$ heroku config:set HALCYON_AWS_ACCESS_KEY_ID=XXXXXXX
$ heroku config:set HALCYON_AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXX
$ heroku config:set HALCYON_S3_BUCKET=myproject
$ heroku config:set HALCYON_S3_ENDPOINT=s3-eu-central-1.amazonaws.com

Then I have tried to deploy app:

$ git push heroku master
Counting objects: 130, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (83/83), done.
Writing objects: 100% (130/130), 8.84 MiB | 1.26 MiB/s, done.
Total 130 (delta 34), reused 130 (delta 34)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Fetching custom git buildpack... done
remote: -----> Haskell app detected
remote:
remote:
remote: -----> Welcome to Haskell on Heroku
remote:        HALCYON_AWS_ACCESS_KEY_ID:                (secret)
remote:        HALCYON_AWS_SECRET_ACCESS_KEY:            (secret)
remote:        HALCYON_NO_BUILD_DEPENDENCIES:            0
remote:        HALCYON_S3_BUCKET:                        myproject
remote:        HALCYON_S3_ENDPOINT:                      s3-eu-central-1.amazonaws.com
remote:
remote: -----> Installing buildpack... done, e52a46e
remote: -----> Installing Halcyon... done, fd5fc7d
remote: -----> Installing bashmenot... done, 5165edd
remote: -----> Installing yesod23-0.0.0
remote:        Label:                                    yesod23-0.0.0
remote:        Prefix:                                   /app
remote:        Source hash:                              4800b73
remote:        External storage:                         private and public
remote:        GHC version:                              7.8.4
remote:
remote: -----> Restoring install directory
remote:        Downloading s3://myproject/linux-ubuntu-14.04-x86_64/ghc-7.8.4/halcyon-install-4800b73-yesod23-0.0.0.tar.gz... 400 (bad request)
remote:        Downloading https://halcyon.global.ssl.fastly.net/linux-ubuntu-14.04-x86_64/ghc-7.8.4/halcyon-install-4800b73-yesod23-0.0.0.tar.gz... 404 (not found)
remote:
remote: -----> Determining constraints
remote:        Label:                                    yesod23-0.0.0
remote:        Prefix:                                   /app
remote:        Source hash:                              4800b73
remote:        Constraints hash:                         00eb166
remote:        External storage:                         private and public
remote:        GHC version:                              7.8.3
remote:        Cabal version:                            1.20.0.3
remote:        Cabal repository:                         Hackage
remote:
remote: -----> Restoring GHC directory
remote:        Downloading s3://myproject/linux-ubuntu-14.04-x86_64/halcyon-ghc-7.8.3.tar.gz... 400 (bad request)
remote:        Downloading https://halcyon.global.ssl.fastly.net/linux-ubuntu-14.04-x86_64/halcyon-ghc-7.8.3.tar.gz... done
remote:        Uploading s3://myproject/linux-ubuntu-14.04-x86_64/halcyon-ghc-7.8.3.tar.gz... 400 (bad request)
remote:        Retrying in 2 seconds (1/5)... 400 (bad request)
remote:        Retrying in 4 seconds (2/5)... 400 (bad request)
remote:        Retrying in 8 seconds (3/5)... 400 (bad request)
remote:        Retrying in 16 seconds (4/5)... 400 (bad request)
remote:    *** ERROR: Failed to upload cached file
remote: -----> Building GHC directory
remote:        Downloading s3://myproject/original/ghc-7.8.3-x86_64-unknown-linux-deb7.tar.xz... 400 (bad request)
remote:        Downloading https://halcyon.global.ssl.fastly.net/original/ghc-7.8.3-x86_64-unknown-linux-deb7.tar.xz... done
remote:        Uploading s3://myproject/original/ghc-7.8.3-x86_64-unknown-linux-deb7.tar.xz... 400 (bad request)
remote:        Retrying in 2 seconds (1/5)... ^C

As you can see there's something going wrong with S3 download/upload. But as far as I can see there is no way to actually find out the details of 400 (bad request) response.

Pass flags to cabal freeze to ensure all targets are included

When trying to find a cached sandbox, Halcyon uses cabal freeze to determine the current dependency tree. By default, this excludes the test target (and maybe others?). The result is that if you add a test dependency, Halcyon may use an existing sandbox and not install the required packages. Trying to then run tests in this environment will fail.

The solution would be to pass --enable-tests to cabal freeze.

Create a tag?

I'd love to submit Halcyon to Homebrew (though it remains to be seen whether they'll accept it), but one of their stipulations is that the project have tagged releases.

Could you please create a tag? 🙏

Increase isolation

Avoid making any changes to the user’s home directory.

When running the setup script for the first time, Halcyon extends ~/.bash_profile to source the script output by halcyon paths. This should be made optional.
When installing an app, Halcyon tries to create a symlink pointing ~/.cabal/config to the config file located in HALCYON_BASE/cabal, and emits a warning if ~/.cabal/config already exists. This is intended to ensure the user is using the same Cabal config as Halcyon, and isn’t needed by Halcyon itself. This should be replaced by setting the (undocumented?) CABAL_CONFIG environment variable in halcyon paths.
Halcyon immediately fails with an error if ~/.ghc exists, due to #5 and the underlying haskell/cabal#2400. This should be turned into a warning.
HALCYON_NO_MODIFY_HOME should be removed as no longer necessary.

CC @snoyberg

Failure with yesod-bin as sandbox-extra-app

% cat .halcyon/sandbox-extra-apps
alex-3.1.3 happy-1.19.4 yesod-bin-1.4.3
% cat bin/build
#!/bin/bash
export HALCYON_NO_SELF_UPDATE=1
export HALCYON_SANDBOX_EXTRA_CONFIGURE_FLAGS='--enable-tests'

source ./.env # AWS-related variables
source <(/path/to/halcyon paths)

halcyon build
% bin/build

--SNIP--

       -----> Restoring build directory
              Downloading s3://thoughtbot-haskell-packages/linux-arch-x86_64/ghc-7.8.3/halcyon-build-yesod-bin-1.4.3.tar.gz... 404 (not found)
              Downloading https://halcyon.global.ssl.fastly.net/linux-arch-x86_64/ghc-7.8.3/halcyon-build-yesod-bin-1.4.3.tar.gz... 404 (not found)
       -----> Configuring app
       -----> Building app
              Building yesod-bin-1.4.3...

--SNIP--

              Preprocessing executable 'yesod' for yesod-bin-1.4.3...
              [ 1 of 10] Compiling GhcBuild         ( GhcBuild.hs, dist/build/yesod/yesod-tmp/GhcBuild.o )
              [ 2 of 10] Compiling Build            ( Build.hs, dist/build/yesod/yesod-tmp/Build.o )
              [ 3 of 10] Compiling HsFile           ( HsFile.hs, dist/build/yesod/yesod-tmp/HsFile.o )
              [ 4 of 10] Compiling Scaffolding.Scaffolder ( Scaffolding/Scaffolder.hs, dist/build/yesod/yesod-tmp/Scaffolding/Scaffolder.o )

              Scaffolding/Scaffolder.hs:66:23:
                  Exception when trying to run compile-time code:
                    hsfiles/minimal.hsfiles: openBinaryFile: does not exist (No such file or directory)
                  Code: embedFile "hsfiles/minimal.hsfiles"
                  In the splice: $(embedFile "hsfiles/minimal.hsfiles")

I'm reporting this because I know part of halcyon is building things in various directories, so the inability to find a source file during compilation may be a halcyon bug. If it's not, I apologize.

Halcyon breaks if ~/.ghc is present

Here's the script I'm running to try and build a local project using halcyon:

#!/bin/bash
set -e

builddir="$HOME/halcyon-build"

#mkdir -p "$builddir"
#git clone https://github.com/mietek/halcyon.git "$builddir/halcyon"

export HALCYON_BASE="$builddir"
export HALCYON_PREFIX="$builddir"
export HALCYON_NO_SELF_UPDATE=1

export HALCYON_AWS_ACCESS_KEY_ID='x'
export HALCYON_AWS_SECRET_ACCESS_KEY='x'
export HALCYON_S3_BUCKET='x'

source <( "$builddir/halcyon/halcyon" paths )

halcyon build

The script is being run from within the project root.

GHC and Cabal installed OK, but it failed on the first extra app, alex:

-----> Installing carnival-0.0.0
-----> Determining constraints
       Label:                                    carnival-0.0.0
       Prefix:                                   /home/patrick/code/carnival/halcyon-build
       Source hash:                              813f5e7
       Constraints hash:                         b829d67
       Magic hash:                               2528ea3
       External storage:                         private and public
       GHC version:                              7.8.3
       Cabal version:                            1.22.0.0
       Cabal repository:                         Hackage
       Sandbox magic hash:                       c0b8c0e
       Sandbox extra apps:                       alex-3.1.3
                                                 happy-1.19.4

-----> Using existing GHC

-----> Using existing Cabal directory
   *** WARNING: Unexpected existing Cabal config
   *** WARNING: To replace with recommended Cabal config:
       $ ln -fs /home/patrick/code/carnival/halcyon-build/cabal/.halcyon-cabal.config ~/.cabal/config

-----> Restoring sandbox directory
       Downloading s3://thoughtbot-haskell-packages/linux-arch-x86_64/ghc-7.8.3/halcyon-sandbox-b829d67.c0b8c0e-carnival-0.0.0.tar.gz... 404 (not found)
       Downloading https://halcyon.global.ssl.fastly.net/linux-arch-x86_64/ghc-7.8.3/halcyon-sandbox-b829d67.c0b8c0e-carnival-0.0.0.tar.gz... 404 (not found)
-----> Locating sandbox directories
       Listing s3://thoughtbot-haskell-packages/?prefix=linux-arch-x86_64/ghc-7.8.3/halcyon-sandbox-... done
       Listing https://halcyon.global.ssl.fastly.net/... done
-----> Building sandbox directory
-----> Creating sandbox
-----> Installing sandbox extra apps
       -----> Unpacking app
       -----> Installing alex-3.1.3
       -----> Determining constraints
          *** WARNING: Using newest versions of all packages
              QuickCheck-2.7.6
              array-0.5.0.0
              base-4.7.0.1
              bytestring-0.10.4.0
              containers-0.5.5.1
              deepseq-1.3.0.2
              directory-1.2.1.0
              filepath-1.3.0.2
              ghc-prim-0.3.1.0
              integer-gmp-0.5.1.0
              old-locale-1.0.0.6
              pretty-1.1.1.1
              primitive-0.5.4.0
              random-1.1
              rts-1.0
              template-haskell-2.9.0.0
              tf-random-0.5
              time-1.4.2
              transformers-0.3.0.0
              unix-2.7.0.1

              Label:                                    alex-3.1.3
              Prefix:                                   /home/patrick/code/carnival/halcyon-build/sandbox
              Source hash:                              86b14de
              GHC version:                              7.8.3
              External storage:                         private and public

       -----> Restoring install directory
              Downloading s3://thoughtbot-haskell-packages/linux-arch-x86_64/ghc-7.8.3/halcyon-install-86b14de-alex-3.1.3.tar.gz... 404 (not found)
              Downloading https://halcyon.global.ssl.fastly.net/linux-arch-x86_64/ghc-7.8.3/halcyon-install-86b14de-alex-3.1.3.tar.gz... 404 (not found)

              Label:                                    alex-3.1.3
              Prefix:                                   /home/patrick/code/carnival/halcyon-build/sandbox
              Source hash:                              86b14de
              Constraints hash:                         9d915b2
              External storage:                         private and public
              GHC version:                              7.8.3
              Cabal version:                            1.22.0.0
              Cabal repository:                         Hackage

       -----> Restoring sandbox directory
              Extracting halcyon-sandbox-9d915b2-alex-3.1.3.tar.gz... done, 7.3MB

       -----> Restoring build directory
              Downloading s3://thoughtbot-haskell-packages/linux-arch-x86_64/ghc-7.8.3/halcyon-build-alex-3.1.3.tar.gz... 404 (not found)
              Downloading https://halcyon.global.ssl.fastly.net/linux-arch-x86_64/ghc-7.8.3/halcyon-build-alex-3.1.3.tar.gz... 404 (not found)
       -----> Configuring app
              Using a sandbox located at /home/patrick/code/carnival/halcyon-build/sandbox
              /home/patrick/code/carnival/halcyon-build/ghc/bin/ghc --numeric-version
              looking for tool ghc-pkg near compiler in
              /home/patrick/code/carnival/halcyon-build/ghc/bin
              found ghc-pkg in /home/patrick/code/carnival/halcyon-build/ghc/bin/ghc-pkg
              /home/patrick/code/carnival/halcyon-build/ghc/bin/ghc-pkg --version
              /home/patrick/code/carnival/halcyon-build/ghc/bin/ghc --supported-languages
              /home/patrick/code/carnival/halcyon-build/ghc/bin/ghc --info
              Reading available packages...
              Choosing modular solver.
              Resolving dependencies...
              creating dist/setup
              /home/patrick/code/carnival/halcyon-build/ghc/bin/ghc --make -odir ./dist/setup -hidir ./dist/setup -i -i. -package-db /home/patrick/code/carnival/halcyon-build/sandbox/x86_64-linux-ghc-7.8.3-packages.conf.d -package-id Cabal-1.20.0.2-842e7912fac8797e77f21298825e4931 ./dist/setup/setup.hs -o ./dist/setup/setup -threaded
              <command line>: cannot satisfy -package-id Cabal-1.20.0.2-842e7912fac8797e77f21298825e4931: 
                  Cabal-1.20.0.2-842e7912fac8797e77f21298825e4931 is unusable due to missing or recursive dependencies:
                    array-0.5.0.0-98aa445e59f3eb0c886795ff07406d84 base-4.7.0.1-e4b74d27ad8c8987c63abc42a80e7335 bytestring-0.10.4.0-9f46be651278ecf2e2b21a220bfff6bf containers-0.5.5.1-23e2a2b94d6e452c773209f31d8672c5 deepseq-1.3.0.2-733fe43e121f761739636bd6670bea77 directory-1.2.1.0-4c475f14fe8252aac085a98bf26e5baf process-1.2.0.0-163b32a503548d746f4aa93b497ff8ea time-1.4.2-bf925e935c287d0b75398fe297453c28 unix-2.7.0.1-0d8e7ccf059d972d7aad20e259a6e218
                  (use -v for more information)
          *** ERROR: Failed to configure app
   *** WARNING: Cannot install sandbox extra apps
   *** WARNING: Cannot build sandbox directory
   *** WARNING: Cannot install app

shell returned 1

When I mv ~/.ghc ~/ghc.bak, the problem went away and the build proceeded.

Add and improve glob-based options

Change HALCYON_EXTRA_DATA_FILES to take find -name patterns instead of GNU bash globs, for consistency with HALCYON_EXTRA_SOURCE_HASH_IGNORE.
Add HALCYON_EXTRA_SOURCE_FILES and HALCYON_EXTRA_DOC_FILES to work around globbing issues with extra-source-files and extra-doc-files — same as with data-files (haskell/cabal#713, haskell/cabal#784).

`--ignore-all-constraints` doesn't seem to be honored.

I'm running this command:

> /app/halcyon/halcyon install /source --ignore-all-constraints --sandbox-extra-apps=.halcyon-ghc-7.8.4/sandbox-extra-apps

There is no .halcyon/constraints file in the package (at /source).
The sandbox-extra-apps is there to add Alex and Happy but does't affect this bug report.

Halcyon output is as follow:

-----> Self-updating bashmenot... done, 5165edd
-----> Examining cache contents
       halcyon-cabal-1.20.0.3.tar.gz
       halcyon-cabal-1.20.0.3-hackage-2015-02-16.tar.gz
       halcyon-ghc-7.8.4.tar.gz

-----> Installing hadley-0.0.0
-----> Installing GHC and Cabal
       External storage:                         public
       GHC version:                              7.8.4
       Cabal version:                            1.20.0.3
       Cabal repository:                         Hackage

-----> Using existing GHC directory

-----> Locating Cabal directories
       Listing https://halcyon.global.ssl.fastly.net/?prefix=linux-ubuntu-14.04-x86_64/halcyon-cabal-1.20.0.3-hackage-... done
-----> Restoring base Cabal directory
       Extracting halcyon-cabal-1.20.0.3.tar.gz... done, 13MB
-----> Updating Cabal directory
       Updating Cabal package database... done, 186MB
-----> Archiving Cabal directory
       Creating halcyon-cabal-1.20.0.3-hackage-2015-02-22.tar.gz... done, 20MB

-----> Determining constraints
       cabal: Could not resolve dependencies:
       trying: hadley-0.0.0 (user goal)
       trying: base-4.7.0.2/installed-bfd... (dependency of hadley-0.0.0)
       next goal: directory (dependency of hadley-0.0.0)
       rejecting: directory-1.2.1.0/installed-07c..., 1.2.1.0, 1.2.0.1, 1.2.0.0
       (conflict: hadley => directory==1.1.*)
       rejecting: directory-1.1.0.2 (conflict: base==4.7.0.2/installed-bfd...,
       directory => base>=4.4 && <4.6)
       rejecting: directory-1.1.0.1 (conflict: base==4.7.0.2/installed-bfd...,
       directory => base>=4.2 && <4.5)
       rejecting: directory-1.1.0.0 (conflict: base==4.7.0.2/installed-bfd...,
       directory => base>=4.2 && <4.4)
       rejecting: directory-1.0.1.2, 1.0.1.1, 1.0.1.0, 1.0.0.3, 1.0.0.0 (conflict:
       hadley => directory==1.1.*)
       Dependency tree exhaustively searched.
   *** ERROR: Failed to determine constraints

In particular, note that it rejects directory because it says hadley (the package being installed) specifies a specific version. I thought that the --ignore-all-constraints would give something similar to manually removing all bounds from the .cabal file.

Add --needed to pacman command

https://github.com/mietek/halcyon/blob/master/setup.sh#L11

If things are already installed, pacman will re-install. If you pass --needed, it will skip them.

Use symlinks instead of copying local/relative directories in `.halcyon/sandbox-sources`?

Would there be any disadvantage to symlinking local sources listed in .halcyon/sandbox-sources instead of copying them to /app/sandbox/.halcyon-sandbox-sources/?

Given...

Directory structure

We use an "uber"-repo for our code that looks similar to this:

app/
- .cabal-sandbox/
- .halcyon/
  - sandbox-sources
- lib/
  - internal-library-1/
    - internal-library-1.cabal
  - internal-library-2/
    - internal-library-2.cabal
- src/
  - Main.hs
- app.cabal

`.halcyon/sandbox-sources`

./lib/internal-library-1
./lib/internal-library-2

Goal

We want to be able to:

Edit the files directly in app/lib/
Use cabal build/cabal test

This wouldn't work since cabal sandbox list-sources would point to /app/sandbox/.halcyon-sandbox-sources/, and we would be editing the wrong code.

Use reduced redundancy storage for S3 private storage

From the Using Reduced Redundancy Storage documentation:

In order to reduce storage costs, you can use reduced redundancy storage for noncritical, reproducible data at lower levels of redundancy than Amazon S3 provides with standard storage. The lower level of redundancy results in less durability and availability, but in many cases, the lower costs can make reduced redundancy storage an acceptable storage solution.

Since Halcyon sandboxes and apps can always be rebuilt from scratch in the event of data loss on S3, it seems like this should be the default.

Support installing Haskell libraries

Currently, Halcyon is intended to install Haskell apps — Cabal packages which provide executables, not just libraries.

Libraries installed with Halcyon are not automatically registered in any GHC package database. However, Halcyon makes the library .conf file available for manual registration.

For example, to install the turtle-1.0.1 library and register it within the Cabal sandbox managed by Halcyon:

$ halcyon install turtle-1.0.1
$ ghc-pkg register --global --package-db=/app/sandbox/x86_64-linux-ghc-7.8.4-packages.conf.d /app/.halcyon/turtle-1.0.1/turtle-1.0.1.conf

The library is now ready to use:

$ cabal --sandbox-config-file=/app/sandbox/cabal.sandbox.config repl
GHCi, version 7.8.4: http://www.haskell.org/ghc/  :? for help
…
Prelude> import Turtle
Prelude Turtle>

This isn’t a recommended workflow, as the Halcyon sandbox is intended to be transient.

Improve support for relative paths

Currently, relative paths which reference the parent directory, such as ../other_dir, are not supported by any option which accepts directory paths, such as HALCYON_SANDBOX_SOURCES.

There should also be an option to locate the source files in a subdirectory of the directory given as an argument to halcyon install.

Closely related to #10 and #14.

Support building GHC from source

Moved from mietek/haskell-on-heroku#15.

Passing --cabal-version didn't seem to work on CicleCI

Our build script passes --cabal-version here.

This works locally and on Travis CI. When we tried to move to CircleCI, the flag seemed to be ignored. See build log here -- it incorrectly uses 1.20

Setting HALCYON_CABAL_VERSION in the environment did work though.

Default to no-clean-private-storage

I have no idea, but I think it's more common to work on a multi-person team where it's desirable to have the --no-clean-private-storage behavior. I'd suggest making this the default. S3 is very cheap, and in general I would err on the side of keeping more data than reducing storage costs.

Speed up stripping GHC on OS X

Stripping most versions of GHC on OS X is, inexplicably, much slower than stripping GHC 7.8.3 on OS X (all with strip -u -r), or than stripping the same GHC versions on other platforms (all with strip --strip-unneeded). This affects both OS X 10.9 and 10.10. All results refer to official bindists or soon-to-be official bindists.

OS X 10.10

GHC	Installed size	Doc-less size	Stripped size	Stripping time
7.10.1-rc2	920 MB	787 MB	728 MB	100 minutes
7.8.4	809 MB	679 MB	635 MB	65 minutes
7.8.3	729 MB	600 MB	559 MB	188 seconds
7.8.2	817 MB	691 MB	644 MB	69 minutes
7.6.3	856 MB	732 MB	636 MB	54 minutes
7.6.1	855 MB	731 MB	635 MB	52 minutes
7.4.2	842 MB	723 MB	629 MB	50 minutes
7.2.2	693 MB	581 MB	527 MB	51 minutes
7.0.4	647 MB	536 MB	487 MB	49 minutes

Ubuntu 14.04

GHC	Installed size	Doc-less size	Stripped size	Stripping time
7.10.1-rc2	1.1 GB	899 MB	811 MB	112 seconds
7.8.4	928 MB	789 MB	701 MB	100 seconds
7.8.3	928 MB	789 MB	701 MB	97 seconds
7.8.2	936 MB	801 MB	708 MB	96 seconds
7.6.3	893 MB	768 MB	650 MB	84 seconds
7.6.1	892 MB	767 MB	649 MB	87 seconds
7.4.2	876 MB	755 MB	641 MB	83 seconds
7.2.2	842 MB	728 MB	644 MB	82 seconds
7.0.4	797 MB	687 MB	598 MB	80 seconds

CC @mzero @etrepum

Setup script without root or sudo access

Enable the setup.sh script to run with an unprivileged user without sudo access when :

the HALCYON_BASE is present and empty and have the correct permissions
all the OS packages dependency are met

Support Amazon Linux

I'm trying to use Halcyon on the Amazon Linux AMI and I get this error:

-----> Welcome to Halcyon
-----> Creating base directory: /app
-----> Installing OS packages
*** ERROR: Unexpected platform: linux-amzn-2014.09-x86_64
*** ERROR: Failed to install OS packages
*** ERROR: Failed to install Halcyon

Wikipedia says that this distro is basically rhel. Is there an environment variable or something I could set to get the installer to pick the right platform?

Support OpenBSD and FreeBSD fully

Most things are expected to work, except automatic setup, or installing native OS packages declared with HALCYON_SANDBOX_EXTRA_OS_PACKAGES or HALCYON_EXTRA_OS_PACKAGES.

This depends on mietek/bashmenot#6.

Use --progress-bar in curl calls

Sometimes the S3 downloads take a bit, it'd be nice to have progress output.

Improve excluding files from source hash

Add an option to customise the list of files which are excluded from consideration when calculating the source hash. Use globs.

Ensure excluded files are available in the source directory after the source hash is calculated.

Equivalent of `cabal install xxx --configure-option=` in halcyon install ?

What is the equivalent of doing
cabal install hruby --configure-option=-- rubyversion=18 --rubylib=ruby --rubyinc=/usr/lib64/ruby/1.8/x86_64-linux/ when using halcyon install?

A bit of context: I am trying to build language-puppet. Using a dinosaur OS (CentOS 6) I need to build the hruby dependency with the above command (it works fine in a cabal sandbox).

I had a go with "HALCYON_EXTRA_CONFIGURE_FLAGS" but without success.

This is not a big issue because I can use cabal sandbox to build the executable but succeeding an halcyon install would be nice.

Install failing on osx 10.9.2

I have this error message:

➜  dev  source <( halcyon/halcyon paths )
/Users/rvion/dev/halcyon/lib/bashmenot/src/platform.sh: line 38: syntax error near unexpected token `;'

Make using local sandboxes easier

(I had a brief look through the other open issues mentioning "sandbox", I hope I didn't miss a duplicate of this one.)

I only found out through the Giant Robots blog post that you can actually link the global sandbox config to your local project, e.g. through

ln -sf /app/sandbox/cabal.sandbox.config .

This means that my editor integration now works again as expected and I can use cabal repl like I'd normally do. Would it be possible to set this up automatically, maybe as part of halcyon build in a project directory? Or would this interfere with your vision of how halcyon should be used?

Installing halcyon breaks pkg-config on my system

I let Halcyon set up its paths in my .bash_profile, but it seems to set PKG_CONFIG_SYSROOT_DIR (why?) so now I get stuff like:

$ pkg-config --cflags libpng12
-I/app/sandbox/usr/include/libpng12

which is wrong, since it was installed to /usr by my package manager. Is this really intentional? Did I misconfigure something?

Ubuntu 14.04, fresh install.

Improve preparing source directory

When installing an app with HALCYON_SANDBOX_EXTRA_APPS, but with no HALCYON_SANDBOX_EXTRA_APPS_CONSTRAINTS, the implicit constraints used for the extra apps should be included in the prepared source directory, and should influence the source hash.

Same applies to HALCYON_EXTRA_APPS with no HALCYON_EXTRA_APPS_CONSTRAINTS.

Already done for HALCYON_CONSTRAINTS.

Tmp directory management ?

Can I safely delete halcyon files in the tmp directory ?

We have a rather strict polity on the allocated space of /tmp so I cannot let it grow too much.

Cheers

Support Arch Linux fully

The setup script already supports Arch Linux. Everything is expected to work, except installing native OS packages declared with HALCYON_SANDBOX_EXTRA_OS_PACKAGES or HALCYON_EXTRA_OS_PACKAGES.

Be aware pacman --sync --refresh may break things.

Depends on mietek/bashmenot#3.

Replace non-POSIX halcyon-paths command in documentation

Neither source nor <( ... ) is POSIX. The most common approach in UNIX seems to be eval:

eval "$(halcyon paths)"

For two examples, see ssh-agent(1) and tset(1), find for "eval".

(OpenBSD links:
http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-agent.1
http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/tset.1
http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/stat.1
)

Automatically determine build tools

Instead of requiring the user to declare Cabal build tools as HALCYON_SANDBOX_EXTRA_APPS, determine and install such packages automatically.

Most of the work towards automation is already done, but is too slow to use on every push:

$ time (
>     lib_constraints=$( cabal_list_planned_lib_constraints /tmp/haskell-on-heroku-examples/hello-yesod-platform ) &&
>     echo "${lib_constraints}" | cabal_get_all_package_files 1 /tmp/lib_package_files &&
>     echo "${lib_constraints}" | cabal_dump_all_package_build_tools /tmp/lib_package_files | sed 's/^/build-tools: /' &&
>     echo "${lib_constraints}" | cabal_dump_all_package_extra_libs /tmp/lib_package_files | sed 's/^/extra-libs: /'
> )
build-tools: hsc2hs
build-tools: alex
build-tools: happy
extra-libs: z

real  0m11.015s
user  0m15.550s
sys   0m2.530s

This could be sped up by cabal get being able to output just the package description. (haskell/cabal#1954 and haskell/cabalPR#1977).

Moved from mietek/haskell-on-heroku#34.

Missing dependencies when trying to install xmobar with configuration flag

I'm trying to get halcyon to install xmobar:

At first I did

halcyon install xmobar

which worked fine. But then I wanted to reinstall it with a configuration flag (with_xft). So I simply did

halcyon install --extra-configure-flags="--flags=with_xft" xmobar

and hoped for the best. However, this fails with:

Configuring xmobar-0.22.1...
   cabal: At least the following dependencies are missing:
   X11-xft >=0.2 && <0.4
*** ERROR: Failed to configure app

I am a bit lost regarding all the options: I tried combining various options like --extra-apps=X11-xft, --sandbox-extra-apps=X11-xft, --sandbox-extra-apps-constraints and --constraints (the latter two I am unsure how to use, but I just put in X11-xft-0.3.1). I tried adding it to the constraints file under .halcyon/xmobar-0.22.1, and I finally tried deleting the latter directory entirely. Still the same failure.

Ensure temporary cache directory is removed on failure

Consider removing other temporary directories on failure, as well.

Thanks to @jshholland for the report.

Install fails on CentOS 6 using a sudo user

I have managed to install halcyon using root.

I then had a go using another user (with sudo privilege). The setup step was a success but when I tried halcyon install I have got:

Restoring GHC directory
       Extracting halcyon-ghc-7.8.4.tar.gz... done, 699MB
touch: cannot touch `/var/tmp/halcyon-cache/halcyon-ghc-7.8.4.tar.gz': Permission denied

I even tried sudo /app/halcyon/halcyon install but then I have got another error:

-----> Self-updating bashmenot... done, a7e57d3
-----> Self-updating Halcyon... done, 8da0d8f
   *** ERROR: Unexpected GHC user package database

       To disable the GHC user package database:
       $ mv ~/.ghc ~/.ghc.original

I am quite confused about how I am supposed to use halcyon. Does it need to be installed by root ? Should I then manually change the permission on app to be able to run it using a non-privilege user ?

Thanks for your help.

Wierd error when trying an install with puppet

I am trying to install halcyon using puppet and I have quite a wicket issue ... It is puppet related so I am just submit it here just in case you have an idea ...

I have tried this using a root user or an unprivilege one but it makes not difference.

Everything works fine without puppet. The same command works perfectly if I use manually on a shell but when puppet try to install cabal I have got:

Debug: Exec[install-ghc](provider=posix): Executing '/bin/bash --login -c 'source <( /app/halcyon/halcyon paths ); halcyon install''
Debug: Executing '/bin/bash --login -c 'source <( /app/halcyon/halcyon paths ); halcyon install''
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns: -----> Installing bashmenot... done, e2f1a00
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns: -----> Installing GHC and Cabal
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        External storage:                         public
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        GHC version:                              7.8.4
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        Cabal version:                            1.20.0.3
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        Cabal repository:                         Hackage
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns: 
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns: -----> Restoring GHC directory
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        Downloading https://halcyon.global.ssl.fastly.net/linux-centos-6-x86_64/halcyon-ghc-7.8.4.tar.gz... done
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        Extracting halcyon-ghc-7.8.4.tar.gz... done, 699MB
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns: 
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns: -----> Locating Cabal directories
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        Listing https://halcyon.global.ssl.fastly.net/?prefix=linux-centos-6-x86_64/halcyon-cabal-1.20.0.3-hackage-... done
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns: -----> Restoring base Cabal directory
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        Downloading https://halcyon.global.ssl.fastly.net/linux-centos-6-x86_64/halcyon-cabal-1.20.0.3.tar.gz... done
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        Extracting halcyon-cabal-1.20.0.3.tar.gz... done, 13MB
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns: -----> Updating Cabal directory
Notice: /Stage[main]/Profile::Jenkins::Slave::Halcyon/Exec[install-ghc]/returns:        Updating Cabal package database... error

This is strange because I do force a bash shell (login or not it makes no difference) by asking puppet to do the following:

  exec { 'install-ghc':
    user    => $jenkins_user,
    command => "/bin/bash --login -c 'source <( /app/halcyon/halcyon paths ); halcyon install'",
    creates => "${halcyon_base}/ghc",
    logoutput => on_failure,
    require => Exec['clone-halcyon']
  }

Any idea ?

Use `cabal install --enable-tests --dependencies-only --dry-run` instead of `cabal freeze`?

Doing cabal install --enable-tests --dependencies-only --dry-run in a clean sandbox gives similar results to using cabal freeze. This would help resolve/avoid the following issues:

#1 – Unexpected constraints difference
- May be not relevant if not using cabal freeze
- I'm assuming cabal install --enable-tests --dry-run would match cabal sandbox hc-pkg list. I haven't confirmed though.
#36 – Pass flags to cabal freeze to ensure all targets are included
- Skips using cabal freeze altogether and grabs the testing dependencies correctly
#41 – Improve Stackage support
- Avoids issues with cabal freeze not respecting constraints
- Should work with Stackage with a cabal.config file located in the app directory
haskell/cabal#1908 – Self-constraint generated by 'freeze'
- Avoids requiring Cabal 1.22
  - Cabal 1.22 with ghc-mod is broken without GHC 7.10
  - GHC 7.10 is not fully supported by Stackage yet
- Added in response to @mietek's comment below

I grabbed the output of both on one of our internal projects, and the output can be seen here. Output from cabal freeze was altered using sed to make differences easier to notice.

Major differences:

cabal freeze includes packages included with GHC. e.g. The ones listed in /app/ghc/lib/ghc-7.8.4/package.conf.d
cabal freeze has a cleaner, more consistent output format
- cabal install --enable-tests --dry-run indicated when there a newer version
- e.g. blaze-builder-0.3.3.4 (latest: 0.4.0.1)

Support RHEL

From the install script I can see that fedora or centos. Is there some support for RHEL ?

Thx

setup.sh only works for Bash

I'd recommend either making it POSIX or clarifying in the documentation that you're expected to be in Bash when you run this.

Halcyon is too pessimistic about changes in .halcyon

My understanding is that touching anything in .halcyon (or changing equivalent variables/passing equivalent flags?) will trigger a full sandbox and app rebuild. I think this is too pessimistic.

Specifically, any files/options which (only) affect the dependency graph should not trigger this, since Halcyon will resolve dependencies and find or build correct sandboxes based on the outcome anyway (right?).

There is one edge case I can think of: If I'm replacing a Hackage package with a modified fork (for example) via sandbox-sources, but I don't bump the version (even though I should), Halcyon would not consider this a change and wouldn't rebuild with the modified sources. However, I think it should be up to me to pass the appropriate rebuild flag or bump the version in the fork.

It would be nice to be able to specify a folder/namespace in a bucket

... so it is possible to have users in an organization pool their private storage in a single bucket (instead of one per user)