Giter VIP home page Giter VIP logo

hydra's Introduction

Hydra

CI

Hydra is a Continuous Integration service for Nix based projects.

Installation And Setup

Note: The instructions provided below are intended to enable new users to get a simple, local installation up and running. They are by no means sufficient for running a production server, let alone a public instance.

Enabling The Service

Running Hydra is currently only supported on NixOS. The hydra module allows for an easy setup. The following configuration can be used for a simple setup that performs all builds on localhost (Please refer to the Options page for all available options):

{
  services.hydra = {
    enable = true;
    hydraURL = "http://localhost:3000";
    notificationSender = "hydra@localhost";
    buildMachinesFiles = [];
    useSubstitutes = true;
  };
}

Creating An Admin User

Once the Hydra service has been configured as above and activate you should already be able to access the UI interface at the specified URL. However some actions require an admin user which has to be created first:

$ su - hydra
$ hydra-create-user <USER> --full-name '<NAME>' \
    --email-address '<EMAIL>' --password-prompt --role admin

Afterwards you should be able to log by clicking on "Sign In" on the top right of the web interface using the credentials specified by hydra-create-user. Once you are logged in you can click "Admin -> Create Project" to configure your first project.

Creating A Simple Project And Jobset

In order to evaluate and build anything you need to create projects that contain jobsets. Hydra supports imperative and declarative projects and many different configurations. The steps below will guide you through the required steps to creating a minimal imperative project configuration.

Creating A Project

Log in as administrator, click "Admin" and select "Create project". Fill the form as follows:

  • Identifier: hello
  • Display name: hello
  • Description: hello project

Click "Create project".

Creating A Jobset

After creating a project you are forwarded to the project page. Click "Actions" and choose "Create jobset". Fill the form with the following values:

  • Identifier: hello
  • Nix expression: examples/hello.nix in hydra
  • Check interval: 60
  • Scheduling shares: 1

We have to add two inputs for this jobset. One for nixpkgs and one for hydra (which we are referencing in the Nix expression above):

  • Input name: nixpkgs

  • Type: Git checkout

  • Value: https://github.com/nixos/nixpkgs-channels nixos-20.03

  • Input name: hydra

  • Type: Git checkout

  • Value: https://github.com/nixos/hydra

Make sure State at the top of the page is set to "Enabled" and click on "Create jobset". This concludes the creation of a jobset that evaluates ./examples/hello.nix once a minute. Clicking "Evaluations" should list the first evaluation of the newly created jobset after a brief delay.

Building And Developing

Building Hydra

You can build Hydra via nix-build using the provided default.nix:

$ nix-build

Development Environment

You can use the provided shell.nix to get a working development environment:

$ nix-shell
$ autoreconfPhase
$ configurePhase # NOTE: not ./configure
$ make

Executing Hydra During Development

When working on new features or bug fixes you need to be able to run Hydra from your working copy. This can be done using foreman:

$ nix-shell
$ # hack hack
$ make
$ foreman start

Have a look at the Procfile if you want to see how the processes are being started. In order to avoid conflicts with services that might be running on your host, hydra and postgress are started on custom ports:

  • hydra-server: 63333 with the username "alice" and the password "foobar"
  • postgresql: 64444

Note that this is only ever meant as an ad-hoc way of executing Hydra during development. Please make use of the NixOS module for actually running Hydra in production.

Checking your patches

After making your changes, verify the test suite passes and perlcritic is still happy.

Start by following the steps in Development Environment.

Then, you can run the tests and the perlcritic linter together with:

$ nix-shell
$ make check

You can run a single test with:

$ nix-shell
$ yath test ./t/foo/bar.t

And you can run just perlcritic with:

$ nix-shell
$ make perlcritic

JSON API

You can also interface with Hydra through a JSON API. The API is defined in hydra-api.yaml and you can test and explore via the swagger editor

Additional Resources

License

Hydra is licensed under GPL-3.0

Icons provided free by EmojiOne.

hydra's People

Contributors

ajs124 avatar aminechikhaoui avatar andir avatar aszlig avatar basvandijk avatar civodul avatar cole-h avatar dasj avatar dependabot[bot] avatar domenkozar avatar edolstra avatar ericson2314 avatar firestack avatar ghudgins avatar gilligan avatar grahamc avatar groxxda avatar knl avatar kquick avatar lheckemann avatar ma27 avatar mic92 avatar mornfall avatar nlewo avatar pingiun avatar rbvermaa avatar samueldr avatar shlevy avatar thufschmitt avatar twhitehead avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hydra's Issues

option to delete a jobset

On the 'edit project' screen it is possible to delete the project, but there is no delete button on the 'edit jobset' screen.

Unavailable systems

Hi,

Since 2012-10-23, our builds on i686-cygwin have failed, with the following output:

ssh_exchange_identification: Connection closed by remote host
unable to open SSH connection to nix@garrison, trying other available machines...
building path(s) /nix/store/w3pcg6hb771pk91vbciv95l02vb761s3-bison-v2.6.5-1021-g064e42b' error: ai686-cygwin' is required to build /nix/store/rg2dwwgqil88pig1wh4vi6sdd5w8b6qm-bison-v2.6.5-1021-g064e42b.drv', but I am ax86_64-linux'

Are you aware of this, and will it be fixed?

Thanks,

Speed up the /all page

The /all page does a count(*) on the Builds table, which taks O(n) time. So it's becoming slower and slower. We should store the number of (finished) builds in the database explicitly and keep it up to date using triggers.

Show log also when not executed in it's own job

currently it looks like that the logs are not showed on execution time for jobs that are executed in another job as dependency : i.e. for webdsl i can see the log of build but I can't see the log of BuildJava which is executed by JavaZip.

It would actually be even better that when JavaZip needs BuildJava, and BuildJava is also in the queue the progress is showed in BuildJava instead of JavaZip

Email notification improvement

E.g. when a build goes from dep. failed / cancelled to something else, it should only compare to previous status before the dep.failed / cancelled status.

Gracefully present "unavailable system" errors

It would be nice if, instead of "error: a i686-cygwin' is required to build ... but I am ax86_64-linux'", Hydra would display the error as "i686-cygwin systems currently unavailable", with a orange cross icon, say.

Send fewer email notifications

If a build fails, then we should should only send notification mail for that build and not all the builds that depend on it. The notification mail should however mention which dependent builds have failed as well.

Alternatively (or additionally) we could combine notifications, so if multiple builds fail in a certain time interval, we send 1 notification message linking to all the builds that failed.

In the build page, oshow nixpkgs revs since the change

It shows now what changed between the previous build and the last one, but it does not note when the store path changed. This usually ends up in a big list of nixpkgs changes, while it could be narrowed to a more specific change easily.

ikwildrpepper sees this related to eval views.

Random "locking protocol" error from SQLite

Example (from http://hydra.nixos.org/build/2536081):

these derivations will be built:
  /nix/store/57fmrf7af0nq9chfac42pbidkxwx5j6z-haskell-hjsmin-ghc7.4.1.20120412-0.1.1.drv
  /nix/store/adqgf30jjlx23gzb9h90rnwcxcbiizib-haskell-utf8-light-ghc7.4.1.20120412-0.4.0.1.drv
  /nix/store/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv
waiting for locks or build slots...
@ build-started /nix/store/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv /nix/store/h39jzxs6jrl7pz86vqd4ydsf7nxbz5js-haskell-language-javascript-ghc7.4.1.20120412-0.5.4 x86_64-linux /nix/var/log/nix/drvs/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv
building `/nix/store/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv' on `buildfarm@kyle'
@ build-remote /nix/store/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv buildfarm@kyle
copying 3 missing paths to ‘buildfarm@kyle’...
importing path `/nix/store/xvqnfrmzvq1dsi8qzi9a4ry4w9cj1xy8-haskell-blaze-builder-ghc7.4.1.20120412-0.3.1.0.drv'
importing path `/nix/store/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv'
querying path in database: locking protocol at /nix/store/wc37ic68y0868dgnaamahgw0d2mvd66j-nix-1.0pre2668_46cdc6a/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/Nix/CopyClosure.pm line 39.
builder for `/nix/store/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv' failed with exit code 11
@ hook-failed /nix/store/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv /nix/store/h39jzxs6jrl7pz86vqd4ydsf7nxbz5js-haskell-language-javascript-ghc7.4.1.20120412-0.5.4 2816 builder for `/nix/store/d32qq39s24lja9npyipyqbj6pjamw4mf-haskell-language-javascript-ghc7.4.1.20120412-0.5.4.drv' failed with exit code 11
cannot build derivation `/nix/store/57fmrf7af0nq9chfac42pbidkxwx5j6z-haskell-hjsmin-ghc7.4.1.20120412-0.1.1.drv': 1 dependencies couldn't be built
error: build of `/nix/store/57fmrf7af0nq9chfac42pbidkxwx5j6z-haskell-hjsmin-ghc7.4.1.20120412-0.1.1.drv' failed

Allow using a subdirectory of a repository as a job input

The Mercurial/Bazaar/Git repository types do not allow using a subdirectory (or single file) of the repository as a job input. This is useful if the repository contains other stuff besides the files we're interested in, and we don't want commits to the other stuff to trigger a rebuild. (With Subversion inputs this is possible because it allows checking out subdirectories.)

Static files are served with a 1970 timestamp

Because Hydra serves its static files from the Nix store, they have a 1970 timestamp:

$ curl --head http://hydra.nixos.org/static/css/hydra.css
HTTP/1.1 200 OK
Date: Fri, 13 Apr 2012 18:45:45 GMT
Content-Length: 214
Content-Type: text/css
Last-Modified: Thu, 01 Jan 1970 00:00:01 GMT
Vary: Accept-Encoding

This is bad because it prevents browsers from invalidating their caches if a static file changes.

Not clear what do to about this. Maybe we can hack Catalyst to send an ETag header equal to the store path hash of the static file.

Clean up the view mechanism

Alternatively, we could get rid of views altogether.

But if we keep them:

  • Views should be subsets of jobset evaluations.
  • Thus views should be tied to a jobset rather than a project.
  • We don't need a "primary" job anymore.
  • The number of SQL queries can be reduced a lot.

Send email to the person(s) who broke the build

Currently we send email to the maintainer of a build (if defined). Another option would be to send email to the person who committed the breaking change. If there were multiple commits, this would be multiple people. Getting the email address should be easy for Git/Mercurial repositories.

Add personalised status pages

It would be very nice if people could log in to Hydra and see the status of all jobs they're interested in. For instance, it could list all failing jobs of which the user is a maintainer.

problems when restoring or restarting old builds

I want to reconstruct this build product: http://hydra.nixos.org/build/2708329

Restart build:
This build cannot be restarted.

Clone build (no changes):
Caught exception in Hydra::Controller::Build->clone_submit "Can't use string ("1") as a HASH ref while "strict refs" in use at /nix/store/pmdx74y2jd9h07bj1cvpi0cvccrxai2i-hydra-0.1pre1084-aa8bf56/libexec/hydra/lib/Hydra/Helper/AddBuilds.pm line 718."

Hydra 0.1pre1084-aa8bf56 (using nix-1.1)

setting journal mode: attempt to write a readonly database in Build.pm

I am trying to run hydra on a nix installation with chroot enabled.

I run the nix-worker in daemon mode, as root.

I run hydra-queue-runner, hydra-evaluator as hydra.

When I inspect the build queue in the web UI, I notice that the first job in the queue fails with:

Caught exception in Hydra::Controller::Build->view_build "setting journal mode: attempt to write a readonly database at /nix/store/z59x87dkg5llg15qg023ky9flp8mv3i8-hydra-0.1pre1086-396a356/libexec/hydra/lib/Hydra/Controller/Build.pm line 44."

When I strace, I see that the nix-worker opens /nix/var/nix/db/db.sqlite just fine

[pid 24079] open("/nix/var/nix/db/db.sqlite", O_RDWR) = 6

The hydra-queue-runner never attempts to open this file.

The file is owned by root, with world having read access.

Store both the submission and the termination date of builds

Currently the timestamp' slot ofBuilds' stores either the time a build was queued, or the time at which it finished.

What about storing both the time at which it was queued, the time at which it was scheduled, and the time at which it finished building?

This would allow Hydra admins to "profile" their build farm.

package hydra with its configuration used for nixos

For me it is currently not possible to debug any hydra issues as I can't reproduce them.

It would be great to easily locally reproduce the hydra setup we are using for nixos. If multiple hosts are involved, that could for debugging purposes be handles by virtual machines.

Once we have the full complex setup, we could also think about simplifications for local debugging of specific corners as well as setups for hydra slaves.

What prevents hydra from being packaged for nix? I want to help making it happen.

Mixed Nix versions from host and hydra can cause forced upgrades to the Nix database.

I am running Hydra on "Ubuntu 10.04.4 LTS" (lucid). The host has Nix 1.1-1 installed from the 1.1 release page, downloaded as a .deb for 10.04.

There is a mismatch between the libsqlite3 library used by the host Nix and the Nix inside the Hydra closure.

The "host" Nix links dynamically against libsqlite3.so version 3.6.22-1, as provided by Ubuntu 10.04.

The Nix inside the Hydra closure links against libsqlite3.so version 3.7.13.

After 3.7.0, sqlite3 supports Write-Ahead Logging, and this is incompatible with older versions of sqlite3.

AFAICT, installing and running Hydra will upgrade the sqlite3 datastore, rendering the host Nix installation unusable.

The simple workaround, just installing the Nix from Hydra's closure into the user-specific Nix profile works just fine, of course, but it's a bit surprising that this is necessary. I didn't see any prominently displayed warning about this breaking upgrade when I started the Hydra services.

Hydra is leaking file descriptors

After running Hydra without interruption for a few weeks it stopped working with

error evaluating job set <project>: Too many open files: pipe() at /nix/store/858s9rnhp0ip3a1clij013xnc5r081vx-hydra-0.1pre1087-c57242c/libexec/hydra/lib/Hydra/Helper/AddBuilds.pm line 698
    ...propagated at /nix/store/858s9rnhp0ip3a1clij013xnc5r081vx-hydra-0.1pre1087-c57242c/libexec/hydra/lib/Hydra/Helper/AddBuilds.pm line 703.

Prevent Hydra from building packages based on license

Hydra (at least the TUD instance) shouldn't build unfree, non-redistributable packages, since that will cause them to end up in public channels. (Alternatively, Hydra should not make them available in channels.) What makes this tricky is that Hydra doesn't see the meta.license attributes of dependencies, so it can't determine whether a package has an unfree dependency.

One hacky way would be to add an assertion to stdenv.mkDerivation that, conditionally on an environment variable, checks meta.license to throw an exception if the license is unfree.

All build input information is lost when validation fails

When I create a new jobset with an invalid input name, then hit Create, I get to an error page. When I hit 'back', the form is emptied and all my build inputs are lost.

This can be reproduced by creating a new job set, then adding an input with a "name" starting with a number, e.g. 3rdparty, that points to an arbitrary local path, then hitting the Create button, then hitting the browser's back button.

I would expect the validation to happen as I type, so that I didn't lose my inputs when I hit Create.

view contents of nested jar file in output

when debugging eclipse plugin builds on hydra, the build result often contains nested jar files with relevant information, it would save a lot of hassle if I could recursively look at contents of jar files in the web interface

Duplicate build products

Example: http://hydra.nixos.org/build/2407133

To see the duplicate entries:

select build, productnr, count(*) from buildproducts 
  group by build, productnr having count(*) > 1;

Note that hydra.sql says that (build, productnr) is the primary key of the BuildProducts table, but the TUD Hydra instance seems to lack this.

"Operation not permitted" errors

Example (http://hydra.nixos.org/build/2488954):

building path(s) `/nix/store/3ap0adrzz2xz5b6my1pfwrry7j08ar9r-nixos-test-driver'
error: linking `/nix/store/2z2ff51yskqm44kddd10hv4xg2mp40lp-nixos-test-driver.drv.chroot/nix/store/c1n81qbnpp98npc6idkkk4a4z8jnzw1s-upstart-udevtrigger.conf' to `/nix/store/c1n81qbnpp98npc6idkkk4a4z8jnzw1s-upstart-udevtrigger.conf': Operation not permitted

Might be a Nix bug.

Implement a better scheduler

The current Hydra scheduler is very bad because it suffers from starvation issues. For instance, the gnu:emacs-trunk:build job has priority 100, while Nixpkgs jobs have a much lower priority. This is a real problem for Cygwin builds, because new gnu:emacs-trunk:build builds tend to come in faster than they can finish building. Thus no Nixpkgs Cygwin build is ever done. This prevents the Nixpkgs channel from being updated.

A naive scheduler that just executes all builds in order of ID would be fairer.

Add a button to cancel a build in progress

The only way right now to cancel a running build is to log in to the server, figure out the right pid, and kill it. Would be nice to have a way to do that from the web interface.

Print timestamp for Evaluation Errors

Please add the timestamp of the evaluation error in the evaluationErrors tab. Sometimes I think I'm getting stale evaluation errors, but it is hard to say for sure without a timestamp.

Thanks.

"unexpected end-of-file" in remote builds

Example Hydra log:

these derivations will be built:
  /nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv
@ build-started /nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv /nix/store/gp8h1d5pyy90vmyc5iid6m3fk1v0jrba-libkexiv2-4.7.4 i686-linux /nix/var/log/nix/drvs/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv
building `/nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv' on `buildfarm@kyle'
@ build-remote /nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv buildfarm@kyle
copying 3 missing paths to ‘buildfarm@kyle’...
waiting for the big garbage collector lock...
importing path `/nix/store/1br0h9i28lzr9kpgzdk0nkl45rs50r4q-libkexiv2-4.7.4.tar.bz2'
importing path `/nix/store/mk82bpir8na9jzs37a0b3cicxrv9jwx0-libkexiv2-4.7.4.tar.bz2.drv'
importing path `/nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv'
error: unexpected end-of-file
copying store paths to remote machine `buildfarm@kyle' failed: 256 at /nix/store/wc37ic68y0868dgnaamahgw0d2mvd66j-nix-1.0pre2668_46cdc6a/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/Nix/CopyClosure.pm line 40.
builder for `/nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv' failed with exit code 1
@ hook-failed /nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv /nix/store/gp8h1d5pyy90vmyc5iid6m3fk1v0jrba-libkexiv2-4.7.4 256 builder for `/nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv' failed with exit code 1
error: build of `/nix/store/jipyz52w50lsxyx56h729vbs16wqc20s-libkexiv2-4.7.4.drv' failed

This may be caused by a timeout triggered by the "waiting for the big garbage collector lock".

Allowing the definition of a minimum version control poll delay

Would be nice if a delay between version control polls ("svn up" and such) could be set, so as to reduce bandwidth consumption and avoid DoS-looking behavior (a couple of svn admins have already reported being annoyed.)

Ideally, one would say, per-project or per-jobset, "poll every 30 seconds, at most." The evaluator would thus need an actual scheduler for all the polls.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.