hexpm / bob Goto Github PK

View Code? Open in Web Editor NEW

231.0 231.0 57.0 610 KB

The Builder

Elixir 66.83% Shell 16.60% Dockerfile 16.26% HTML 0.31%

bob's Introduction

Hexpm

API server and website for https://hex.pm.

Contributing

To contribute to Hexpm you need to properly set up your development environment.

Also see the client repository: hex. The client uses hexpm for integration tests, so hexpm needs to support all versions the client supports. CI ensures that tests are run on all supported versions.

Setup

Run mix setup to install dependencies, create and seed database etc
Run mix test
Run iex -S mix phx.server and visit http://localhost:4000/

After this succeeds you should be good to go!

See setup alias in mix.exs and sections below for more information or when you run into issues.

PostgreSQL Modules And Version

PostgreSQL version should be >= 9.4, as Hexpm uses the jsonb type, that is available from PostgreSQL 9.4 onward.

Hexpm requires the PostgreSQL modules pg_trgm and pgcrypto to be available.

This is located in the "postgresql-contrib" package, however the package name can vary depending on your operating system. If the module is not installed the ecto migrations will fail.

Database

By default, Hexpm connects to a localhost PostgreSQL database hexpm_dev using the username postgres with the password postgres.

Create the database and user 'postgres' if not already done:

docker-compose up -d db

Now you are fine to create the hexpm_dev database and run the ecto migrations:

mix do ecto.create, ecto.migrate

Sample Data

Using the following command you can seed your local Hexpm instance with some sample data:

mix run priv/repo/seeds.exs

Node Dependencies

For assets compilation we need to install Node dependencies:

cd assets && yarn install

If you don't have yarn installed, cd assets && npm install will work too.

Running Hexpm

Once the database is set up you can start Hexpm:

# with console
iex -S mix phx.server

# without console
mix phx.server

Hexpm will be available at http://localhost:4000/.

License

Copyright 2015 Six Colors AB

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

bob's People

Contributors

Stargazers

Watchers

bob's Issues

Latest erlang 25.0.3 builds not found on docker hub

Erlang 25.0.3 was released 4 days ago, but I can't find the builds for it on docker hub:

Is Bob on annual leave?

Add OTP builds

Bob builds elixir 1.12 with unsupported otp 21

Elixir 1.12 requires otp 22+, but bob builds for otp 21.

See: https://repo.hex.pm/builds/elixir/v1.12-otp-21.zip

Not all ubuntu images were rebuilt for 25.0.1

Looks like only jammy got rebuilt for 25.0.1

All others are still on 25.0.0 which has ssl issues.

Speed up travis builds by splitting docker stages

Docker only seems to pull the cache for the final stage when doing multi-stage builds.

Alpine images with OTP 19 can't install rebar

Heyo,
Thanks for the great work here! I really appreciate it.

I ran into an issue when trying to use the Erlang 19 tags of the alpine docker images; I can't seem to get passed mix local.rebar due to an SSL issue.

I have a small repo up with mix new here and the Github Actions setup to run a bunch CI tasks like running dialyzer, credo, mix format checks, and tests:

https://github.com/dbernheisel/hexbob-github-ci-sample
specifically: https://github.com/dbernheisel/hexbob-github-ci-sample/actions/runs/46820506

I have this setup for Alpine:

apk update
apk upgrade
apk add --no-cache make bash zlib openssl openssh git g++ wget curl ca-certificates

Here's the error I get:

Run mix local.rebar --force
** (MatchError) no match of right hand side value: {:error, {:ssl, {'no such file or directory', 'ssl.app'}}}
    (mix) lib/mix/utils.ex:432: Mix.Utils.read_httpc/1
    (mix) lib/mix/utils.ex:376: Mix.Utils.read_path/2
    (mix) lib/mix/local.ex:139: Mix.Local.read_path!/2
    (mix) lib/mix/local.ex:118: Mix.Local.find_matching_versions_from_signed_csv!/2
    (mix) lib/mix/tasks/local.rebar.ex:98: Mix.Tasks.Local.Rebar.install_from_s3/4
    (mix) lib/mix/tasks/local.rebar.ex:53: Mix.Tasks.Local.Rebar.run/1
    (mix) lib/mix/task.ex:300: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:58: Mix.CLI.run_task/2

##[error]Process completed with exit code 1.

No docker images for OTP 22.3.4.1

👋 For some reason, there are no docker images published for OTP 22.3.4.1. At the same time, this version seems to be present in the builds list for alpine for instance: https://repo.hex.pm/builds/otp/alpine-3.10/builds.txt

hexpm/elixir Ubuntu/Debian images don't support ICMP protocol while Alpine ones do

The test consists in performing the following commands:

$ docker run --rm -ti hexpm/elixir:1.13.2-erlang-24.2.1-ubuntu-focal-20211006 sh
# iex
Erlang/OTP 24 [erts-12.2.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1]

Interactive Elixir (1.13.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :socket.open(:inet, :raw, :icmp)
{:error, {:invalid, {:protocol, :icmp}}}

and repeating for each Debian/Ubuntu images available.

Then if you perform the same for Alpine Linux, it actually works:

$ docker run --rm -ti hexpm/elixir-arm64:1.13.2-erlang-24.2.1-alpine-3.15.0 sh
/ # iex
Erlang/OTP 24 [erts-12.2.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1]

Interactive Elixir (1.13.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> :socket.open(:inet, :raw, :icmp)
{:ok, {:"$socket", #Reference<0.4054872251.4063100932.145697>}}

The problem seems to be related to some missing package at build time. Reference seems to be here: https://github.com/erlang/otp/blob/aa7f201ac2405ef6bf25d3111934f89dc3e23b73/erts/emulator/nifs/common/socket_nif.c#L169. The definition HAVE_LINUX_ERRQUEUE_H seems to be missing during build time.

The current v1.5.0-otp-20.zip and v1.5.0-otp-19.zip point to 1.5.0-rc.2

v1.5.0-otp-20.zip shows 1.5.0-rc.2.

Provide otp build for each ubuntu version

I found we have only https://repo.hex.pm/builds/otp/ubuntu-14.04/builds.txt - but not for other ubuntu releases.

Is there any reason? I don't think erlang built on ubuntu 14.04 would work on newer ubuntu "out of box" since some libraries are bumped major versions by default.

Probably actions/setup-elixir#31 is related

OTP 25.0-rc2 for Elixir 1.13.4

Should we build OTP 25.0-rc2 for Elixir >= 1.13.4?

The release page it should work ;) https://github.com/elixir-lang/elixir/releases/tag/v1.13.4

Build docker images for Elixir release candidate builds?

Elixir v1.12.0-rc.1 (and v1.12.0-rc.0 ) have been release in the past few weeks. Would it make sense to publish these docker images. I would love an easy way to test if an upcoming release will break anything in our project, and the easiest way to do that would be to just run all of our ci tests with the new image. However since Bob currently doesn't publish docker images for Elixir release candidates, I can't easily test my project against upcoming elixir versions. I know that I can clone bob and then build the images locally, but thats quite a bit of work and then that doesn't even consider that the image is only local to my computer, so I'd either have to recreate my ci tests locally, or push my locally created image to my own docker registry. Either way it would really nice if Bob builds these release candidate versions.

I can understand not wanted to spam too many tags, but it seems like it does build Erlang release candidates. Perhaps pushing these release candidate builds to a new docker repo (say hexpm/elixir-edge, or something) would be a good compromise.

Docker: Some 1.10.4/23.0.4 ARM64 builds are missing

It looks like the following images have no ARM64 version on Docker Hub:

1.10.4-erlang-23.0.4-debian-jessie-20200607
1.10.4-erlang-23.0.4-ubuntu-focal-20200703
1.10.4-erlang-23.0.4-ubuntu-bionic-20200630

Add to hexpm-ops

Bad zipfile error

Hi there,

I'm not sure if this is an issue for bob or for https://github.com/asdf-vm/asdf-elixir. I was hoping someone might be willing to point me in the right direction so that I could possibly offer a PR.

When I run asdf install elixir 1.7.0, that works... but it looks like it's compiled against OTP 19:

Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Elixir 1.7.0 (compiled with Erlang/OTP 19)

According to this commit I'm thinking that may not be right?

I tried specifying the otp version like so:

✦  asdf install elixir 1.7.0-otp-21
/var/folders/7q/zx_b307j1hv9pxvmx37c1p8h0000gn/T//elixir-precompiled-1.7.0-otp-21.zip
** Resuming transfer from byte position 5126599
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
file #1:  bad zipfile offset (local header sig):  0

but I'm getting that bad zipfile error which led me to think that it may be an issue with bob?

Running asdf install elixir 1.6.5-otp-21 works great by the way. Curiously, asdf install elixir 1.6.6-otp-21 has the bad zipfile too.

I thought I'd try compiling from source using asdf install elixir ref:v1.7.0. That looks like it's working perfectly:

✦  elixir --version
Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Elixir 1.7.0 (compiled with Erlang/OTP 21)

All in all, I was thinking that it would be great to be able to specify the otp version and not get the bad zipfile and I was thinking that asdf install elixir 1.7.0 should potentially use otp-20.

Thanks so much!

hexpm/elixir:1.0.5-erlang-17.3.4-ubuntu-groovy-20201022.1 doesn't have crypto.app

I wanted to do some tests on Elixir v1.0.x and noticed this:

$ docker run --rm -it hexpm/elixir:1.0.5-erlang-17.3.4-ubuntu-groovy-20201022.1 bash
root@225bca8ec9e7:/# mix new foo
{"init terminating in do_boot",{{badmatch,{error,{crypto,{"no such file or directory","crypto.app"}}}},[{elixir,start_cli,0,[{file,"src/elixir.erl"},{line,78}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()

This image was first in the list at https://hub.docker.com/r/hexpm/elixir/tags?page=1&ordering=last_updated&name=1.0.5-erlang.

Fortunately I was able to do use an OTP-18 based image: hexpm/elixir:1.0.5-erlang-18.3.4.1.1-ubuntu-bionic-20200403 so it's not a blocker for me.

Alternate Docker Registries

It would be great if the images could be published via multiple Docker registries. Docker Hub is no longer very great considering the rate limits.

I would recommend:

GitHub Packages (i might be able to get a sponsoring for the storage costs)
Quay.io (free)
Google Container Registry (i might be able to get a sponsoring for the storage costs)

Provide Ubuntu 22.04 (jammy) builds

I have confirmed in a fresh lxc Ubuntu 22.04 (jammy) container that the apt dependencies listed in 20.04 dockerfile are also working for building Erlang OTP-24.3.3 (without jinterface and wx webview support I think).

I followed steps from build_otp_ubuntu.sh, and also successfully extracted the resulting archive with minimal install, and was able to compile Elixir and run iex

Versioned docs for Hex

this doesn't work yet for some reason: https://hexdocs.pm/hex/0.18.2/Mix.Tasks.Hex.html

Does JIT work in docker?

https://elixirforum.com/t/erlang-otp-24-0-released/39630/43?u=princemaple

Consider caching docker images we use for OTP and Elixir builds

Add support for debian bullseye docker image

Currently, I see elixir tag support only buster as a latest release on debian. But bullseye which's latest release of debian is missing.

Ref: https://hub.docker.com/_/debian.

Segfault on 25.x when building via qemu for arm64

Hello, since 25.0 got released, I am not able to get past this step on arm64 builds via qemu. Previous 24.x were working as expected.

I am not sure what could be the cause, probably unrelated to the effort made in this repo, but I felt it was useful to open an issue here in case somebody else also ran into this.

> [builder  4/12] RUN mix local.hex --force &&   mix local.rebar --force:
#14 1.761 Segmentation fault (core dumped)
------
Dockerfile:9
--------------------
   8 |     
   9 | >>> RUN mix local.hex --force && \
  10 | >>>   mix local.rebar --force
  11 |     
--------------------
error: failed to solve: process "/bin/sh -c mix local.hex --force &&   mix local.rebar --force" did not complete successfully: exit code: 139

Here is a minimal repo that reproduces the issue: https://github.com/razielgn/test-arm64-elixir-docker-build-on-qemu-crash/runs/7090312681?check_suite_focus=true

EDIT: this image works correctly when building natively on ARM.

Add rollbar error reporting on action failures

Create docs_config.js for Elixir, ExDoc, Hex etc

Ref: hexpm/hexdocs#9

Access denied for Elixir 1.6.4

When attempting to retrieve Elixir 1.6.4 (via asdf elixir plugin), the following error is returned:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>D402D51C8E04D764</RequestId>
<HostId>woNgufQ2OUZI1U0DzWVP6QdWaWm2qBjc+/nAyAMXlP99HhtIb+IjZZetIT57Z7Sr5ZVmAwJW/hs=</HostId>
</Error>

URL: https://repo.hex.pm/builds/elixir/v1.6.4.zip

Add support for 32-bit arm

Hello everyone,

I'm currently using the arm32v7/elixir image on a balena iot project using the older 32-bit raspberry pis.
As there does not seem to be any update soon over there it'd be great if bob could also build 32-bit arm images.

Keep up the great work!

Provide debian-slim images

The only slim images we currently have are alpine but alpine compilation of C code with musl can be problematic, slow, and limited in the context of the JIT, which requires libc to be installed.

By providing debian-slim images, we can provide an alternative to alpine.

checksum of existing file was changed

Related: #16

My Dockerfile fetches precompiled elixir from bob, and checks the checksum to verify it's expected file.

RUN set -xe \
    && apk add --no-cache curl unzip \
    && curl -fSL -o elixir-precompiled.zip https://repo.hex.pm/builds/elixir/v1.7.1-otp-21.zip \
    && echo "c0954712e8969efff79ccf47a0847f9e336abca786cc145ffa9030cc6ed6055d  elixir-precompiled.zip" | sha256sum -c - \
    && unzip -d /usr/local elixir-precompiled.zip \
    && rm elixir-precompiled.zip \
    && apk del curl unzip \
    && mix local.hex --force \
    && mix local.rebar --force

It works well.. until the sha256 checksum of the file is changed - now it's aa768d5e9ece02939f76be6a212c617500c1e84af23ba27fc1c10491ce7a2c5d

Is it on purpose? For example, will it be changed when new elixir is released?

Elixir 1.10.4 & OTP 23 not building

Hey,

I just spotted that the new 1.10.4 image for OTP 23 has not been pushed to Docker hub?

https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=1.10.4-erlang-22 - pushed
https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=1.10.4-erlang-23 - no image found

Thanks
Sam

Tag docker images without using the date to allow for automation

Thanks for this awesome project :)

I am trying to use the hexpm images to automate an Elixir Docker Stack but in my analysis of available tags it looks like that only the Alpine ones will be suitable to use in an automated way because they don't have a date on them.

This works:

$ docker pull hexpm/elixir:1.12.3-erlang-24.1.2-alpine-3.14.2                                                                                                                                                 
1.12.3-erlang-24.1.2-alpine-3.14.2: Pulling from hexpm/elixir
a0d0a0d46f8b: Already exists 
74e66af6dba9: Pull complete 
078c0fc5e943: Pull complete 
656529acb1e6: Pull complete 
Digest: sha256:21f80e0ef4fea14624df49e0904bdeb65448ef50ba4da9ae7d0c1a9cb393da83
Status: Downloaded newer image for hexpm/elixir:1.12.3-erlang-24.1.2-alpine-3.14.2
docker.io/hexpm/elixir:1.12.3-erlang-24.1.2-alpine-3.14.2

It fails for Debian default image without a date on the tag:

$ docker pull hexpm/elixir:1.12.0-erlang-24.1.2-debian-buster                                                                                                                                                 
Error response from daemon: manifest for hexpm/elixir:1.12.0-erlang-24.1.2-debian-buster not found: manifest unknown: manifest unknown

Same goes for Debian slim images:

$ docker pull hexpm/elixir:1.12.0-erlang-24.1.2-debian-buster-slim                                                                                                                                            
Error response from daemon: manifest for hexpm/elixir:1.12.0-erlang-24.1.2-debian-buster-slim not found: manifest unknown: manifest unknown

Would it be possible to release tags for Ubuntu and Debian without using the date on them?

For example, a docker pull for hexpm/elixir:1.12.0-erlang-24.1.2-debian-buster-20210902-slim would be also possible with docker pull hexpm/elixir:1.12.0-erlang-24.1.2-debian-buster-slim.

Add task timeout

Changing Docker Tags

It would be great if the images could be tagged using multiple tags. For example:

hexpm/elixir:1.10.2-erlang-22.2.8-alpine-3.11.3
hexpm/elixir:1.10.2-erlang-22.2-alpine
hexpm/elixir:1.10.2-erlang-22-alpine
hexpm/elixir:1.10.2-alpine
hexpm/elixir:1.10-alpine
etc.

This would allow to get automatic security updates / bugfixes.

An example how that could work can be found in the PR: https://github.com/c0b/docker-elixir/pull/127/files#diff-54a405a70b571bc6fec28b63fc91cdd1e7a47d3edae70251442be7bd4bf15ce5R1

Docs missing

Hi thanks for pushing towards building docs into the images.

However I'm still not getting the docs for some reason. I use this tag: 1.11.2-erlang-23.1-alpine-3.12.0

docker image : Reverse-searching (^R) a long command crashes iex

Docker Image

hexpm/elixir:1.12.2-erlang-24.0.5-alpine-3.13.5 (same issue with 3.14.0)

Description

In iex, when using the reverse-search (Ctrl+R) to get a long command (about 50 chars), the REPL crashes.

Reproducing steps

docker run -it hexpm/elixir:1.12.2-erlang-24.0.5-alpine-3.13.5 iex

iex(1)> "a short string"
"a short string"

*do Ctrl+R here, then press Enter: no problem*

iex(3)> "this is a very long string that will crash iex"
"this is a very long string that will crash iex"

*do Ctrl+R here, then press Enter: iex crashes with no error message*

This doesn't happen with the official image (elixir:1.12.2-alpine) but unfortunately I need to explicitly pick versions for erlang and alpine.

Note: if I run sh first, then iex, I get an "Illegal Instruction" error instead of a crash.

Include hex + rebar in Elixir Docker Images?

Hello there 👋🏼

I use Bob-built docker images for CI tasks across a number of projects, and it's a great resource you've provided for the community. Thanks for maintaining this.

Apologies if this has been discussed before (couldn't find any such discussion immediately via search): would you consider pre-installing hex and rebar in the Elixir docker images? Although there are many "nice-to-have" things that someone could put into these images for various use-cases, mix local.hex --force and mix local.rebar --force are two commands that are fairly universal in my experience.

Of course, there may be reasons why this isn't possible or desirable, which I'd be happy to learn about.

Thank you!

Docs missing

Hi,

Not sure if this was discussed before. The new 1.11 images don't seem to have erlang docs included. Maybe it's a good idea to consider?

My windows installation does have it included but awkwardly does not have colors support, although I'm using a terminal that's capable of showing colors, as can be seen in the above screenshot

hexpm/elixir:1.13.1-erlang-24.2-alpine-3.14.3 image is missing

#103 bumped the 3.14.x alpine version to 3.14.3 but this image doesn't seem to be built.

I can see 1.13.1-erlang-23.3.4.10-alpine-3.14.3 and 1.13.1-erlang-24.2-alpine-3.14.2 on dockerhub but not 1.13.1-erlang-24.2-alpine-3.14.3.

It seems there are no images for alpine 3.14.3 with erlang 24.x for any elixir version.

Support for Amazon Linux

Hi, we're running Elixir on AWS EC2 with Amazon Linux as the base OS. Would you consider adding support for this OS to Bob?

Deploy as release

1.8.2 not available?

When one tries to download 1.8.2 related files from BOB, one sees a 403 Forbidden HTTP response.

Perhaps similar to #7 and #23?

Publish hex docs on hexdocs

Ref hexpm/hex#625 (comment).

Yo dawg I heard you want to publish hex docs on hexdocs.

Have Erlang/OTP 17-19 work with Ubuntu 18.04

As per @wojtekmach's erlef/setup-beam#7 and @ericmj's erlef/setup-beam#7 (comment) Erlang/OTP should work (there's an SSL-related issue) with Ubuntu 18.04.

~~I'm not sure this would solve #70, but potentially.~~

Provide checksum

How can I get the expected checksum of zip files so that I can check it against the calculated checksum from downloaded files?

[Vulnerability] alpine-3.15 has gdbm 1.22, which has a high-risk vulnerability

Hey y'all,

gdbm 1.22 contains a high-risk heap overflow, which was fixed in this commit and subsequently built into the 1.23 version.
alpine-3.15 repo currently contains the 1.22 version, its version being bumped in the alpine-3.16 repo

This vulnerability can be used as RCE and DoS attack vectors.
I wanted to start a thread on this, do you plan to bump the alpine version? I can do the necessary changes if needed, I want to confirm this with you though.

Support pre-build package for macOS?

I've create a PR that supports to working on macOS. setup-elixir on linux uses pre-build package that is built by bob. It takes about 10 seconds. On the other hand, on macOS builds from source code takes about 8-9 minutes. Using pre-build package is 500x faster than build from source code.

I wonder that if it could use bob built packages for macOS, it reduces setup time as like linux. Would you have plan to support for macOS?

Alternate Docker Registry

It would be great if the images could be published via multiple Docker registries. Docker Hub is no longer very great considering the rate limits.

I would recommend:

GitHub Packages (i might be able to get a sponsoring for the storage costs)
Quay.io (free)
Google Container Registry (i might be able to get a sponsoring for the storage costs)

/builds/elixir/builds.txt should be ordered

Currently we use natural ordering:

$ curl https://repo.hex.pm/builds/elixir/builds.txt
(...)
v1.1.1-otp-18 139d2eac5998259379862ff6ed2a5287f7de4ea6 2020-03-25T22:13:44Z bdfa8f86fc0ca296c045b67d6c7d9a7b25f3dddcb2388c066dddfe03e231a0aa
v1.10 1145dc01680aab7094f8a6dbd38b65185e14adb4 2020-07-04T11:27:55Z 0ff70d25a8332cc8aeb8da75de28f9ff36f0b124c16320255db6213e7fadc2dc 32a996366fb543d65a526bb5b4be8e2b09c57f5b9013e3ff7af50bf6d628742f
v1.2 9c91d2cc91fa2ee2ceb88075e69e5eb18bbb1af8 2020-03-25T22:16:42Z 9bc461faad6f92f10f09e5d3b61b216dc4b28122cdae3a28b3bbfac8a693160f
(...)

we should use semver based ordering instead.

Consider installing Git on images

If a project has a dependency that is linked through git compiling will fail with:

** (Mix) Error fetching/updating Git repository: the "git" executable is not available in your PATH. Please install Git on this machine or pass --no-deps-check if you want to run a previously built application on a system without Git.

I would ask that we consider pre-installing git on the base images so in scenarios like this we would not require consumers of the images to install it themselves.

v1.7.4 missing

Sorry if I'm just too quick about this since the release is only 13 hours old, but I tried updating to Elixir 1.7.4 this morning with asdf and found that the binaries are not yet available.