Giter VIP home page Giter VIP logo

whalebrew's Introduction

Whalebrew

Whalebrew creates aliases for Docker images so you can run them as if they were native commands. It's like Homebrew, but with Docker images.

Docker works well for packaging up development environments, but there are lots of tools that aren't tied to a particular project: awscli for managing your AWS account, ffmpeg for converting video, wget for downloading files, and so on. Whalebrew makes those things work with Docker, too.

$ whalebrew install whalebrew/whalesay
Unable to find image 'whalebrew/whalesay' locally
Using default tag: latest
latest: Pulling from whalebrew/whalesay
c60055a51d74: Pull complete
755da0cdb7d2: Pull complete
969d017f67e6: Pull complete
Digest: sha256:5f3a2782b400b2b23774709e0685d65b4493c6cbdb62fff6bbbd2a6bd393845b
Status: Downloaded newer image for whalebrew/whalesay:latest
🐳  Installed whalebrew/whalesay to /usr/local/bin/whalesay
$ whalesay cool
 ______
< cool >
 ------
   \
    \
     \
                   ##        .
             ## ## ##       ==
          ## ## ## ##      ===
      /""""""""""""""""___/ ===
 ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
      \______ o          __/
       \    \        __/
         \____\______/

Whalebrew can run almost any CLI tool, but it isn't for everything (e.g. where commands must start instantly). It works particularly well for:

  • Complex dependencies. For example, a Python app that requires C libraries, specific package versions, and other CLI tools that you don't want to clutter up your machine with.
  • Cross-platform portability. Package managers tend to be very closely tied to the system they are running on. Whalebrew packages work on any modern version of macOS, Linux, and Windows (coming soon).

Install

First, install Docker. The easiest way to do this on macOS is by installing Docker for Mac. You can install Docker for Mac via Homebrew:

brew install --cask docker

Next, you can install whalebrew via Homebrew on macOS and Linux:

brew install whalebrew

If you're not using Homebrew, you can download a binary and use that:

curl -L "https://github.com/whalebrew/whalebrew/releases/download/0.4.1/whalebrew-$(uname -s)-$(uname -m)" -o /usr/local/bin/whalebrew; chmod +x /usr/local/bin/whalebrew

Windows support is theoretically possible, but not implemented yet.

Usage

Install packages

$ whalebrew install whalebrew/wget

This will install the image whalebrew/wget as /usr/local/bin/wget.

The images in the whalebrew organization are a set of images that are known to work well with Whalebrew. You can also install any other images on Docker Hub too, but they may not work well:

$ whalebrew install bfirsh/ffmpeg

Find packages

$ whalebrew search
whalebrew/ack
whalebrew/awscli
whalebrew/docker-cloud
whalebrew/ffmpeg
whalebrew/gnupg
...

$ whalebrew search wget
whalebrew/wget

List installed packages

$ whalebrew list
COMMAND     IMAGE
ffmpeg      bfirsh/ffmpeg
wget        whalebrew/wget
whalebrew   whalebrew/whalebrew
whalesay    whalebrew/whalesay

Uninstall packages

$ whalebrew uninstall wget

Upgrade packages

To upgrade a single package, just pull its image:

$ docker pull whalebrew/wget

Configuration

Whalebrew reads configuration from either configuration files or environment variables.

The configuration file location can be specified using the WHALEBREW_CONFIG_DIR environment variable and defaults to ~/.whalebrew. The configuration file must be named config.yaml.

Description Default (if not specified anywhere) Format in config files Format in environment variables
The folder containing config.yaml ~/.whalebrew N/A WHALEBREW_CONFIG_DIR=$HOME/my-config
The directory to install packages in. /usr/local/bin install_path: $HOME/.whalebrew/bin WHALEBREW_INSTALL_PATH=$HOME/.whalebrew/bin

On a general basis, any configuration configured through environment variable will be prioritary compared to values from config files.

For example, if you have a whalebrew config of install_path: $HOME/.whalebrew/bin and an environment variable of WHALEBREW_INSTALL_PATH=$HOME/.local/bin, all whalebrew programs will be installed in $HOME/.local/bin.

Configuration path lookup

Environment variables have precedence on any other value. As soon as the WHALEBREW_CONFIG_DIR it defines the whalebrew installation directory. When not set, whalebrew considers the first existing file between ~/.whalebrew/config.yaml, $XDG_CONFIG_HOME/whalebrew/config.yaml, and for each path in $XDG_DATA_DIRS, whether $path/whalebrew/config.yaml exists

Using custom registries

⚠️ This feature is currently under development. Any feedback or comments are welcome!

Whalebrew now supports using several registries when searching for packages.

Each repository will be searched sequentially and matches on whalebrew packages will be shown, one per line.

To enable this feature, ensure you have a configuration file configured as defined above.

You can make one by running:

mkdir -p ${WHALEBREW_CONFIG_DIR:-~/.whalebrew}
cat > ${WHALEBREW_CONFIG_DIR:-~/.whalebrew}/config.yaml <<EOF
registries:
- dockerHub:
    owner: whalebrew
- dockerHub:
    owner: my-org
EOF

⚠️ Note that if you provide a single docker hub owner, only this owner will be searched for registries, replacing the default whalebrew docker hub organisation.

How it works

Whalebrew is simple, and leans as much as possible on native Docker features:

  • Packages are installed as files in /usr/local/bin (or a directory that you configure) with a shebang to make them executable. The content of the file is YAML that describes the options to pass to docker run, similar to a Compose service. For example:

      #!/usr/bin/env whalebrew
      image: whalebrew/whalesay
    
  • When a package is executed, Whalebrew will run the specified image with Docker, mount the current working directory in /workdir, and pass through all of the arguments.

    To understand what it is doing, you can imagine it as a shell script that looks something like this:

    docker run -it -v "$(pwd)":/workdir -w /workdir $IMAGE "$@"
    

Creating packages

Packages are Docker images published on Docker Hub. The requirements to make them work are:

  • They must have the command to be run set as the entrypoint.
  • They must only work with files in /workdir.

That's it. So long as your image is set up to work that way, it'll work with Whalebrew.

Configuration

There are some labels you can use to configure how Whalebrew installs your image:

  • io.whalebrew.name: The name to give the command. Defaults to the name of the image.

  • io.whalebrew.config.environment: A list of environment variables to pass into the image from the current environment when the command is run. For example, putting this in your Dockerfile will pass through the values of TERM and FOOBAR_NAME in your shell when the command is run:

      LABEL io.whalebrew.config.environment '["TERM", "FOOBAR_NAME"]'
    
  • io.whalebrew.config.volumes: A list of volumes to mount when the command is run. For example, putting this in your image's Dockerfile will mount ~/.docker as /root/.docker in read-only mode:

      LABEL io.whalebrew.config.volumes '["~/.docker:/root/.docker:ro"]'
    
  • io.whalebrew.config.ports: A list of host port to container port mappings to create when the command is run. For example, putting this in your image's Dockerfile will map container port 8100 to host port 8000:

      LABEL io.whalebrew.config.ports '["8100:8000"]'
    
  • io.whalebrew.config.networks: A list of networks to connect the container to.

      LABEL io.whalebrew.config.networks '["host"]'
    
  • io.whalebrew.config.working_dir: The path the working directory should be bound to in the container. For example putting this in your image's Dockerfile will ensure the working directory is available in /working_directory in the container

      LABEL io.whalebrew.config.working_dir '/working_directory'
    
  • io.whalebrew.config.keep_container_user: Set this variable to true to keep the default container user. When set to true, whalebrew will not run the command as the current user using the docker -u flag

      LABEL io.whalebrew.config.keep_container_user 'true'
    
  • io.whalebrew.config.missing_volumes: The behaviour to handle missing files or volumes into the container.

      LABEL io.whalebrew.config.missing_volumes 'skip'
    

    Possible values are

    • error to raise an error when trying to mount a non existing volume this is the default behaviour
    • skip to prevent binding the volume
    • mount to mount the volume anyway. This will result in docker creating a host directory
  • io.whalebrew.required_version: Specifies the minimum whalebrew version that is required to run the package. Examples: <1.0.0, >0.1.0, >0.1.0 <1.0.0

  • io.whalebrew.config.volumes_from_args: A list of command line arguments of the program passed at runtime that must be considered and mounted as volumes:

      LABEL io.whalebrew.config.volumes_from_args '["-C", "--exec-path"]'
    

Using user environment variables

The labels io.whalebrew.config.working_dir, io.whalebrew.config.volumes and io.whalebrew.config.environment are expanded with user environment variables when the container is launched.

For example, if your image has this line in your Dockerfile:

    LABEL io.whalebrew.config.working_dir '$PWD'

At runtime, it will bind your working directory into the container at the same path and set it as the working directory.

Using hooks

In some cases, you might want to execute custom actions, like checking the integrity of the image or adding the whalebrew scripts to your whalebrew repository. To do so, whalebrew will call git-like hooks when handling installation/uninstallation of a package. Those hooks must be executable files located in ${WHALEBREW_CONFIG_DIR}/hooks.

Whalebrew supports the following hooks:

command & arguments description
pre-install ${DOCKER_IMAGE} ${EXECUTABLE_NAME} This hook is called before installing a package. If it fails, the whole installation process fails
post-install ${EXECUTABLE_NAME} This hook is called after a package is installed. If it fails, the installation process fails, but the package is not uninstalled
pre-uninstall ${EXECUTABLE_NAME} This hook is called before uninstalling a package. If it fails, the whole uninstallation process fails
post-uninstall ${EXECUTABLE_NAME} This hook is called after a package is uninstalled. If it fails, the uninstallation process fails, but the package is not uninstalled

Whalebrew images

We maintain a set of packages which are known to follow these requirements under the whalebrew organization on GitHub and Docker Hub. If you want to add a package to this, open a pull request against whalebrew-packages.

Thanks

whalebrew's People

Contributors

aidanmelen avatar bfirsh avatar dependabot[bot] avatar diegopomares avatar haya14busa avatar jacobbednarz avatar keijiyoshida avatar larkinscott avatar loyalsoldier avatar mapitman avatar marcosinger avatar nithin-bose avatar nphyatt avatar osalbahr avatar peterdemartini avatar snodnipper avatar soloman1124 avatar stigkj avatar superbrothers avatar terev avatar themalkolm avatar tjamet avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

whalebrew's Issues

unknown command "upgrade" for "whalebrew"

On OS X:

Installed via: curl -L "https://github.com/bfirsh/whalebrew/releases/download/0.0.1/whalebrew-$(uname -s)-$(uname -m)" -o /usr/local/bin/whalebrew; chmod +x /usr/local/bin/whalebrew

returns this error: unknown command "upgrade" for "whalebrew"

Configure image so homebrew runs without -it?

This is a clarification and a question. From the readme here, it looks like the following is basically what is run when a Docker image is installed through Whalebrew and run: docker run -it -v "$(pwd)":/workdir -w /workdir $IMAGE "$@", as in it will always include -it, correct?

I'm trying to work through an issue where the Docker image doesn't work properly when -t is included for pseudo-TTY. So I'm thinking that's why it also doesn't work when installing/running through Whalebrew. While I work through that issue, Is there a way to configure the Dockerfile so Whalebrew does not include -it?

Document how to add custom volumes, ports, etc

There are some applications when dockerized need extra params like env variables or volumes to run. It would be cool to have an easy way to change the exec command generated by whale brew.

I was thinking about maybe a config to specify the exec command to be used but then the current workflow is so simple that I would say its almost beautiful.

Another way might be to have more options in the generated YAML file. Why didnt you just generate the docker compose file?

Trying to start a discussion. Again thanks a lot for the wonderful project.

standardize volumes and labels effecting runtime behavior?

Hey it's super exciting to see whalebrew! I've been working on a similar concept that also uses labels to augment runtime and a /woirkdir concept, and think it would be cool to make my project compatible with whalebrew images. In a perfect world we (the community) could create, publish, and pull from eachothers images.

Admittedly dex hardly has a community outside my workplace ATM. It also suffers documentation clarity issues and the runtime is a work-in-progress(I kind of stalled on darwin/MacOS X11 and am worried about over/under architecting some labels) -- but I wanted to reach out nonetheless! :) Do you think sharing thought-space solving these issues (e.g. label names and behaviors, mount points, &c) worth pursuing?

whalebrew version: need to bump the version number

whalebrew version does not show correct version number in 0.2.2.

$ curl -s -L -o ./whalebrew https://github.com/whalebrew/whalebrew/releases/download/0.2.2/whalebrew-Darwin-x86_64
$ chmod +x whalebrew
$ ./whalebrew version
Whalebrew 0.1.0

Enhancement: sync_blob_args

Currently, whalebrew compatible images can only work with images that only alter one directory (/workdir). However, this makes packages such as the rsync merge I retracted impossible to utilize. I have an Idea on how to make this work.

Whalebrew Spec Proposal: Blob arg handling

In order to make packages that may work with more than 1 directory, for example rsync, implementing Dockerfiles must:

  • add the io.whalebrew.options.sync_blob_args
  • create a /syncdir in the image
  • Add the /syncdir as a volume in your Dockerfile

Implementing additional mounts during command executing:

  • the /usr/local/bin command will take any blobs given as args after the named command, and map the corresponding host volumes to /syncdir/hostvolume

Challenges

  • Would we need to add a way for package creators to specify a delimeter?
  • Would this satisfy images that require host device access?

go-wrapper: not found

go-wrapper has been deprecated and removed from the go image 10 and above.

$ ./script/run
Sending build context to Docker daemon  262.7kB
Step 1/7 : FROM golang:1.11
 ---> fb7a47d8605b
Step 2/7 : RUN mkdir -p /go/src/github.com/whalebrew/whalebrew
 ---> Using cache
 ---> 564f29685282
Step 3/7 : WORKDIR /go/src/github.com/whalebrew/whalebrew
 ---> Using cache
 ---> 07ffec6aee59
Step 4/7 : COPY . /go/src/github.com/whalebrew/whalebrew
 ---> Using cache
 ---> dc0b7b173e90
Step 5/7 : RUN go-wrapper download -t ./...
 ---> Running in 9cecac3ce1a0
/bin/sh: 1: go-wrapper: not found
The command '/bin/sh -c go-wrapper download -t ./...' returned a non-zero code: 127

Allow to expose multiple executable commands from a single docker image

Why

Google Cloud SDK, for example, contains multiple executable files -- bq, gcloud, and gsutil.
Without being able to expose each commands (meaning create files in /usr/local/bin), we have to create individual docker images for them, which is ridiculous.

What

So here's my naive suggestion:

# Specify which commands to expose:
LABEL io.whalebrew.config.commands '["bq", "gcloud", "gsutil"]'

# And to configure volumes and ports:
LABEL io.whalebrew.config.command.bq.volumes ...
LABEL io.whalebrew.config.command.bq.ports ...
LABEL io.whalebrew.config.command.gcloud.ports ...
...

But it may brings a sort of complexity, wanna hear ur opinion.

Allow installation of a specific version

Let me first say... I love this project. I had started to write exactly this (I called it Dockerbin). However, I went to go back to it and finish it this weekend, and found your project instead.

I'm going to scrap Dockerbin, but I'd like to open some issues and contribute to whalebrew to add some of the features described in the target spec (see dockerbin/README. The first of those is:

Version supported commands

One of the awesome bits of using docker images instead of binaries, is that you can have multiple versions of a command available at once! I propose we create the ability to install and switch between multiple tags for images:

  • whalebrew install [email protected]
  • Add a new 'use' command. Ex... whalebrew use [email protected]
  • Allow the whalebrew list [imagename] command to take an optional [imagename], which will result in a list of tags available to install as that command

Make Whalebrew a Docker image

It'd be neat if we could ship Whalebrew as a Docker image, and make the binary a shell script which runs docker run whalebrew/whalebrew ....

A number of barriers to this:

  • It is an extra level of indirection, making it slower and less reliable. In particular, running any command will require running at least two containers. Maybe there are some tricks here, e.g. Whalebrew ships in a container, except whalebrew run which is handled by a shell script on the host that is installed/updated by the container.
  • We need to pass arbitrary envvars, ports, and volumes to containers that are run that are defined by the package.

Permission denied when running the executable it installed

Hi.

Running on Ubuntu 18, I get permission denied errors, not sure why.
Here is what I do and what I tested.

$ sudo whalebrew install dannyben/rush
Installed dannyben/rush to /usr/local/bin/rush

$ ls -la /usr/local/bin/rush
-rwxr-xr-x 1 root root 64 Apr 21 18:29 /usr/local/bin/rush

$ cat /usr/local/bin/rush
#!/usr/bin/env whalebrew
image: dannyben/rush
working_dir: /app

$ rush
bash: /usr/local/bin/rush: Permission denied

# and to verify the image is ok
$ docker run --rm -it dannyben/rush --version
0.5.8

I found a couple of duplicate issues, but they were closed and from 2017.

Integration with homebrew/homebrew bundle?

First of all, thanks a lot for this wonderful project. Makes my life easier trying to run awscli without really install these cli on my host system.

I also heavily rely on homebrew's bundle to bootstrap my mac's apps. I have to reinstall apps many times on new machines.

Is it possible to package whalebrew as a cask and then tap that cask and install apps via whalebrew? This will make it easier for people to also use whalebrew with scripts like brew bundle.

Or is it already working with brew's bundle?

Ask permission from user to mount volumes and pass environment variables

The io.whalebrew.config.volumes and io.whalebrew.config.environment labels allow images to request volumes and envvars to be passed to the container at runtime. Since we allow installing arbitrary Docker images, this is potentially a vector for attack.

Arguably, this is similar to the risk of just running normal command-line commands, but because we are sandboxing things anyway, let's do what sandboxes do well and have granular permissions!

When a package with these labels is installed, Whalebrew should ask the user if it all looks okay before installing:

$ whalebrew install oscorp/kittens
Pulling oscorp/kittens...
...
This package needs additional access to your system. It wants to:
* Mount the directory /etc/passwd
* Read the environment variable AWS_ACCESS_KEY
* Read the environment variable AWS_SECRET
Is this okay? y/n (y)

file permissions

I noted that when I used the whalebrew/wget package as a non-root user (e.g. ubuntu user in the docker group) that the files I downloaded had the permissions set to root.

ubuntu@ip-172-37-43-241:~/$ wget https://www.google.com/favicon.ico
ubuntu@ip-172-37-43-241:~/$ ls -al
-rw-r--r-- 1 root root     5430 Dec  8 01:00 favicon.ico

I have amended the code to use the user's id and primary group id but I guess this could be something for further consideration.

ubuntu@ip-172-37-43-241:~/$ wget https://www.google.com/favicon.ico
ubuntu@ip-172-37-43-241:~/$ ls -al
-rw-r--r-- 1 ubuntu ubuntu     5430 Dec  8 01:00 favicon.ico

I can certainly raise a PR but the thoughts of others might be useful first.

snodnipper@6b06198
@jskeates

whalebrew doesn't accept stdin

I would like to be able to script uninstalling packages with whalebrew. I have tried this:

~ ❯ whalebrew install whalebrew/terraform
Unable to find image 'whalebrew/terraform' locally
Using default tag: latest
latest: Pulling from whalebrew/terraform
c9b1b535fdd9: Already exists
2e6efbb09be4: Pull complete
d5eeca0eed63: Pull complete
6c9147d76a12: Pull complete
5241a1b707ee: Pull complete
b387ed038821: Pull complete
Digest: sha256:75f2d0e6f0c38cc05e7345361164fa7713ff1bb6b5d62cf32d2e777361f78dea
Status: Downloaded newer image for whalebrew/terraform:latest
docker.io/whalebrew/terraform:latest
🐳  Installed whalebrew/terraform to /usr/local/bin/terraform
~ ❯ which terraform
/usr/local/bin/terraform
~ ❯ terraform
Terraform v0.12.23
~ ❯ whalebrew uninstall terraform
This will permanently delete '/usr/local/bin/terraform'. Are you sure? (y/n) [n]: ^C
~ ❯ echo "y"
y
~ ❯ echo "y" | whalebrew uninstall terraform
This will permanently delete '/usr/local/bin/terraform'. Are you sure? (y/n) [n]: %                  ~ ❯
~ ❯ whalebrew list
COMMAND         IMAGE
terraform       whalebrew/terraform

post use cases in the readme

I saw the repo and it looks interesting but I cant see a practical reason for its use.

Is it just to avoid dependency issues?

whalebrew search panics when called without arguments

On macOS (11.0.1), using whalebrew, version 0.2.5, running whalebrew search without any arguments cause a panic. I'd expect the help text for search to be shown instead.

$ whalebrew search
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/whalebrew/whalebrew/cmd.glob..func4(0x1a68600, 0x1aa25b8, 0x0, 0x0, 0x0, 0x0)
	/private/tmp/whalebrew-20201115-55548-14de8lv/cmd/search.go:30 +0x216
github.com/spf13/cobra.(*Command).execute(0x1a68600, 0x1aa25b8, 0x0, 0x0, 0x1a68600, 0x1aa25b8)
	/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x1a683a0, 0x168af85, 0x1, 0x1008200)
	/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:850 +0x30b
github.com/spf13/cobra.(*Command).Execute(...)
	/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
	/private/tmp/whalebrew-20201115-55548-14de8lv/main.go:15 +0x46

whalebrew search doesn't return any results

whalebrew search doesn't return any results, but I can whalebrew install if I know the container name.

bash-3.2$ whalebrew search
bash-3.2$ whalebrew search ansible
bash-3.2$ whalebrew install whalebrew/ansible
Unable to find image 'whalebrew/ansible' locally
Using default tag: latest
latest: Pulling from whalebrew/ansible
c9b1b535fdd9: Already exists
394a08e0f610: Pull complete
214a4266a527: Pull complete
21a86fecc2be: Pull complete
Digest: sha256:e38524b06074f0a666ea21c536340577cd6f057c7540d70f7ee7f7a737eade68
Status: Downloaded newer image for whalebrew/ansible:latest
docker.io/whalebrew/ansible:latest
🐳  Installed whalebrew/ansible to /usr/local/bin/ansible
bash-3.2$

I'm running 0.23 (installed with homebrew per the readme) and have this happening on both of my Macs, both running Catalina.

Consider label-schema.org?

Hi, awesome project, seems like this idea was finally ready for someone to pick up and run with!!

Would you consider adopting http://label-schema.org/rc1/ ? Not sure if you are aware of it, but I think it would make a lot of sense in this context

Add support for Windows

This project would be really useful on a Windows system. I have a branch in my fork where I'm attempting to make it work in Windows. I'll submit a pull request if I get it working!

Operation not permitted error.

I keep getting the following error even when I run the installation script with sudo access.

chmod: changing permissions of '/usr/local/bin/whalebrew': Operation not permitted

open /usr/local/bin/jamf: permission denied

I can install and run whalebrew commands with no problem. When I run whalebrew list, however, I get open /usr/local/bin/jamf: permission denied. I'm running macOS Sierra 10.12.2 (Darwin Kernel Version 16.3.0) and Docker for Mac 1.13.0.

Enhancement: Defaults for packages rather than building augmented images

The motivation behind this issue would be to eliminate the need to build docker images for repositories that require manual configuration for environment variables volumes, etc. These things likely don't change between patch, minor or even major versions. So i'm proposing a way to distribute a set of default settings based on public image repository name. And possibly a way to map an image's tag to a built tag in order to version defaults.

The whalebrew-packages repository could be transformed to a set of skeletons used as defaults that can be overridden by the actual image labels when installing the package. The skeletons would just serve as a repository for these defaults.

A skeleton for awscli might look like:

FROM scratch
LABEL io.whalebrew.name aws
LABEL io.whalebrew.config.environment '["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN", "AWS_DEFAULT_REGION", "AWS_DEFAULT_PROFILE", "AWS_CONFIG_FILE"]'
LABEL io.whalebrew.config.volumes '["~/.aws:/.aws"]'

On my system after building this image docker says it's size is 0B
Essentially using dockerhub (and possibly private repos if the image is from there) as a store for these settings.

Require at least a particular Whalebrew version

We haven't got a good upgrade strategy yet. A simple one would be to allow images to specify a minimum Whalebrew version that they are compatible with, and Whalebrew will refuse to install it if it is less than that.

whalebrew in whalebrew

If whalebrew enabled docker in docker (see #24), and also installed itself in all containers, then you could have your tools talk to each other!

Goal: my whalebrew packages can call each other.

Example: when you build with microsoft/dotnet you might want to call node or npm to do run frontend tasks.

Benefits:

  • Each tool is downloaded as needed (e.g. docker run downloads what it needs)
  • Using official images gives you faster updates and well-tested tools
  • Your environment is now composable, and updating or adding a tool automatically updates your whole environment
  • You can run complicated tools in any Linux distro (e.g. installing and running dotnet in Arch Linux is a nightmare)

Let me know what you think or if you think it's not possible!

Thanks!

Support for arbitrary paths

Problem

Currently, Whalebrew only mounts the working directory in /workdir. This causes unexpected behaviour if you reference files outside the working directory, because they won't exist inside the container.

For example:

$ wget -O ../google.html https://www.google.com

The user intended to save google.html one directory up, but that isn't mounted as a volume in the container.

Potential solution

Whalebrew could match arguments which look a bit like a path, then additionally mount those inside the container.

Concerns:

  • Security - is this a vector for tricking the user into doing stupid things?
  • Clobbering files inside the container - trying to overwrite /usr isn't going to work

Is WhaleBrew compatible with Docker Toolbox on the Apple Mac

Apple Mac Pro (early 2009) Firmware 4.1->5.1
macOS 10.13.1 High Sierra
Docker Toolbox version 17.10.0-ce, build f4ffd25

Hello, I have an ancient but powerful Apple Mac Pro which, apparently is too old to run Docker for Mac. It does run the earlier Docker Toolbox though. Is WhaleBrew compatible with Docker Toolbox?

Run package container with --init option

whalebrew run command does not use --init option of docker run command for running packages. Due to this, some packages need tiny init such as dumb-init, for example https://github.com/whalebrew/whalebrew-packages/blob/master/wget/Dockerfile#L5.

dev@penguin:~/src/github.com/whalebrew/whalebrew$ git dc
diff --git a/cmd/run.go b/cmd/run.go
index 7ac79c7..d128496 100644
--- a/cmd/run.go
+++ b/cmd/run.go
@@ -44,6 +44,7 @@ var runCommand = &cobra.Command{
                        "run",
                        "--interactive",
                        "--rm",
+                       "--init",
                        "--workdir", os.ExpandEnv(pkg.WorkingDir),
                        "-v", fmt.Sprintf("%s:%s", cwd, os.ExpandEnv(pkg.WorkingDir)),
                }

TBW, what version of docker CLI is supported in whalebrew? --init option of docker run command was added in docker 1.13. If whalebrew has support for less than 1.13, whalebrew needs to detect the version of docker CLI used for resolving this issue.

// I'm happy that this project will be active again. 😊

Enhancement: Automatic Alternate command detection | override system command

Sometimes people name programs the same. A good example of this is packer, which is a popular image building tool from Hashicorp, but also exists in RHEL as a core binary that manages parts of the keyring.

whalebrew already supports custom command name configuration via the io.whalebrew.name label, however it would be great if whalebrew checked for the existence of that command during the install, and had some sort of option to automatically switch to an alternate command name, or an option to override the system command on the user path.

Example Use Cases

Ruby

Ruby is installed on every mac, but you should basically never use it when doing any project that utilizes Ruby, gems, rails, etc...

  • There could be a io.whalebrew.options.override_system_binary label, or flag during install

packer

As described above, there is a conflict with packer, in that they're two totally different tools. Unlike the system conflict example for ruby, we can use any name for the packer command if the system already has packer on the system path.

  • There could be a io.whalebrew.alternate_name that I could set to something like packerd

Add ARM/ARM64 support

Goal: Run whalebrew on ARM/ARM64 architecture

Example: Run whalebrew on Raspberry Pi boards or WD NAS boxes

Benefits:

  • Provide alternative package management on embedded systems
  • Promote multiarch image development

Missing files creates directories

The default behaviour of Docker is to create directories of locations that do not exist.

For example, I created a git package example that initially mounted locations like so:

LABEL io.whalebrew.config.volumes '["~/.gitconfig:/root/.gitconfig", "~/.ssh:/root/.ssh"]'

Consequently, when ~/.gitconfig not exist then a directory named .gitconfig is created. It is also documented that inode changes produce errors such as “[d]evice or resource busy”.

Example error:

ubuntu@ip-172-32-43-16:~/whalebrew$ git config --global user.email “[email protected]"
error: could not write config file /root/.gitconfig: Resource busy

The above was resolved by using:

LABEL io.whalebrew.config.volumes '["~/:/root/"]'

Clearly, the above gives git access to the entire home directory.

The above leads me to question whether it would be useful for whalebrew to mount a surrogate parent directory (e.g. /tmp/random/) when files (e.g. .gitconfig) are encountered (copying files, if necessary, into the surrogate). This restricts access to the host filesystem and gives the opportunity for the command to create default file(s), which I’d guess whalebrew could copy back.

@jskeates

Version 0.2.4 binary wont run (Linux x86_64)

$ uname -s
Linux

$ uname -m
x86_64

$ curl -L "https://github.com/whalebrew/whalebrew/releases/download/0.2.4/whalebrew-$(uname -s)-$(uname -m)" -o whalebrew

$ ls -l |grep whale
-rwxrwxrwx 1 root root 16422642 Oct 16 14:06 whalebrew

$ ./whalebrew
-bash: ./whalebrew: No such file or directory

Version 0.2.3 works fine.

Start detached container

I would like to create a whalebrew package for mongod which starts MongoDB as a daemon. I think it would be awesome if I could just run mongodb and whalebrew would start my MongoDB in a detached Docker container. As far as I can tell, all we need for that is a Docker image called whalebrew/mongod from the following Dockerfile:

FROM mongo

EXPOSE 27017

CMD mongod

And a new config label for whalebrew called io.whalebrew.config.detach which would set the -d flag when running the container. What do you guys think?

unkown command xxx for whalebrew since update to 0.2.3

Since I updated from 0.1.0 to 0.2.3 I can't start any installed whalebrew packages.

Uninstalled everything and reinstalled the packages without success.

λ ~ .whalebrew/bin/wget
unknown command ".whalebrew/bin/wget" for "whalebrew"
λ ~ whalebrew version                                                                                                                                                        Whalebrew 0.2.3
λ ~ docker version
Client:
 Version:           19.03.1-ce
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        74b1e89e8a
 Built:             Fri Aug 16 14:00:42 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.1-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       74b1e89e8a
  Built:            Fri Aug 16 13:58:02 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.7.m
  GitCommit:        85f6aa58b8a3170aec9824568f7a31832878b603.m
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
λ ~ docker run whalebrew/wget
zsh: correct 'whalebrew/wget' to '.whalebrew/wget' [nyae]? n
wget: missing URL
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

Build scripts failing: script/build and script/run

I get this error when I try to build the source, or build the docker image:

../client/client.go:16: cli.UpdateClientVersion undefined (type *client.Client has no field or method UpdateClientVersion)
whalebrew $git reset --hard HEAD
HEAD is now at cf6c366 Fix titles in readme
whalebrew $
whalebrew $
whalebrew $
whalebrew $./script/build 
Number of parallel builds: 3

-->    darwin/amd64: github.com/bfirsh/whalebrew
-->     linux/amd64: github.com/bfirsh/whalebrew

2 errors occurred:
--> darwin/amd64 error: exit status 2
Stderr: # github.com/bfirsh/whalebrew/client
../client/client.go:16: cli.UpdateClientVersion undefined (type *client.Client has no field or method UpdateClientVersion)

--> linux/amd64 error: exit status 2
Stderr: # github.com/bfirsh/whalebrew/client
../client/client.go:16: cli.UpdateClientVersion undefined (type *client.Client has no field or method UpdateClientVersion)

Build image:

whalebrew $./script/run 
Sending build context to Docker daemon  274.9kB
Step 1/7 : FROM golang:1.7
 ---> 225948c9a8b1
Step 2/7 : RUN mkdir -p /go/src/github.com/bfirsh/whalebrew
 ---> Using cache
 ---> e177700e163b
Step 3/7 : WORKDIR /go/src/github.com/bfirsh/whalebrew
 ---> Using cache
 ---> b7b41bfeafa5
Step 4/7 : COPY . /go/src/github.com/bfirsh/whalebrew
 ---> b35091b04657
Removing intermediate container ba333628eedc
Step 5/7 : RUN go-wrapper download -t ./...
 ---> Running in 7577b4895939
+ exec go get -v -d -t ./...
github.com/Songmu/prompter (download)
github.com/mattn/go-isatty (download)
Fetching https://golang.org/x/crypto/ssh/terminal?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ssh/terminal?go-get=1 (status code 200)
get "golang.org/x/crypto/ssh/terminal": found meta tag main.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh/terminal?go-get=1
get "golang.org/x/crypto/ssh/terminal": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag main.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
github.com/docker/docker (download)
Fetching https://gopkg.in/yaml.v2?go-get=1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag main.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
gopkg.in/yaml.v2 (download)
github.com/spf13/cobra (download)
github.com/spf13/pflag (download)
github.com/spf13/viper (download)
github.com/fsnotify/fsnotify (download)
github.com/hashicorp/hcl (download)
github.com/magiconair/properties (download)
github.com/mitchellh/mapstructure (download)
github.com/pelletier/go-toml (download)
github.com/spf13/afero (download)
Fetching https://golang.org/x/text/transform?go-get=1
Parsing meta tags from https://golang.org/x/text/transform?go-get=1 (status code 200)
get "golang.org/x/text/transform": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/transform?go-get=1
get "golang.org/x/text/transform": verifying non-authoritative meta tag
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
golang.org/x/text (download)
Fetching https://golang.org/x/text/unicode/norm?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/norm?go-get=1 (status code 200)
get "golang.org/x/text/unicode/norm": found meta tag main.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/norm?go-get=1
get "golang.org/x/text/unicode/norm": verifying non-authoritative meta tag
github.com/spf13/cast (download)
github.com/spf13/jwalterweatherman (download)
github.com/stretchr/testify (download)
 ---> 9fecb8e44b62
Removing intermediate container 7577b4895939
Step 6/7 : RUN go-wrapper install
 ---> Running in dd92b5641aa4
+ exec go install -v
github.com/mattn/go-isatty
golang.org/x/sys/unix
github.com/docker/docker/vendor/github.com/opencontainers/go-digest
github.com/docker/docker/vendor/github.com/docker/distribution/digestset
github.com/docker/docker/vendor/github.com/docker/distribution/reference
github.com/docker/docker/api
github.com/docker/docker/api/types/blkiodev
github.com/docker/docker/api/types/mount
github.com/docker/docker/api/types/strslice
github.com/docker/docker/vendor/github.com/docker/go-connections/nat
github.com/docker/docker/vendor/github.com/docker/go-units
github.com/docker/docker/api/types/container
golang.org/x/crypto/ssh/terminal
github.com/docker/docker/api/types/versions
github.com/docker/docker/api/types/filters
github.com/Songmu/prompter
github.com/docker/docker/api/types/network
github.com/docker/docker/vendor/github.com/opencontainers/image-spec/specs-go
github.com/docker/docker/vendor/github.com/gogo/protobuf/proto
github.com/docker/docker/vendor/github.com/opencontainers/image-spec/specs-go/v1
github.com/docker/docker/api/types/registry
github.com/docker/docker/api/types/events
github.com/docker/docker/api/types/image
github.com/docker/docker/api/types/time
github.com/docker/docker/vendor/golang.org/x/net/proxy
github.com/docker/docker/vendor/github.com/docker/go-connections/sockets
github.com/docker/docker/vendor/github.com/pkg/errors
github.com/docker/docker/vendor/github.com/docker/go-connections/tlsconfig
github.com/docker/docker/vendor/golang.org/x/net/context
github.com/docker/docker/vendor/golang.org/x/net/context/ctxhttp
gopkg.in/yaml.v2
github.com/docker/docker/api/types/swarm/runtime
github.com/docker/docker/api/types/swarm
github.com/docker/docker/api/types
github.com/bfirsh/whalebrew/version
github.com/spf13/pflag
github.com/docker/docker/api/types/volume
github.com/docker/docker/client
github.com/spf13/cobra
github.com/fsnotify/fsnotify
github.com/hashicorp/hcl/hcl/strconv
github.com/hashicorp/hcl/hcl/token
github.com/bfirsh/whalebrew/client
github.com/bfirsh/whalebrew/packages
# github.com/bfirsh/whalebrew/client
client/client.go:16: cli.UpdateClientVersion undefined (type *client.Client has no field or method UpdateClientVersion)
github.com/hashicorp/hcl/hcl/ast
github.com/hashicorp/hcl/hcl/scanner
github.com/hashicorp/hcl/json/token
github.com/hashicorp/hcl/hcl/parser
github.com/hashicorp/hcl/json/scanner
github.com/hashicorp/hcl/json/parser
github.com/magiconair/properties
github.com/hashicorp/hcl
github.com/mitchellh/mapstructure
github.com/pelletier/go-toml
github.com/spf13/afero/mem
golang.org/x/text/transform
golang.org/x/text/unicode/norm
github.com/spf13/cast
github.com/spf13/jwalterweatherman
github.com/spf13/afero
github.com/spf13/viper
The command '/bin/sh -c go-wrapper install' returned a non-zero code: 2

I'm running

whalebrew $sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.12.6
BuildVersion:	16G29

If I comment out line 16, the build succeeds. Thoughts?

This passes the build:

whalebrew $git diff
diff --git a/client/client.go b/client/client.go
index 2b683dc..5bb6317 100644
--- a/client/client.go
+++ b/client/client.go
@@ -13,6 +13,6 @@ func NewClient() (*client.Client, error) {
        if err != nil {
                return cli, err
        }
-       cli.UpdateClientVersion(DefaultVersion)
+  //cli.UpdateClientVersion(DefaultVersion)
        return cli, nil
 }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.