nadirizr / dazel Goto Github PK

Somewhere along the line, bazel shells out to the patch binary when building:

Error in fail: Error applying patch @io_bazel_rules_go//third_party:go_googleapis-deletebuild.patch:
bash: patch: command not found

The fix was easy, just added this to my Dockerfile.dazel:

RUN apt update && apt install -y patch

It might be useful to people if this was pre-installed in the default dazel/dazel image.

Hello! Thanks for a neat tool. Currently giving it a spin on OS-X with Python2.7.
So dazel.py seems to depend on Python 3 per #!/usr/bin/env python3 and usage of some Python3 functionality i.e.shutil.get_terminal_size(). Is dazel supposed to work with python 2 at all? If no, should it be py2 compatible on Pypi?

If you put a Dockerfile.dazel at the root of your workspace, you can build from the workspace root and everything works nicely. If, however, you cd into a subdirectory and invoke dazel build, dazel will try to pull the default dazel/dazel image instead of using the Dockerfile.dazel.

This appears to be because the workspace path is not joined to the dockerfile variable (source).

It can be worked around by adding the following to .dazelrc at the workspace root:

import dazel
DAZEL_DOCKERFILE = os.path.join(
    dazel.DockerInstance._find_workspace_directory(),
    "Dockerfile.dazel")

But this is obviously fragile.

dazel also appears to create a .dazel_run file in the subdirectory in this case, but this just results in an unnecessary file and and a one-time unnecessary container restart.

I'm thinking it doesn't since I'm seeing:
2017/04/27 14:58:27 exec: "git": executable file not found in $PATH

hi, developers,

I am new to this tool. I can set up dazel run bu parsing the local .dazelrc. But if I can the directory for launching the dazel run. How to designate the .dazelrc at command line?

Thanks!

Wei

ERROR: /home/nathan/.cache/bazel/_bazel_nathan/9a314670874df4bb3656ae2f987d35fc/action_cache (Device or resource busy)

I believe this fails because action_cache is a volume mount by default in dazel. Is this a known issue with a proper workaround?

Trying to use bazel on OSX yeilds this

Any ideas?

(env) gregbowyer@USSFRGBO-ML ~/work/persephone $ dazel build '//persephone:tmvax'
Loading:
Loading: 0 packages loaded
ERROR: error loading package '': Encountered error while reading extension file 'docker/docker.bzl': no such package '@io_bazel_rules_docker//docker': Error cloning repository: /Users/gregbowyer/.cache/bazel/_bazel_gregbowyer/external/io_bazel_rules_docker/.git/objects/incoming_3769060350482030842.pack (Operation not permitted) caused by /Users/gregbowyer/.cache/bazel/_bazel_gregbowyer/external/io_bazel_rules_docker/.git/objects/incoming_3769060350482030842.pack (Operation not permitted) caused by /Users/gregbowyer/.cache/bazel/_bazel_gregbowyer/external/io_bazel_rules_docker/.git/objects/incoming_3769060350482030842.pack (Operation not permitted)
ERROR: error loading package '': Encountered error while reading extension file 'docker/docker.bzl': no such package '@io_bazel_rules_docker//docker': Error cloning repository: /Users/gregbowyer/.cache/bazel/_bazel_gregbowyer/external/io_bazel_rules_docker/.git/objects/incoming_3769060350482030842.pack (Operation not permitted) caused by /Users/gregbowyer/.cache/bazel/_bazel_gregbowyer/external/io_bazel_rules_docker/.git/objects/incoming_3769060350482030842.pack (Operation not permitted) caused by /Users/gregbowyer/.cache/bazel/_bazel_gregbowyer/external/io_bazel_rules_docker/.git/objects/incoming_3769060350482030842.pack (Operation not permitted)
INFO: Elapsed time: 0.587s
FAILED: Build did NOT complete successfully (0 packages loaded)
(env) gregbowyer@USSFRGBO-ML ~/work/persephone $

I'm trying build a rules_docker python image, and am seeing the following error:

$> dazel build reporting/ad_hoc:cronjob
dazel build reporting/ad_hoc:cronjob
ERROR: error loading package '': Encountered error while reading extension file 'requirements.bzl': no such package '@py_deps//': pip_import failed:  (java.io.IOException: Cannot run program "python" (in directory "/Users/mtsgrd/.cache/bazel/_bazel_mtsgrd/external/py_deps"): error=2, No such file or directory)

I have the following BUILD file:

pip_deps = [
        requirement("urllib3"),
        ...
]

py_image(
    name = "cronjob",
    srcs = glob(["*.py"]),
    deps = pip_deps,
    main = "report_cli.py"
)

Does this seem like an issue that can be solved by including python in the Dazel container?

Currently, to use Dazel, I need to

1. Install Docker client/server
2. Install Python 3 (there are Python 2 versions of Dazel, but they don't work #28)
3. Install Pip
4. Install Dazel

Being the entry point to a controlled, consistent environment, it would be nice if Dazel had minimal dependencies.

Have you considered implementing it in Go, where it would be easy to make a per-platform statically compiled binary (faster startup too)? It would remove the dependency on Python 3 and pip. It could even remote the requirement for the docker client by using the Go APIs (https://github.com/moby/moby/tree/master/client or https://github.com/docker/go-docker), though this is probably unnecessary.

Hi,

I think it will be useful to have a feature in which any file from the host can be copied to the docker container. I was thinking of adding a variable called LOCAL_FILES in the .dazelrc file. These can either be files or directories. When dazel is run these files will be copied to your workspace on the host and then you have the option to use ADD or COPY in the dockerfile to then copy these files into the container. After dazel is done running these files will be removed from the workspace. I am requesting this feature because I want copy my ssh and git credentials for a private repo onto the container.

Firstly thanks for building this! It makes setting people up to develop a bazel repo much easier.

The one problem I'm having is that we need to mount secrets from our machines into the dazel container (using DAZEL_VOLUMES) but since these secrets are located on different locations in each developers' machine (typically somewhere within their $HOME directory) we each need to change the value to this locally.

I'd like to have the ability to override values in the standard .dazelrc with custom values from a separate .dazelrc.local file which we wouldn't source control. I'll probably put in a PR for this but it seems like it should be pretty straightforward to implement.

In previous commits a support for receiving (workspace_hex, delegated_volume, user) in __init__ was added to DockerInstance, while the instantiation code inside _start_run_deps does not pass these variables.
This in turn causes dazel installation to fail.
The relevant code is inside dazel.py:

class DockerInstance:
    """Manages communication and runs commands on associated docker container.
    A DockerInstance can build the image for the container if necessary, run it,
    set it up through configuration variables, and pass on commands to it.
    It streams the output directly and blocks until the command finishes.
    """

    def __init__(self, instance_name, image_name, run_command, docker_command, dockerfile,
                       repository, directory, command, volumes, ports, network,
                       run_deps, docker_compose_file, docker_compose_command,
                       docker_compose_project_name, docker_compose_services, bazel_user_output_root,
                       bazel_rc_file, docker_run_privileged, docker_machine, dazel_run_file,
                       workspace_hex, delegated_volume, user):

and

def _start_run_deps(self):
        """Starts the containers that are marked as runtime dependencies."""
        for (run_dep_image, run_dep_name) in self.run_deps:
            run_dep_instance = DockerInstance(
                instance_name=run_dep_name,
                image_name=run_dep_image,
                run_command=None,
                docker_command=None,
                dockerfile=None,
                repository=None,
                directory=None,
                command=None,
                volumes=None,
                ports=None,
                network=self.network,
                run_deps=None,
                docker_compose_file=None,
                docker_compose_command=None,
                docker_compose_project_name=None,
                docker_compose_services=None,
                bazel_rc_file=None,
                bazel_user_output_root=None,
                docker_run_privileged=self.docker_run_privileged,
                docker_machine=self.docker_machine,
                dazel_run_file=None)

Any idea?

dazel info output_base
/home/ensonic/.cache/bazel/_bazel_ensonic
bazel info output_base
/home/ensonic/.cache/bazel/_bazel_ensonic/556f802681b2087c6d9a6bcc3d0d954b

If I work on multiple projects, each get their own cache dir when using 'bazel' directly.

Hello,
I am trying to use dazel on M1 mac. I installed the latest version and I encounter this error message when running dazel build and google tells me it is related to M1 architecture.

Opening zip "/proc/self/exe": lseek(): Bad file descriptor
FATAL: Failed to open '/proc/self/exe' as a zip file: (error: 9): Bad file descriptor

Is there a way to overcome this?

If I run dazel==0.0.33 I get the following error:

> virtualenv -p python3 env
> env/bin/python3 --version
Python 3.4.3
> env/bin/pip install dazel==0.0.33
Collecting dazel==0.0.33
  Using cached dazel-0.0.33-py2.py3-none-any.whl
Installing collected packages: dazel
Successfully installed dazel-0.0.33
> env/bin/dazel version    
Traceback (most recent call last):
  File "env/bin/dazel", line 7, in <module>
    from dazel import main
  File "/usr/local/google/home/rodrigoq/git/google/glog/env/lib/python3.4/site-packages/dazel.py", line 138
    *shutil.get_terminal_size(),
               ^
SyntaxError: can use starred expression only as assignment target

This appears to be a Python 3.4 compatibility issue: karlch/vimiv#7

This is a great tool! I was trying to get tensorflow to work on GPUs, and found I had to use nvidia-docker to get the device driver to load. Are you open to adding support for that? I'm thinking it just needs a new parameter (something like DAZEL_DOCKER_COMMAND) that the user can put in .dazelrc. I can send you a pull request if you are interested. Thanks!

For example the query for fetching the locations of all protocol buffer files you have to escape some characters.

Note the extra \ characters
Bazel
bazel query 'filter("^[^@].*\.proto$", deps(//target/...))'
Dazel
dazel query 'filter(\"^[^@].*\\.proto$\", deps(//target/...))'

Basically if you have a script which should work with both bazel and dazel you'd have to write two different queries.

Looking at the source code, I think it might be the result of the way arguments are being quoted when formulating the docker command

If you change the .dazelrc file to add addition volumes, or run dependencies, etc, dazel does not automatically restart the docker container with the new volumes/etc.

I believe the easiest way to resolve this is to just check the timestamp of .dazelrc and if newer than the dazel run file, restart the container. Much like how it is done if Dockerfile is newer.

Right now I get:

docker exec -it dazel printenv
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=e8a036c516a4
BAZEL_VERSION=0.6.1
HOME=/root

I'd like to have this in my checked in tools/bazel.rc:

build --experimental_repository_cache=$HOME/.cache/bazel/repositories

For that $HOME would need to be mapped into the container via -e
https://github.com/nadirizr/dazel/blob/master/dazel.py#L315

Does that make sense? Would you accept such a PR?

If DAZEL_DOCKERFILE specifies a Dockerfile that has relative COPY directives, dazel cannot create the image and the build will fail.

Here is example output:

$ dazel build '//...'
Sending build context to Docker daemon   51.2kB
Step 1/3 : FROM insready/bazel
 ---> 8f30918a7d5f
Step 2/3 : COPY "hello-world" /
COPY failed: stat /var/lib/docker/tmp/docker-builder954504696/hello-world: no such file or directory

This is generated using the following repo: https://github.com/redshirtrob/dazel-dockerfile

The problem is that dazel generates the PATH argument from the self.directory field. The solution is to derive the PATH argument from self.dockerfile field and let self.directory continue to point to the workspace.

Hello,

since I switched my base-image to a non-root user, the bazel installation inside the container exits with:
FATAL: Output base directory '/home/jan/.cache/bazel/_bazel_jan' must be readable and writable.
Permissions for the directory are:
drwxrwxr-x 6 jan jan 4096 Jul 2 16:47 _bazel_jan
The UID inside and outside of the container are also matching. Is there any way to make dazel work with a non-root user inside the container?
Thanks,
Jan

When i try running dazel build, there's error messages:
Using default tag: latest
Pulling repository docker.io/dazel/dazel_fed7773bed7a1f4bb3232ec2307c080e
Error: image dazel/dazel_fed7773bed7a1f4bb3232ec2307c080e:latest not found

When i try with sudo docker pull, there's an error message:
Using default tag: latest
Pulling repository docker.io/library/dazel
Error: image library/dazel:latest not found

What can be done to solve this?

Thanks!!!

I really like this project but I'm a bit weary to depend on it for my whole team without any tests and CI/CD. Right now the process for deployment is very manual and it seems like you guys have been manually pushing the package to pip when new changes are merged.

There are a few options for CI/CD tools that can be used, but I've worked with travis-CI and circleCI in the past which are both free for open-source projects.

Thoughts on setting up CI/CD? I'm happy to help get this set up (even if there are no tests currently).