prefecthq / prefect-docker Goto Github PK

View Code? Open in Web Editor NEW

43.0 10.0 13.0 2.32 MB

Prefect integrations for working with Docker

Home Page: https://prefecthq.github.io/prefect-docker/

License: Apache License 2.0

Python 100.00%

docker prefect python

prefect-docker's Introduction

Note

Active development of this project has moved within PrefectHQ/prefect. The code can be found here and documentation here. Please open issues and PRs against PrefectHQ/prefect instead of this repository.

prefect-docker

Welcome!

Prefect integrations for working with Docker.

Note! The DockerRegistryCredentials in prefect-docker is a unique block, separate from the DockerRegistry in prefect core. While DockerRegistry implements a few functionality from both DockerHost and DockerRegistryCredentials for convenience, it does not allow much configuration to interact with a Docker host.

Do not use DockerRegistry with this collection. Instead, use DockerHost and DockerRegistryCredentials.

Getting Started

Python setup

Requires an installation of Python 3.8+.

We recommend using a Python virtual environment manager such as pipenv, conda, or virtualenv.

These tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to the Prefect documentation.

Installation

Install prefect-docker with pip:

pip install prefect-docker

Then, register to view the block on Prefect Cloud:

prefect block register -m prefect_docker

Note, to use the load method on Blocks, you must already have a block document saved through code or saved through the UI.

Pull image, and create, start, log, stop, and remove Docker container

from prefect import flow, get_run_logger
from prefect_docker.images import pull_docker_image
from prefect_docker.containers import (
    create_docker_container,
    start_docker_container,
    get_docker_container_logs,
    stop_docker_container,
    remove_docker_container,
)


@flow
def docker_flow():
    logger = get_run_logger()
    pull_docker_image("prefecthq/prefect", "latest")
    container = create_docker_container(
        image="prefecthq/prefect", command="echo 'hello world!' && sleep 60"
    )
    start_docker_container(container_id=container.id)
    logs = get_docker_container_logs(container_id=container.id)
    logger.info(logs)
    stop_docker_container(container_id=container.id)
    remove_docker_container(container_id=container.id)
    return container

Use a custom Docker Host to create a Docker container

from prefect import flow
from prefect_docker import DockerHost
from prefect_docker.containers import create_docker_container

@flow
def create_docker_container_flow():
    docker_host = DockerHost(
        base_url="tcp://127.0.0.1:1234",
        max_pool_size=4
    )
    container = create_docker_container(
        docker_host=docker_host,
        image="prefecthq/prefect",
        command="echo 'hello world!'"
    )

create_docker_container_flow()

Resources

If you encounter any bugs while using prefect-docker, feel free to open an issue in the prefect-docker repository.

If you have any questions or issues while using prefect-docker, you can find help in the Prefect Slack community.

Feel free to ⭐️ or watch prefect-docker for updates too!

Development

If you'd like to install a version of prefect-docker for development, clone the repository and perform an editable install with pip:

git clone https://github.com/PrefectHQ/prefect-docker.git

cd prefect-docker/

pip install -e ".[dev]"

# Install linting pre-commit hooks
pre-commit install

prefect-docker's People

Contributors

Stargazers

Watchers

Forkers

saint1991 emilrex emada jpeggcv alexrogalskiy kvalenti strcom geoffrp enlitia id-2 rpeden vinzenz

prefect-docker's Issues

Create `stop_docker_container` task

Create a tasks that is able to stop a running Docker container. The task should accept a DockerHost block and the ID of the container to stop running. The container should then be stopped, but not removed. The container would then be available to start again at a later time.

Sync `cruft` with the latest template commit

This repo doesn't have a template sync workflow. Running cruft update and committing the result should pull in the latest template sync workflow and enable automation from here one out.

Add `start_container` task

Add `start_docker_container` task

Create task that is capable of starting a Docker container. The task should accept a DockerHost block and the ID of the container to start. Once the task is run, the specify container should be running.

Add `get_docker_container_logs` task

Create a task that retrieves the logs from a running container. The task should accept a container ID and DockerSettings block and return a str representation of the container logs at the point that the tasks is invoked.

Release `prefect-docker`

Complete the v0.1.0 release of prefect-docker. To be performed after all other work in the milestone is complete.

Verify documentation
Create GitHub release and trigger publish to PyPI
Verify installation from PyPI
Setup GitHub pages for documentation

Wrong type hint

task function actually does not typing.Container but it returns docker.models.containers.Container.

Add `pull_docker_image` task

Create a task that is capable of pulling a Docker image from a Docker registry onto the current machine. Special consideration will need to be given to how this task will interact with the DockerRegistry block that already exists in the core library.

How do we wait for container to exit on its own?

Given a very simple use case: Prefect flow that runs a container that waits for 30 seconds (sleep 30), there seems to be no way for the flow or task to "wait" for the container to exit on its own.

Minimally reproducible example

python3 -m venv venv && source venv/bin/activate
python -m pip install 'prefect-docker==0.1.0' 'prefect==2.7.11'
python flows/example.py && docker ps -a --filter "name=prefect-docker-mre"

example_flow.py

from prefect import flow, get_run_logger
from prefect_docker.images import pull_docker_image
from prefect_docker.containers import (
    create_docker_container,
    start_docker_container,
    get_docker_container_logs,
    stop_docker_container,
    remove_docker_container,
)


@flow
def docker_flow():
    logger = get_run_logger()
    pull_docker_image("ubuntu", "latest")
    container = create_docker_container(
        image="ubuntu", command="sleep 30"
    )
    start_docker_container(container_id=container.id)
    logs = get_docker_container_logs(container_id=container.id)
    logger.info(logs)
    return container

docker_flow()

What happens

Image is pulled
Container is successfully started
Logs are collected immediately after container start
Container is immediately scheduled for shut down
The kernel kills the container resulting in exit code 137

What is expected to happen

Image is pulled
Container is successfully started
Prefect flow / task waits for container to exit
-- there is no need for stopping container
Full logs are captured

Logs

22:22:04.279 | INFO    | prefect.engine - Created flow run 'literate-manatee' for flow 'docker-flow'
22:22:05.795 | INFO    | Flow run 'literate-manatee' - Created task run 'pull_docker_image-0' for task 'pull_docker_image'
22:22:05.797 | INFO    | Flow run 'literate-manatee' - Executing 'pull_docker_image-0' immediately...
22:22:06.369 | INFO    | Task run 'pull_docker_image-0' - Pulling image: ubuntu:latest.
22:22:07.872 | INFO    | Task run 'pull_docker_image-0' - Finished in state Completed()
22:22:08.197 | INFO    | Flow run 'literate-manatee' - Created task run 'create_docker_container-0' for task 'create_docker_container'
22:22:08.199 | INFO    | Flow run 'literate-manatee' - Executing 'create_docker_container-0' immediately...
22:22:08.834 | INFO    | Task run 'create_docker_container-0' - Creating container with 'ubuntu' image.
22:22:09.062 | INFO    | Task run 'create_docker_container-0' - Finished in state Completed()
22:22:09.319 | INFO    | Flow run 'literate-manatee' - Created task run 'start_docker_container-0' for task 'start_docker_container'
22:22:09.321 | INFO    | Flow run 'literate-manatee' - Executing 'start_docker_container-0' immediately...
22:22:09.886 | INFO    | Task run 'start_docker_container-0' - Starting container '6cdba73ebbf60426f9d27a7fc4de452bc782fa8aa008c4f5ee1e777807903ca7'.
22:22:10.283 | INFO    | Task run 'start_docker_container-0' - Finished in state Completed()
22:22:10.474 | INFO    | Flow run 'literate-manatee' - Created task run 'get_docker_container_logs-0' for task 'get_docker_container_logs'
22:22:10.476 | INFO    | Flow run 'literate-manatee' - Executing 'get_docker_container_logs-0' immediately...
22:22:11.703 | INFO    | Task run 'get_docker_container_logs-0' - Retrieving logs from '6cdba73ebbf60426f9d27a7fc4de452bc782fa8aa008c4f5ee1e777807903ca7' container.
22:22:12.122 | INFO    | Task run 'get_docker_container_logs-0' - Finished in state Completed()
22:22:12.124 | INFO    | Flow run 'literate-manatee' - 
22:22:12.323 | INFO    | Flow run 'literate-manatee' - Created task run 'stop_docker_container-0' for task 'stop_docker_container'
22:22:12.324 | INFO    | Flow run 'literate-manatee' - Executing 'stop_docker_container-0' immediately...
22:22:13.050 | INFO    | Task run 'stop_docker_container-0' - Stopping container '6cdba73ebbf60426f9d27a7fc4de452bc782fa8aa008c4f5ee1e777807903ca7'.
22:22:23.401 | INFO    | Task run 'stop_docker_container-0' - Finished in state Completed()
22:22:23.707 | INFO    | Flow run 'literate-manatee' - Finished in state Completed()
CONTAINER ID   IMAGE     COMMAND      CREATED          STATUS                      PORTS     NAMES
6cdba73ebbf6   ubuntu    "sleep 30"   16 seconds ago   Exited (137) 1 second ago             prefect-docker-mre

As one can see, the "sleep 30" is still running, although the prefect task is done..

How do we have the prefect task "running" for the duration of the container in order to retrieve full logs of the container as well as its status for indication of success or failure?

Add `remove_docker_container` task

Create a task that is capable of removing a Docker container. The task should accept a DockerHost block and a container ID. The container with the given ID should then be deleted from the Docker host.

Create `DockerSettings` block

Create a block that is capable of storing configuration for interacting with a Docker host and provides a method to retrieve a Docker client capable of communicating with the configured host.

Add `create_docker_container` task

Create a task that allows a Docker container to be created from a Docker image. The user should be able to specify the name of the image that they want to use to create the container, an optional name for the container, an optional command to run in the created container, and an option to run the container in the background.

Create `DockerWorker`

Create a BaseWorker implementation that executes flow runs by running them in a Docker container. A DockerWorker class should include all the configuration options and functionality offered by the existing DockerContainer infrastructure block.

Docker registry authentication will require more investigation to design a solution that will work with a variety of registry services. In the interim, authentication can be handled Docker CLI outside of the worker.