I've been on a GitHub issues research spree today looking for a viable alternative to Makefiles with their archaic syntax, checking out doit and fabfile in the process and making some detours over other task runners, however none of them really fit my definition of modern Python or just didn't play nice with Poetry... and then eventually I came across your post in the Poetry repo and gotta say, this is exactly what I've been looking for. It's similar to npm scripts, easy to use, comes with all the batteries included that I need and worked perfectly out of the box. It looks like there finally is a solution that integrates well with Poetry without the learning curve that other tools bring...

Man, kudos to you, you made my day :)

Be sure to create some donations page so I can invite you to a few beers!
(And feel free to quote this in your README if you want)

Setting this bound forces every project that depends on yours to also set it. Reasons why it's not obviously a good thing to do: https://iscinumpy.dev/post/bound-version-constraints/ (I'll let you form your own opinion of course).

It would be nice that the poet support multiple pyproject.toml files. When a pyproject.toml file exist in a cloned project, it would be nice to be able to add new commands without modifying the project. In such case I can add a config file in parent directories.

Thanks in advance.

My tasks:

[tool.poe.tasks]
default_array_item_task_type = "shell"

[tool.poe.tasks.clean]
ignore_fail = true
sequence = [
    "rm -rf .coverage* 2>/dev/null",
    "rm -rf .mypy_cache 2>/dev/null",
    "rm -rf .pytest_cache 2>/dev/null",
    "rm -rf build 2>/dev/null",
    "rm -rf dist 2>/dev/null",
    "rm -rf pip-wheel-metadata 2>/dev/null",
    "rm -rf site 2>/dev/null",
    "find . -type d -name __pycache__ | xargs rm -rf 2>/dev/null",
    "find . -name '*.rej' -delete 2>/dev/null",
]

Running task clean:

$ poetry run poe clean
Traceback (most recent call last):
  File "/path/to/venv/bin/poe", line 8, in <module>
    sys.exit(main())
  File "/path/to/venv/site-packages/poethepoet/__init__.py", line 19, in main
    result = app(cli_args=sys.argv[1:])
  File "/path/to/venv/site-packages/poethepoet/app.py", line 50, in __call__
    return self.run_task() or 0
  File "/path/to/venv/site-packages/poethepoet/app.py", line 81, in run_task
    dry=self.ui["dry_run"],
  File "/path/to/venv/site-packages/poethepoet/task/base.py", line 145, in run
    return self._handle_run(executor, list(extra_args), project_dir, env, dry)
  File "/path/to/venv/site-packages/poethepoet/task/sequence.py", line 64, in _handle_run
    extra_args=tuple(), project_dir=project_dir, env=env, dry=dry,
  File "/path/to/venv/site-packages/poethepoet/task/base.py", line 145, in run
    return self._handle_run(executor, list(extra_args), project_dir, env, dry)
  File "/path/to/venv/site-packages/poethepoet/task/ref.py", line 41, in _handle_run
    task = self.from_config(self.content, self._config, ui=self._ui)
  File "/path/to/venv/site-packages/poethepoet/task/base.py", line 86, in from_config
    raise Exception(f"Cannot instantiate unknown task {task_name!r}")
Exception: Cannot instantiate unknown task 'rm -rf .coverage* 2>/dev/null'

It seems poe (v0.7.0) still tries to match each item as a task reference.

How to reproduce

With the following setup

pyproject.toml

[tool.poe]
include = "./tasks.toml"

tasks.toml

[tool.poe.tasks]
foo = { cmd = "echo bar }

Run

poe foo

Output

Traceback (most recent call last):
  File "/Users/.../bin/poe", line 8, in <module>
    sys.exit(main())
  File "/Users/.../poethepoet/__init__.py", line 32, in main
    result = app(cli_args=sys.argv[1:])
  File "/Users/.../poethepoet/app.py", line 44, in __call__
    self.config.load(self.ui["project_root"])
  File "/Users/.../poethepoet/config.py", line 113, in load
    self._load_includes(self._project_dir)
  File "/Users/.../poethepoet/config.py", line 250, in _load_includes
    self._merge_config(tomli.load(file), include_path)
  File "/Users/.../poethepoet/config.py", line 284, in _merge_config
    own_tasks = self._poe["tasks"]
KeyError: 'tasks'

Workaround

Add an empty tasks inline table to the config

pyproject.toml

[tool.poe]
include = "./tasks.toml"
tasks = {}

Environment

I am trying to set up a poe task which runs a script with a single argument:

[[tool.poe.tasks.download-env]]
shell = "./scripts/download-env.sh"
help = "Download an environment profile"

    [[tool.poe.tasks.download-env.args]]
    name = "profile"
    help = "The name of the environment profile to download"
    default = "dev"
    options = ["-p", "--profile"]
    required = true

Running the above task produces the following error:

% poetry run poe download-env --profile dev    
usage: poe download-env[0] -p PROFILE
poe download-env[0]: error: the following arguments are required: -p/--profile

I have tried running this from a clean project, bash, zsh, as well as removing around with the "required" option, which produces the following:

poetry run poe download-env --profile staging
Poe the Poet - A task runner that works well with poetry.
version 0.13.1

Error: Sequence task 'download-env' does not accept arguments 
...

I am unsure if this is an issue with the project or my environment but as I've been unable to find much documentation on the issue I figured this is the best forum to ask. Thanks!

Another model for task composition (in addition to sequence tasks) would be to support declaring dependencies between tasks, such that invoking a task with dependencies would first run all its dependencies (and their dependencies and so on).

Thus an execution DAG of tasks could easily be defined as in the following

Config proposal A

[tool.poe.tasks]
test = "pytest src"
version.script = "scripts:get_current_version"

  [tool.poe.tasks.build]
  cmd = "docker build -f Dockerfile -t my_app:$VERSION ."
  deps = { "" = ["test -v"], VERSION = "version" }

Config proposal B

[tool.poe.tasks]
test = "pytest src"
version.script = "scripts:get_current_version"

  [tool.poe.tasks.build]
  cmd = "docker build -f Dockerfile -t my_app:$VERSION ."
  deps = ["test -v"]
  uses = { VERSION = "version" }

Note that in this example scenario the dependency on the test task demonstrates that dep tasks could be passed arguments (or even env vars).

This example also demonstrates the output of one task being captured and made available to the dependant task as an env var hence addressing #21. Since this required silencing (capturing) the output of the dependency, this could allow for multiple tasks to be run in parallel if their dependencies are all satisfied at the same time and neither of them will write to the console (related #24).

This could be particularly powerful if combined with #12 , getting us closer to the power of make.

I've started prototyping this already, but comments are more than welcome.

Original issue title: Poe tries to enforce WSL on Windows 10

When running a poe shell-type task in Windows Powershell Core 7.0.3, it seems to attempt execution of the command inside the Windows Subsystem for Linux (WSL), but I am happy using the cross-platform Powershell for most use cases.

Using the following configuration:

[tool.poe.tasks]
test = { shell = 'test' }

It complains that the WSL is not installed:

PS C:\projects\someproject> poe test
Poe => test
Windows Subsystem for Linux has no installed distributions.
Distributions can be installed by visiting the Microsoft Store:
https://aka.ms/wslstore

Hi,

Maybe I'm missing something, but it looks like a task sequence with ingore_fail = true will run all the tasks and in the end will report zero-exit status even if one of the tasks ended with non-zero exit status. Which is logical considering the name of the option.

But I think that it'll be really nice to have a similar option (or to change the logic behind the ignore_fail option) to report non-zero exit status when any of the tasks in sequence finished with the error. 

I can draft a pull-request if you think this is a good idea.

Hello. I am not able to get a "script" task to actually run on Windows (git bash). For sanity testing, I am trying to just run the "poe" task from this repo. It is configured in your pyproject.toml as:

  [tool.poe.tasks.poe]
  help   = "Execute poe from this repo (useful for testing)"
  script = "poethepoet:main

I am running it via:

poetry run poe poe

On windows, I get nothing:

[13:38:53] Tim@WINDOWS ~/temp/poethepoet/poethepoet (main)
$ poetry run poe poe
Poe => poe

On ubuntu, it works:

[13:38:26] tcamise@UBUNTU ~/temp/poethepoet (main)
$ poetry run poe poe
Poe => poe
Poe the Poet - A task runner that works well with poetry.
version 0.12.2

Result: No task specified.

USAGE
  poe [-h] [-v | -q] [--root PATH] [--ansi | --no-ansi] task [task arguments]

GLOBAL OPTIONS
  -h, --help     Show this help page and exit
  --version      Print the version and exit
  -v, --verbose  Increase command output (repeatable)
  -q, --quiet    Decrease command output (repeatable)
  -d, --dry-run  Print the task contents but don't actaully run it
  --root PATH    Specify where to find the pyproject.toml
  --ansi         Force enable ANSI output
  --no-ansi      Force disable ANSI output

CONFIGURED TASKS
  format         Run black on the code base
  clean          Remove generated files
  test           Run unit and feature tests
  test-quick     Run unit and feature tests, excluding slow ones
  types          Run the type checker
  lint           Run the linter
  style          Validate code style
  check-docs     Validate rst syntax in the docs
  check          Run all checks on the code base
  poe            Execute poe from this repo (useful for testing)

Am I doing something dumb here?

Dear Nat,

within wireviz-web, we want convert a make release target [1] to a Poe the Poet task. Currently, this target can be invoked by, for example, either make release or make release bump=patch.

Therefore, we need some features which have not been obvious to us from reading the documentation. Let me share some insights about what our needs are, maybe those features are already implemented.

• We want to make this task obtain an optional extra argument bump from the command line. If it is omitted, it should use a specific default value (here: minor).
• We want to set a variable at runtime (here: version), which obtains the current package version by invoking poetry version --short. This variable should be available within subsequent task steps.

We started implementing this as a "shell" task [2] but are currently blocked by

• uiri/toml#348 - due to black and isort not able to parse the given section [2] and
• #17 - the possibility to make the "shell" task obtain any extra arguments at all.

Maybe this use case sparks your interest. We will be happy if you or someone from the community will be able to do something about it.

With kind regards,
Andreas.

[1] https://github.com/daq-tools/wireviz-web/blob/f9be8b7b/Makefile#L4-L11
[2] https://github.com/daq-tools/wireviz-web/blob/f9be8b7b/pyproject.toml#L84-L92

I just added poetry (latest, 1.0.10) and poethepoet (latest, 0.8.0) to a virtualenv, and after installing poethepoet I get the following message:

$ pip install poethepoet
Collecting poethepoet
  Downloading poethepoet-0.8.0-py3-none-any.whl (23 kB)
Collecting tomlkit<0.8.0,>=0.7.0
  Downloading tomlkit-0.7.0-py2.py3-none-any.whl (32 kB)
Requirement already satisfied: pastel<0.3.0,>=0.2.0 in ./venv/lib/python3.8/site-packages (from poethepoet) (0.2.1)
Installing collected packages: tomlkit, poethepoet
  Attempting uninstall: tomlkit
    Found existing installation: tomlkit 0.5.11
    Uninstalling tomlkit-0.5.11:
      Successfully uninstalled tomlkit-0.5.11
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

poetry 1.0.10 requires tomlkit<0.6.0,>=0.5.11, but you'll have tomlkit 0.7.0 which is incompatible.

For a complicated, multi-threaded application that uses zmq, I get the following error when I Ctrl-C the application that I started using poe to run the task "poetry run python .\\main.py"

Traceback (most recent call last):
  File "C:\Users\dmwyatt\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\dmwyatt\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\Scripts\poe.exe\__main__.py", line 7, in <module>
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\__init__.py", line 32, in main
    result = app(cli_args=sys.argv[1:])
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\app.py", line 52, in __call__
    return self.run_task() or 0
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\app.py", line 80, in run_task
    return self.task.run(
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\task\base.py", line 142, in run
    return self._handle_run(context, extra_args, env)
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\task\cmd.py", line 39, in _handle_run
    return context.get_executor(env, self.options.get("executor")).execute(cmd)
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\executor\poetry.py", line 31, in execute
    return self._execute_cmd(cmd, input)
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\executor\poetry.py", line 64, in _execute_cmd
    return self._exec_via_subproc(
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\executor\base.py", line 134, in _exec_via_subproc
    proc.communicate(input)
  File "C:\Users\dmwyatt\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1126, in communicate
    self.wait()
  File "C:\Users\dmwyatt\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1189, in wait
    return self._wait(timeout=timeout)
  File "C:\Users\dmwyatt\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1470, in _wait
    result = _winapi.WaitForSingleObject(self._handle,
  File "C:\Users\dmwyatt\AppData\Local\pypoetry\Cache\virtualenvs\win-backup-exclude-MbZQpObE-py3.9\lib\site-packages\poethepoet\executor\base.py", line 129, in handle_signal
    proc.send_signal(signum)
  File "C:\Users\dmwyatt\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1545, in send_signal
    raise ValueError("Unsupported signal: {}".format(sig))
ValueError: Unsupported signal: 2

Other notes

• Other sorts of tasks like format = "black ." run just fine, so it has something to do with how poe is running my application.
• If I don't use poe to run my application it exits just fine without a traceback.
• The last things my application does when exiting via a try...finally block is terminating the zmq context and closing several sockets...but removing those actions doesn't solve this traceback popping up.
• I can try to come up with a minimal reproduction, but I expect that to be difficult so I wanted to see if anyone had any ideas first.

Would it possible to add a variable that allows disabling path expansion using asterisk? See pyinstaller/pyinstaller#6751 for why. Thanks!

I'm trying to get the result of a poe or (shell) command as a environment variable for another POE task. It is dynamic and cannot be set beforehand.

Is it possible?

Thank you

Is your feature request related to a problem?

I want to define environment variables for tasks with values that will be substituted with already defined environment variables.

Describe the solution you'd like

Let's consider this dummy task:

[tool.poe.tasks]
"path" = { cmd = "echo $MY_PATH", env = { MY_PATH = "$PATH" }}

When a run poe path, I want the value of $PATH to be resolved instead of only echoing $PATH as a string.

Describe alternatives you've considered

Using shell tasks by:

• defining the environment variable inline: no success

  [tool.poe.tasks]
  "path" = { shell = "MY_PATH=$PATH echo $MY_PATH" }

• defining the environment variable with Poe: no success

  [tool.poe.tasks]
  "path" = { shell = "echo $MY_PATH", env = { MY_PATH = "$PATH" }}

Poe the Poet is already quite modular by design, and easy to extend with new task or executor types. However I can imagine that there may be use cases for kinds of tasks or ways of executing them that I wouldn’t relate to so easily or wouldn’t know enough about the context. For example defining tasks in terms of plumbum scripts, or executing them in an environment managed with nix.

To support this poe could be made to support specifying a plugin as a python module referenced by a global config option (e.g. tool.poe.plugins = ["poethepoet_contrib_pipenv.executor"]). Providing such a config would cause poe to load the specified module on every run, with the expectation that the module will define and register PoeExecutor/PoeTask classes that may then be used by tasks defined in the same file.

One caveat is that the python module containing the plugin would have to be available from the same environment as poe itself, and so not have any dependencies that require activating a different virtual environment for instance.

When creating a Docker image FROM python:3.8-slim AS base, we set poetry config virtualenvs.create false as the image itself already isolates the Python environment. In that situation, Poe will still try to activate the environment with every task, which works but does generate some unnecessary noise.

Workaround: we currently set ENV POETRY_ACTIVE 1 to avoid the unnecessary activations.

I'm trying poethepoet by migrating my Makefile to poe tasks 🙂

One thing I do in the Makefile is to declare the paths to directories to be scanned by code quality tools:

PY_SRC = src/ tests/

check-code-quality:
	@flake8 $(PY_SRC)

I would like to do the same with poe:

[tool.poe.env]
PY_SRC = "src/ tests/"

[tool.poe.tasks]
check-code-quality = "flake8 $PY_SRC"

Based on the README, I don't think this is possible yet.

I've read the code and I can't find any way to make a sequence accept an argument. The code actually says to throw an exception if any extra arguments are given :(

I need to run the task setup-artifacts before I run the task poetry add. Since poetry add takes an argument I need to feed it to the last command in the list. This seems unsupported. I read the code and I get lost but this tells me SequenceTasks don't accept arguments:

My pyproject.toml reads:

[tool.poe.tasks]
setup-artifacts = "scripts/setup_artifacts.sh setup-poetry"
add = [{ref="setup-artifacts"}, { cmd = "poetry add" }]

I run poe add moto and I get:

poe add moto

Poe the Poet - A task runner that works well with poetry.
version 0.10.0

Error: Sequence task 'add' does not accept arguments

USAGE
  poe [-h] [-v | -q] [--root PATH] [--ansi | --no-ansi] task [task arguments]

GLOBAL OPTIONS
  -h, --help     Show this help page and exit
  --version      Print the version and exit
  -v, --verbose  More console spam
  -q, --quiet    Less console spam
  -d, --dry-run  Print the task contents but don't actaully run it
  --root PATH    Specify where to find the pyproject.toml
  --ansi         Force enable ANSI output
  --no-ansi      Force disable ANSI output

CONFIGURED TASKS
  setup-artifacts
  add

This makes me a sad panda :( What would I need to do to make poe take arguments to sequences? Maybe put an argument for them like args_to=2 to reference the command in the sequence?

This code hasn't been tested on windows and probably won't work, but it probably could. A windows developer is needed to try it out and find a more robust cross platform solution.

So right now poe checks the $SHELL variable and uses that shell to run the commands when using shell type.

Why was this the choice? Seems a bit strange. While I do use fish myself, I do not want to write poe tasks on my projects to also use it. I would like everyone to be able to jus run poe commands if they checkout the code without having fish.

make for example ignores $SHELL. And this makes sense because these command should be cross platform and not require a developer setup a specific environment to be able to use the project.

Anyway, this would be a breaking change so maybe configuration is the way to go? I could try make a pull request myself if there is some consensus on what's the best way to handle this.

I've needed to define a poe task that strings together shell commands with &&.

{ shell = "foo && bar && ..." }

How is this different from the non-shell (?) poe tasks? What should I be aware of when distributing to other devs on my team (e.g., will it break if they have a different shell... just spitballing this)?

On the theme of making Poe the Poet work well without poetry: in the same way that poe makes it seamless to run tasks for a poetry based project, it could offer similar integrations for project environments managed in other ways, such as with a regular venv. This would require selecting the appropriate implementation of PoeExecutor at runtime.

By default poe should make a good guess about which executor to use, so if the pyproject.toml configures dependencies with poetry then the existing PoetryExecutor would be used as usual. Otherwise it should check if there’s a virtualenv at ./venv or ./.venv, and if so it should use a VirtualenvExecutor, and otherwise fallback to an executor that simply runs the tasks without doing any setup or linking to a specific environment

Of course it should also be possible to explicitly configure when executor should be used, and relevant details such as a relative path to the virtualenv directory.

Environment

• OS: Windows 10
• Poetry: 1.2.0b1
• POE: 0.13.1

Issue

If you have a script that returns an error as system exit code, for example:

include sys

def test_script() -> None:
    sys.exit(1) 

In a file called sample.py in your repository root, and configured as follows on the pyproject.toml file:

[tool.poe.tasks]
    [tool.poe.tasks.test-script]
    help    = "Check if the returnvalue is being relayed"
    script  = "sample:test_script"

You'll see (windows):

> poe test-script
Poe => test-script
> echo %errorlevel%
0

Instead of:

> python .c "import sample as s; s.test_script()"
> echo %errorlevel%
1

Is this intended? Is there a way to relay the returncode from the failed script out of poe?

I'm trying to setup some commands and some require a number of Environment Variables, so I decided to try the Deep Keys method described at https://github.com/nat-n/poethepoet#environment-variables

Tasks configured:

[tool.poe.tasks]
init_db    = "alembic upgrade head"
dev.cmd    = "python -m flask run -p 5001"
dev.env    = { FLASK_ENV="development", FLASK_DEBUG="1" }

this breaks with poethepoet.exceptions.PoeException: Cannot instantiate unknown task 'flask_dev' from poethepoet/task/base.py:L85

Replacing the "dev" task with:

dev        = { cmd="python -m flask run -p 5001", env = { FLASK_ENV="development", FLASK_DEBUG="1" }}

fixes the problem

python -V = "Python 3.9.7"
poe --version = "version: 0.10.0"

I use poethepoet in Mac 2020 with pycharm. I start fastapi server with uvicorn. When i want to terminate the server ctrl+c or any other escape combinations does not work. It terminates the operation but does not turn to the console. What is the issue?

First of all, thank you for creating this tool!

I tried converting some commands to use poe and ran into a small issue. Namely, I am using a command-line tool that requires passing the path to a config file as ENV var.
Therefore, I was using a task like this:

{cmd = "tool", env = { CONFIG_PATH = "path/to/config.json"}}

This works great, if I use the absolute path to the config. However, the tool I am using does not support realitve paths to the config file (i.e. "./subdir/config.json") would not work.

Poe is aware of the current directory of the Python project. However, at the moment, it is not possible (at least I think) to directly subsititute this path into a command.

For example, what whould be awesome, if you could do something like this:

{cmd = "tool", env = { CONFIG_PATH = "{{POE_ROOT}}/subdir/config.json}}

The idea would be, that "{{POE_ROOT}}" would get replaced by poe, before the command is executed by the shell. This would be more powerful than just providing POE_ROOT (and others) as a global variable, as it would be independent of how different operating systems would handle variable substitution.

The "{{...}}" syntax is of course just a suggestion.

Hence, my feature request is:

• Add a simple templating syntax that is supported in the actual command strings and the env variables.
• The template strings are replaced by poe before the execution with the respective global variable (maybe simply the configured global env-vars?
• If no matching variable is found poe throws an error
• There could be a configuration to disable this functionality: e.g. {..., disable_var_substitution= true}

Let me know what you think of it and I would be open to give it a try :)

P.S.: I know that I could rewrite my task as a Python task to get the functionality I need, but it would be cool if there would be a way to write such tasks more easily.

Not sure if this is already possible, but don't see it mentioned anywhere yet.

Ideally I'd like to include some tasks for my package (A) and be able to import them as local tasks (B).
Alternatively, it may be simpler to have a local tasks subdirectory (B) and copy/sync the mypackage.tasks (from A).

The challenge is that pyproject.toml is large and manually syncing these tasks can be error-prone and tedious.
This could also open the possibility of writing a script to generate those tasks based on the local configuration.
While it could probably be done by editing pyproject.toml directly, a separate file would be a lot safer.

Thanks for considering.
Kevin

Hey @nat-n, first of all thanks for that great tool! ❤️

I couldn't find any functionality to map more than one arg to a single arg.

Example

Maybe someone wants to have a python formatting task a la

[tool.poe.tasks.format]
cmd = "black ${files}" 
help = "Format files
args = [
  { name = "files", positional = true, default = ".", help = "List of files (optional)" },
]

Then, this task may be run e.g. via

• poe format
• poe format "."
• poe format /path/to/file
• poe format "/path/to/file"
• poe format "/path/to/file/one /path/to/file/two", etc.

but not poe format /path/to/file/one /path/to/file/two. – Note the missing "'s in the example.

In my special case, I want to run tasks as pre-commit hooks using the pre-commit tool which, unfortunately, passes multiple filenames without the "'s.

Any idea how this can be resolved? – Maybe I have also just missed something :-)

When using Poe the Poet as an entrypoint in a Docker image, it would be nice to be able to exec [1] commands. For example: exec gunicorn my_service.api:app. In Python, this can be achieved with os.execvp [2].

[1] https://man7.org/linux/man-pages/man3/exec.3.html
[2] https://docs.python.org/3/library/os.html#os.execvp

I just want to add a disclaimer: I'm not sure whether poe was designed the accept extra arguments beyond what was declared for a task, but I noticed that supplying extra args after -- seems to include the extra arguments, so that's what I was trying here

For example, given the following task declaration:

[tool.poe.env]
PYTHONPATH = "${PWD}/examples"
DJANGO_SETTINGS_MODULE = "config.settings"

[tool.poe.tasks.migrate]
cmd = "django-admin migrate --pythonpath ${PWD}/examples/$example"
args = [{name = "example", default = "my_app_example", positional = true}]
envfile = ".env"

If I run it using poe migrate -- --help, poe will translate it to:

django-admin migrate --pythonpath /home/username/projects/my_django_app/examples/--help

This causes an error because as you can see, there is no space between the last argument and --help, however without the extra -- --help, everything works, but I can't pass more arguments to the command.

Please help me know if this is defined behavior and how to work around it, or whether it's a bug. Thanks

So far tab completion is available for tasks across multiple shells. For zsh is is also available for global CLI options of poe.

The ability to declare CLI options for tasks would be more delightful if at least the ZSH completions also worked for task arguments.

I really want this feature, but don't know when I'll have time to relearn how zsh completions work, so if someone else out there already has this knowledge and could extend the logic here to cover task arguments then that would be Ace!

Certain packages, such as onnxruntime rely on external DLLs such as CUDA.

To provide some context, on Windows, Python uses %PATH% or os.add_dll_directory when searching for DLLs. On Linux, Python uses $LD_LIBRARY_PATH when searching for DLLs.

Based on https://stackoverflow.com/questions/856116/changing-ld-library-path-at-runtime-for-ctypes however, Python does not notice changes to $LD_LIBRARY_PATH at runtime. This means adding DLL directories to $LD_LIBRARY_PATH via os.environ does not work.

To give an example:

[tool.poe.tasks]
# Fails to load CUDA DLLs
cmd_based.cmd = "python app.py"
cmd_based.env = { LD_LIBRARY_PATH = "~/code/cuda/lib" }

# successfully loads CUDA DLLs
shell_based.shell = "LD_LIBRARY_PATH=~/code/cuda/lib python app.py"

While a workaround would be to use shell tasks in such cases, in my use case, app.py accepts an arbitrary number of positional arguments for configuration (e.g. python app.py foo.a=1 bar.b=2 ...). However, based on #17, I don't think there is a way to specify an arbitrary number of positional arguments for a shell task? Hence, how would I go about solving this issue?

I see in the TODO section of the README:

support declaring specific arguments for a task

Could you elaborate on what it means 🙂 ?

I'm currently trying to migrate a Makefile to poe tasks, and I'd need to pass named arguments to a task:

.PHONY: do-this
do-this:
ifndef a
	$(error Pass A with 'make do-this a=X.Y.Z')
endif
ifndef b
	$(error Pass B with 'make do-this b=X.Y.Z')
endif
ifndef c
	$(error Pass C with 'make do-this c=X.Y.Z')
endif
	@echo "actual commands..."
	@echo $(a) $(b) $(c)

I guess it gets a bit hard to replicate such features in a declarative way.

Maybe something like this?

[tool.poe.tasks.do-this]
args = [
    { name = "a", help = "Pass A with 'a=X.Y.Z'", required = true },
    { name = "b", help = "Pass B with 'b=X.Y.Z'", required = true },
    { name = "c", help = "Pass C with 'c=X.Y.Z'", required = true },
]
shell = """
    echo "actual commands...";
    echo $a $b $c
"""

...letting us do poe do-this a=hey b=ho c=heyho.

A more verbose equivalent:

[tool.poe.tasks.do-this]
  shell = """
    echo "actual commands...";
    echo $a $b $c
  """

  [[tool.poe.tasks.do-this.args]]
    name = "a"
    help = "Pass A with 'a=X.Y.Z'"
    required = true
  [[tool.poe.tasks.do-this.args]]
    name = "b"
    help = "Pass B with 'b=X.Y.Z'"
    required = true
  [[tool.poe.tasks.do-this.args]]
    name = "c"
    help = "Pass C with 'c=X.Y.Z'"
    required = true

How would I write the following rule as a poe task?

.PHONY: clean
clean:  ## Delete temporary files.
	@rm -rf .coverage* 2>/dev/null
	@rm -rf .mypy_cache 2>/dev/null
	@rm -rf .pytest_cache 2>/dev/null
	@rm -rf build 2>/dev/null
	@rm -rf dist 2>/dev/null
	@rm -rf pip-wheel-metadata 2>/dev/null
	@rm -rf site 2>/dev/null
	@find . -type d -name __pycache__ | xargs rm -rf 2>/dev/null
	@find . -name "*.rej" -delete 2>/dev/null

I'd be tempted to do this:

[tool.poe.tasks.clean]
cmd = """
    rm -rf
        .coverage*
        .mypy_cache
        .pytest_cache
        build
        dist
        pip-wheel-metadata
        site
        2>/dev/null;
    find . -type d -name __pycache__ | xargs rm -rf 2>/dev/null;
    find . -name "*.rej" -delete 2>/dev/null;
"""

...but then I think the ; would be considered as arguments instead of command separator.

If I use a shell type instead:

[tool.poe.tasks.clean]
shell = """
    rm -rf \
        .coverage* \
        .mypy_cache \
        .pytest_cache \
        build \
        dist \
        pip-wheel-metadata \
        site \
        2>/dev/null;
    find . -type d -name __pycache__ | xargs rm -rf 2>/dev/null;
    find . -name "*.rej" -delete 2>/dev/null;
"""

...I guess I now have to escape the new lines?

Maybe I should keep each command on its own line, and separate them with ;?

[tool.poe.tasks.clean]
shell = """
    rm -rf .coverage* 2>/dev/null;
    rm -rf .mypy_cache 2>/dev/null;
    rm -rf .pytest_cache 2>/dev/null;
    rm -rf build 2>/dev/null;
    rm -rf dist 2>/dev/null;
    rm -rf pip-wheel-metadata 2>/dev/null;
    rm -rf site 2>/dev/null;
    find . -type d -name __pycache__ | xargs rm -rf 2>/dev/null;
    find . -name "*.rej" -delete 2>/dev/null;
"""

I could also define "hidden" tasks and make a sequence of them, but I would need to declare them as shell type:

[tool.poe.tasks]
default_task_type = "shell"
_rm_coverage = "rm -rf .coverage* 2>/dev/null"
_rm_mypy = "rm -rf .mypy_cache 2>/dev/null"
_rm_pytest = "rm -rf .pytest_cache 2>/dev/null"
_rm_build = "rm -rf build 2>/dev/null"
_rm_dist = "rm -rf dist 2>/dev/null"
_rm_wheel = "rm -rf pip-wheel-metadata 2>/dev/null"
_rm_site = "rm -rf site 2>/dev/null"
_rm_pycache = "find . -type d -name __pycache__ | xargs rm -rf 2>/dev/null"
_rm_rej = "find . -name '*.rej' -delete 2>/dev/null"
clean = [
    "_rm_coverage",
    "_rm_mypy",
    "_rm_pytest",
    "_rm_build",
    "_rm_dist",
    "_rm_wheel",
    "_rm_site",
    "_rm_pycache",
    "_rm_rej",
]

But then, the ideal would be to be able to define the array task type locally (which I think is not possible yet):

[tool.poe.tasks.clean]
array_item_task_type = "shell"
ignore_fail = true
sequence = [
    "rm -rf .coverage* 2>/dev/null",
    "rm -rf .mypy_cache 2>/dev/null",
    "rm -rf .pytest_cache 2>/dev/null",
    "rm -rf build 2>/dev/null",
    "rm -rf dist 2>/dev/null",
    "rm -rf pip-wheel-metadata 2>/dev/null",
    "rm -rf site 2>/dev/null",
    "find . -type d -name __pycache__ | xargs rm -rf 2>/dev/null",
    "find . -name '*.rej' -delete 2>/dev/null",
]

What do you feel is the most maintainable way? Which one is the less error-prone? Any thought is welcome here 🙂

Hello --

I've been using poe to launch a few console tools written in shell recently.
It works!

However, I've noticed that when I run interactive tools like mysql or psql clients,
it fails to launch them, even in simple scenarios.

Is this a known issue? Curious if there might be a workaround.

Thanks for the tools! Cheers --

Is it possible to create a task that executes multiple tasks in parallel?

I know I can create a compond task as

test = ["mypy", "pylint", "pytest"]

Calling poe test will run each task in sequence one after the other. It would be nice to be able to configure these task lists as safe to start in parallel.

Parallel should of course not be parallel by default since some tasks requires output from previous tasks (coverage being the prime example that needs a completed test run to before generating a coverage report).

Poe the poet is growing in complexity to the point where a single readme.rst isn’t the clearest form of documentation. Setting up a documentation website, probably sphinx based would be a lot better, and lay the groundwork for me useful docs.

Thanks for the great project!

I installed poe as a plugin using poetry plugin add poethepoet[poetry_plugin] with the latest pre-release of poetry. However, when I now run poetry in a directory where there is not pyproject.toml I get the error

  PoeException

  Poe could not find a pyproject.toml file in /my/path or its parents

  at .local/share/pypoetry/venv/lib/python3.9/site-packages/poethepoet/config.py:219 in find_pyproject_toml
      215│ 
      216│         maybe_result = self.cwd.joinpath(self.TOML_NAME)
      217│         while not maybe_result.exists():
      218│             if maybe_result.parent == Path("/"):
    → 219│                 raise PoeException(
      220│                     f"Poe could not find a pyproject.toml file in {self.cwd} or"
      221│                     " its parents"
      222│                 )
      223│             maybe_result = maybe_result.parents[1].joinpath(self.TOML_NAME).resolve()

This is inconvenient, because running poetry new is no longer possible this way.
I think the desired behavior should just be that the fact that no pyproject.toml exists should just be ignored.

The roadmap for poetry includes a plugin system python-poetry/poetry#1237 which should allow for poe to run inside the poetry CLI.

Creating a Poe the Poet plugin for poetry would ideally entail that the exact same config in the pyproject.toml could be used to the same effect either via the existing poe CLI, or via poe running as a poetry plugin and invoked via the the poetry CLI.

It'll probably look something like:

poetry poe test

which would be functionally equivalent to the following but with a shorter command a perhaps a minor performance boost:

poetry run poe test

Dear Nat,

thanks a stack for conceiving and maintaining Poe the Poet. I wish you a happy new year.

We are about to use it within wireviz-web and currently observe the error Shell task does not accept arguments when trying to make a "shell" task obtain extra arguments from the command line.

A minimal reproducible example would be

[tool.poe.tasks]
hello = { shell = "echo $1" }

which croaks with

$ poe hello world
Poe the Poet - A task runner that works well with poetry.
version 0.9.0

Error: Shell task 'hello' does not accept arguments

Is there anything you can do about it?

With kind regards,
Andreas.

I defined alias poe='poetry run poe', and followed "Enable tab completion for your shell" section, but not works.

Any help would be appreciated.

The goal of this project is to be the obvious choice of task runner for python projects, with the simplicity of npm scripts for simple use cases, but also with powerful features comparable to make (at least as far any python project is likely to need).

One key feature of make is being able to skip build targets that already exist.

I need to analyse this problem a bit more but so far I’m thinking an appropriate comparable feature would be for all tasks to support a condition option, which can either represent a relative path to a file or directory (with globbing?), the existence of which would make the condition false, or it could reference a python function to evaluate.

Open questions:

• what mechanisms should be available for overriding a condition? Is if enough to have a global --force CLI flag?

• how should this work for composite tasks? should it be possible to skip the condition for a single subtask? Or is all or nothing good enough?

• If conditions can be defined in multiple ways then this could work similarly to tasks, which a generally defined as dictionaries which can be differentiated between different types, or as just a string which is interpreted as a default type. What should the default type be? (I'm thinking python function) Is there a better way of representing this?

I noticed that poe takes much more time than the Makefile equivalent to run a sequence of shell commands .

$ time make clean

real    0m0.085s
user    0m0.027s
sys     0m0.037s

$ time poetry run make clean

real    0m0.683s
user    0m0.566s
sys     0m0.114s

$ time poe clean

real    0m1.567s
user    0m1.244s
sys     0m0.231s

Could it be because poe tries to run each item of a sequence through poetry run? Even though I'm running it with the poe='poetry run poe' alias?

Thanks for creating such a convenient tool 🙏🏻

One thing that is a bit confusing to me is command output redirect does not work:

>file: pyproject.toml
[tool.poe.tasks]
create-dotenv="echo 'VARIABLE=VALUE' > .env"

> poetry run poe create-dotenv
> ls -l .env
ls: .env: No such file or directory

Current workaround:

>file: pyproject.toml
[tool.poe.tasks]
create-dotenv="echo 'VARIABLE=VALUE'"

poetry run poe -q create-dotenv > .env

However, it does not play with chained tasks :(

I'm having an issue with the following in my pyproject.toml

[tool.poe.tasks]
run-tests = "python -m unittest discover -v ./tests/"

This works fine:

PS D:\PS\services\projects\scaler> poetry run poe run-tests
Poe => python -m unittest discover -v ./tests/
.... <Tests run>

However, this fails:

PS D:\PS\services\projects\scaler> poetry run poe run-tests -p "test_unit*.py"
Poe => python -m unittest discover -v ./tests/ -p
usage: python.exe -m unittest discover [-h] [-v] [-q] [--locals] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [-s START] [-p PATTERN] [-t TOP]
python.exe -m unittest discover: error: argument -p/--pattern: expected one argument

Specifically, "py_unit*.py" has disappeared. This argument is a glob-pattern to be passed as-is to Pytest, so should not be expanded. The bash-like behaviour of "use glob as-is if not matched" would be less-surprising, producing

Poe => python -m unittest discover -v ./tests/ -p "py_unit*.py"

The behaviour of course is closer-to-expected if I happen to be in 'tests'

PS D:\PS\services\projects\scaler\tests> poetry run poe run-tests -p "test_unit*.py"
Poe => python -m unittest discover -v ./tests/ -p test_unit_all_the_tests.py
... <Runs tests from that file>

i.e. when I first set this up, it appeared to be working, but only because I happen to have just one test file matching the glob here. It also will then fail to match any subdirectory-held test files that aren't also named test_unit_all_the_tests.py.

In bash, 'py_unit*.py' would mean "Don't expand this", but there's no way to communicate that to poethepoet as the quotes are already gone before it gets the argument.

[tool.poe.tasks]
run-unit-tests = "python -m unittest discover -v ./tests/ -p 'test_unit*.py'"

has the same problem, except it's always globbed from the top-level, not from the directory I happen to be in, leading to:

PS D:\PS\services\projects\scaler\tests> poetry run poe run-unit-tests -p "test_unit*.py"
Poe => python -m unittest discover -v ./tests/ -p -p test_unit_all_the_tests.py

where the same glob has expanded two different ways.

I'm not sure the 'auto-glob' behaviour makes sense, as there's literally no way for me to express "This is a glob, that isn't" through a cmd task, as it unconditionally tries to expand globs.

It's also a somewhat surprising behaviour, since if I wanted that, I'd use the shell runner, or e.g., a pwsh runner per #15, and let it do the globbing.

I don't see a particular justification (or documentation) for the "auto-glob" behaviour, so personally I would suggest removing it from cmd.py, along with the vestigates in shell.py. That would mean the rm -rf example in the Command Tasks list moves to shell, where it belongs, and is rewritten to use a bash glob, instead of a Python glob, i.e. there's no ** in bash.

If it's preferred to be kept, then having a way for the task to specify "No, do not auto-glob my commands" would be handy. I'd just use that all the time, so that poetry run poe X Y Z is literally poetry run <definition of X> Y Z as I would expect.

As another argument towards dropping auto-globbing, I've just realised this would make grep or sls nigh-unusable here, as a regexp on the command line will always be globbed-and-deleted, unless you're very unlucky and it globs into an existing filename, which would not have matched the regexp.

.env is a file that specifies the environment variables that should be loaded. AFIAK Poetry doesn't load these. Will poe do, though?

Hello there,

I'm trying to add something like this:

# pyproject.toml (lines 48-49)
[tool.poe.tasks]
get_random_secret_key = { script = "django.core.management.utils:get_random_secret_key()" }

The function however, returns a string and does not print its result. Would it be possible to have an option to print the return values of a given function?

Thank you.

Environment:

• poethepoet-0.10.0
• poetry-1.1.6

Problem using Poe on Windows.

[tool.poe.tasks]
test = "poetry build"

Expected:
No error.

Actual Result:

  FileNotFoundError

  [WinError 2] The system cannot find the file specified

  at C:\Python39\lib\subprocess.py:1420 in _execute_child
      1416│             sys.audit("subprocess.Popen", executable, args, cwd, env)
      1417│
      1418│             # Start the process
      1419│             try:
    → 1420│                 hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
      1421│                                          # no special security
      1422│                                          None, None,
      1423│                                          int(not close_fds),
      1424│                                          creationflags,

Non-portable workaround.

[tool.poe.tasks]
test = "poetry.bat build"

Not sure if it is a Poe problem, because Poetry has no exe file in distribution, and Popen will fail running just "poetry" from the PATH without bat extension.

 Directory of C:\Users\a_selivanov\.poetry\bin

21.04.2021  20:56    <DIR>          .
21.04.2021  20:56    <DIR>          ..
21.04.2021  20:56               475 poetry
21.04.2021  20:56                59 poetry.bat