fabienarcellier / alfred-cli Goto Github PK

Importing a module that contains commands can fail, for example, when a syntax error is present in the code.

The commands of the other modules must remain accessible in alfred. When loading commands, alfred displays an error and remains usable.

import of command module alfred/... failed. {exception text}

alfred.option is a pass-through that instantiates an option on click. Click offers a relevant set of arguments in the context of Alfred. It is necessary to summary the arguments in documentation of alfred. It is also essential to indicate this heritage to click.

• https://click.palletsprojects.com/en/8.1.x/api/#click.Option

Discarded alternative

• alfred.option implements a subset of the optional click parameters: alfred.option implements only a subset of the parameters tested in the click CI. This model ensures that the parameters are fully supported by alfred. This solution restricts the possible options and the click bugs. I feel like the click arguments mostly matter.

alfred juggles with python environments and environment variables. To be able to ensure a minimum of support, it is important to have information on the environment in which alfred is executed.

alfred --inspect

The order of the keys must be preserved in the json. The indentation is 2 spaces.

[
    {
        "project_directory": "./",
        "manifest": "./.alfred.toml",
        "name": "name",
        "description": "my description",
        "subprojects": [
            "./products/*"
        ],
        "project": {
            "command": [
                "alfred/*.py"
            ],
            "python_path_project_root": false,
            "python_path_extends": [],
            "venv": null,
            "venv_dotvenv_ignore": true
        },
        "runtime": {
            "executable": "",
            "pythonpath": [],
            "path": []
        }
    },
    {
        "project_directory": "./product/product1",
        "manifest": "./product/product1/.alfred.toml",
        "name": "name",
        "description": "my description",
        "subprojects": [
            "./products/*"
        ],
        "project": {
            "command": [
                "alfred/*.py"
            ],
            "python_path_project_root": false,
            "python_path_extends": [],
            "venv": null,
            "venv_dotvenv_ignore": true
        },
        "runtime": {
            "executable": "",
            "pythonpath": [],
            "path": []
        }
    },
]

The working folder when an alfred command is launched is always the root of the project where the .alfred.toml manifest is located. The developer can change the folder without composing path from the alfred.project_directory function.

For example, if he wants the command to run in the frontend folder, he just uses os.chwd('frontend').

• implement the function alfred.execution_directory returns the folder in which a user invoked the alfred command.
• documenter la fonction alfred.execution_directory in Command toolkit

This command throws an exception. ModuleNotFoundError: No module named 'sqlite3'

@alfred.command("test")
def test():
    import sqlite3
    print(sqlite3.sqlite_version)

Libraries like sqlite3 and optparse cannot be imported at runtime but math library can be imported

This code, however, works.

import sqlite3

@alfred.command("test")
def test():
    print(sqlite3.sqlite_version)

work around

The management of the path of the libraries is wrong in the code of alfred. Some paths are erased when invoking a command. A work around is to manually add the missing paths in the python_path_extends parameter.

[alfred.project]
python_path_extends = ["/usr/lib/python3.10", '/usr/lib/python3.10/lib-dynload']

These 2 paths are visible in the interpreter info

import sys
print(sys.path)

badges summarize the following information:

• the current version on pypi
• Python versions supported
• continuous integration status
• documentation status
• the link to the discord server
• the project license

Inspiration : https://github.com/FabienArcellier/qjoin

The stdout flag allows you to control whether the alfred.run command displays its information on a terminal.

_, stdout_content, _ = alfred.run("echo hello world", print_stdout=False)

• add the flag print_stdout
• add the flag print_stderr
• document both flag in api section of documentation

Alfred's debug logs are displayed when a command is invoked in a third-party interpreter.
The --debug flag should be passed in the invocation of the subinterpreter.

alfred.run execute a text command without prior preparation. The first element in text command should be the executable.

alfred.run("pytest test")
alfred.run("mv /var/yolo /var/yolo1", exit_on_error=False)

alfred text commands are more intuitive to implement.

The instruction is analyzed with shlek. We raise error on command that contains statement as '&', '|', '<', '>','>>','<<'

alfred detects the presence of a virtual environment .venv at the root of a project. It uses the python of this virtual environment without prior configuration.

If an environment is specified in the manifest and a .venv virtual environment is also detected, the python of the virtual environment specified in the manifest is used. If the specified virtual environment is invalid, an exception is thrown.

A venv_dotvenv_ignore parameter in the alfred.toml manifest allows to disable the automatic detection of the virtual environment at the root. By default, this parameter is false.

The python_path_extends option adds folders in the PYTHONPATH environment variable to make module resolution work from those folders.

[alfred.project]
pythonpath_extends = [ "tests" ]

In a python project, the fixtures package is not installed in the virtual environment. These modules are inaccessible for other modules, in unit tests for example. Adding a folder to the PYTHONPATH will cause python to use that folder in these dependency resolution attempts. from fixtures import alfred_fixture will be usable.

└── tests
    ├── acceptances
    ├── fixtures
    │   ├── ...
    │   ├── __init__.py
    │   └── alfred_fixture.py
    ├── __init__.py
    ├── integrations
    ├── __pycache__
    ├── spikes
    └── units

Behaviors

• relative paths are transformed into absolute folder from the manifest's parent folder
• absolute paths are directly added to the pythonpath
• the pythonpath separator depends on the os (using os.pathsep)
• paths are added at the end of the pythonpath

Adding a folder in the pythonpath variable allows you to expose packages without declaring them in the manifest.

This pattern is useful with poetry to be able to reuse the code of the package tests in this one for example.

The alfred.pythonpath decorator adds the project root. You can save specific folders here.

@alfred.command('ci', help="execute continuous integration process of alfred")
@alfred.pythonpath()
def ci():
    with alfred.env(SCREEN="display"):
        bash = alfred.sh("bash")
        alfred.run(bash, ["-c" "echo $SCREEN"])

@alfred.command('ci', help="execute continuous integration process of alfred")
@alfred.pythonpath(['tests'], append_root=False)
def ci():
    with alfred.env(SCREEN="display"):
        bash = alfred.sh("bash")
        alfred.run(bash, ["-c", "echo $SCREEN"])

The .alfred.yml file is a manifest present at the root of an alfred project. It's obsolte now. It describes the project settings. The configuration of tools in python is evolving and more and more tools offer integration with the pyproject.toml file.

The first step for alfred is to migrate the .alfred.yml manifest to the toml format. At the same time, it is about making its structure more relevant. The previous manifest was structured for example to overcome the limitations of pipenv in the case of a monorepo.

Here is the new format of the .alfred.toml file.

[alfred]
name = "fixtup" #optional
description = "str" #optional
subprojects = [ "product/*", "lib/*" ] #optional

[alfred.project]
command = [ "alfred/*.py" ]
env = { }
python_path_project_root = true
python_path_extends = []
venv = "src/.."

• expressions like alfred/*.py supports recursive wildcards like `alfred/**/*.py
• a warning message should be raised if alfred encounters an old-format manifest .alfred.yml

A developer can use a command through the shell directly.

alfred.run("mypy src/* > mypy.log", shell=True)

alfred --help will raise an exception TypeError: expected str, bytes or os.PathLike object, not NoneType on a project that haven't been initialized.

Expected behavior

A message show an error message and recommand an action to the user to initialize its project.

not an alfred project (or any of the parent directories), you should run alfred init command

the alfred.flag decorator declares a flag on an alfred command. This decorator allows you to specify all-or-nothing options.

Usage

A flag allows a developer to play a subpart of a command as with alfred ci --frontend. This flag allows running continuous integration only on the code of the frontend part of a project.

Implementation

@alfred.command("ci")
@alfred.flag("--frontend")
def continuous_integration(frontend: bool):
  pass

Short form of a flag

@alfred.command("ci")
@alfred.flag("--frontend", "-f")
def continuous_integration(frontend: bool):
  pass

Flag in negative logic

@alfred.command("ci")
@alfred.flag("--ignore-windows-tasks", default=True)
def continuous_integration(ignore_windows_tasks: bool):
  pass

Discarded alternative

a flag as a specific optional argument : click considers a flag to be a particular optional argument. this model makes full sense for click which is a toolkit for developing a command line. The counterpart is that this approach requires richer documentation. I regularly search the documentation for how to write a flag. 80% of the options argument I use with alfred are simple flags. I prefer to think of a flag as a dedicated instruction.

alfred --new trigger a wizard to create a new command. The wizard guides the developper in the process.

alfred --new

What command do you want to create ? 
$ ci

What does the `ci` command do [optional] ?
$

In which module do you want to create the `ci` command?
$ ci

Do you want to create the following `ci` command in `alfred/ci.py` [y, n]?

>>> @alfred.command('ci', help="")
>>> def ci():
>>> 	alfred.run("echo 'ci command not implemented yet'")

Documentation is loaded from the docs folder by read the docs.

This documentation is titled alfred and uses the template sphinx_rtd_theme

A developer can launch a command which depends on a virtual environment other than the one he has activated in his terminal. In this case, alfred switch the virtual environment by itself but lost interactivity in the operation. Color and rich prompt are note supported anymore.

During this call, there is no need to capture standard output and error output. By ignoring the standard and error outputs, we can recover interactivity.

alfred tests

For commands invoked in a command, we keep the current mode to make the standard output and error output accessible at the end of the command.

@alfred.command('ci', help="execute continuous integration process of alfred")
@alfred.option('-v', '--verbose', is_flag=True)
def ci(verbose: bool):
    alfred.invoke_command('alfred_check')
    if alfred.is_posix():
        alfred.invoke_command('lint', verbose=verbose)
    else:
        print("linter is not supported on non posix platform as windows")

    alfred.invoke_command('tests', verbose=verbose)

the alfred --check command checked that all commands are loaded. When calling, it loads all command modules. Those with errors are displayed.

command module "/home/{...}/alfred/ci.py" is not valid.
 SyntaxError : expected ':' (ci.py, line 6)

If at least one of the command modules is in error, the command exits with code 1 to trigger an error in the continuous integration.

Use alfred --check in alfred command

The alfred --check command has its place in a continuous integration workflow. It checks that the commands implemented in alfred are all usable. An user can use this check as a subcommand with the alfred.invoke_itself(check=True) function

@alfred.command('ci', help="execute continuous integration process of alfred")
@alfred.option('-v', '--verbose', is_flag=True)
def ci(verbose: bool):
    alfred.invoke_itself(check=True)
    alfred.invoke_command('lint', verbose=verbose)
    alfred.invoke_command('tests', verbose=verbose)

Expected documentation

The alfred --autocomplete command shows the user the shell statement to execute to enable autocomplete.

The shell supported are the one from click documentation :

• bash
• zsh
• fish

The command displays after instructions to permanently configure autocompletion.

execute the following statement to activate autocompletion for alfred on bash (shell we have detected)

>>> eval "$(_ALFRED_COMPLETE=bash_source alfred)"

Adding this statement to your shell profile installs alfred's autocompletion permanently. 
You will avoid running this command every time you open a terminal window.

The profile file for bash is ~/.bashrc

The alfred init command works if the alfred.toml manifest is missing, even if the folder that contains the commands already exists.

The contextmanager alfred.parallel allow a developper to run multiple commands simulatneously and in a single terminal. It's useful to run fullstack apps, for example to run npm, fastapi and a database in same time.

When the user interrupts a command with CTRL+C, all commands that are currently executing are interrupted. If it is another signal as sigquit, it is forwarded to the commands being executed by alfred.

Log messages are prefixed with the command name in square brackets like in docker-compose to be able to track what is happening.

[front] 2022-12-01 12:56:54 hello world

usage

The developer declare the commands to be executed simultaneously in an alfred.parallel context block. Each of the commands will run independently of each other.

with alfred.parallel() as p:
  p.run(npm, ["run"], name="front")
  p.run(python, ["run"], name="back")

The commands are waited for indefinitely before exiting the alfred.parallel block. It is possible to disable this behavior and explicitly check the end of a command by setting the attribute wait to False. The instruction wait allow to wait the end of a command explicitely.

with alfred.parallel(wait=False) as p:
  p.run(npm, ["run"], name="front")
  p.run(python, ["run"], name="back")
  p.wait(commands=['front', 'back'])

alfred commands can be invoked in an alfred.parallel block.

with alfred.parallel() as p:
  p.invoke_command("frontend:run", name="front")
  p.invoke_command("backend:run", args={'verbose': verbose}, "back")

alfred.CMD_RUNNING is a macro to avoid loading huge dependencies while listing commands.

import alfred

if alfred.CMD_RUNNING():
    import torch
    

@alfred.command("predict")
def predict():
    example_list = [1,2,3]
    x = torch.tensor(example_list)
    print(x)

documentation

• write the section advanced > accelerate commands discovery

In the case of a cross-platform build, it is necessary to execute a different workflow between linux, windows and macos. For example, github action is limited on macOS and windows in using docker container. Excluding tests that use docker on both platform is a valid strategy if you can't host your own CI runner. That's the cas in FabienArcellier/fixtup.

the alfred package provides out-of-the-box functions to verify the os

alfred.is_posix()
alfred.is_linux()
alfred.is_macos()
alfred.is_windows()

I want to replace the use of Pipenv with Poetry to manage my project's dependencies, to take advantage of Poetry's package and dependency management benefits.

alfred's autocompletion should work with 2 dots as command separator as alfred backend:tests.

A subcommand executed in Windows is not streamed. The standard output is displayed intermittently at the end of each invocation.

fix in #56

A confirmation requested with click.prompt is visited in a command which is executed by an interpreter launched by alfred.

@alfred.command("publish")
def publish():
  value = click.prompt("Confirm", type=Choice(['y', 'n']), show_choices=True, default='n')
  print(value)

programs like mypy and pytest, installed in the virtual environment of the project, are not resolved by the functionalfred.sh.

mypy = alfred.sh("mypy", "mypy should be present")

Design

When alfred identifies that he must play a command in a virtual environment different from the one used, the path to the binaries of this virtual environment is added to the PATH

Discarded alternative

• enable virtual environment at OS level : this method is complex to implement because it uses the shell. I have seen strange behaviors when a virtual environment is activated in another virtual environment.

the autocompletion fails on the commande alfred myproject:docker:run. It will propose run after myproject:

Autocompletion does not offer commands that contain :.

The content of a command like streamsync is buffered. The end of the command is missing. The server address and port remains invisible.

export PYTHONUNBUFFERED=1 make them visible.

Error output is hidden when a command fails in another interpreter.

• ignore this workflow when venv_poetry_ignore is set
• detect the presence of poetry inside the pyproject.toml
• detect the presence of poetry executable
• display warning if poetry is in pyproject.toml and poetry executable is missing

Poetry manages this project. Poetry is missing from your system.
https://python-poetry.org/docs/#installation

• detect the venv using poetry

When pyenv is used, poetry cannot identify the virtual environment attached to the project with the poetry env info command. It manages to list virtual environments and activate its shell.

Alfred cannot change environments because he does not have a reliable way to identify the currently installed virtual environment.

2024-02-13 20:04:18,652 WARNING - Poetry virtual environment is missing. You should run poetry install. [poetry.py:39]

Running any command on an uninitialized project should show an error message, but not an exception :

not an alfred project (or any of the parent directories), you should run alfred init

Currently alfred --new echo test displays a complete exception:

Traceback (most recent call last):
  File "/home/far/.local/bin/alfred", line 8, in <module>
    sys.exit(cli())
  File "/home/far/.local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/far/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/far/documents/projects/20210821_1530__alfred-cli/src/alfred/cli.py", line 118, in invoke
    self_command.new(fullarg)
  File "/home/far/documents/projects/20210821_1530__alfred-cli/src/alfred/self_command.py", line 31, in new
    project_dir = manifest.lookup_project_dir()
  File "/home/far/documents/projects/20210821_1530__alfred-cli/src/alfred/manifest.py", line 67, in lookup_project_dir
    raise NotInitialized("not an alfred project (or any of the parent directories), you should run alfred init")
alfred.exceptions.NotInitialized: not an alfred project (or any of the parent directories), you should run alfred init

Nodejs installs utilities like jest in the node_modules folder. These utilities are in the node_modules/.bin folder. To use them directly, a developer must insert them into their PATH.

[alfred]

[alfred.project]
path = [] # optional

Adding elements to the PATH from an Alfred manifest allows you to use the commands directly.

import alfred

@alfred.command('frontend:test', help="tests the frontend")
def tests():
    jest = alfred.sh('jest', "jest is not installed in your system")
    alfred.run(jest)

definition of done

• implement path support in _context_middleware
• document path on section Project

@alfred.command("cleandir")
def cleandir():
  with alfred.chdir('/tmp'):
    # clean the tmp directory

the working directory returns to its previous value when leaving the block.

Replacing text transcription errors could prevent errors due to poor codec choice.

On certain environments, alfred triggers this error. I removed the dependency which was not essential.

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
Traceback (most recent call last):
  File "/home/debian/.pyenv/versions/3.10.4/lib/python3.10/io.py", line 54, in <module>
ImportError: cannot import name 'text_encoding' from 'io' (unknown location)

ae478d2