umccr / cwl-ica-cli Goto Github PK

View Code? Open in Web Editor NEW

1.0 5.0 0.0 1000 KB

The CLI used for creating/syncing CWL workflows from GitHub to ICAv1

License: MIT License

Dockerfile 0.60% Shell 18.28% Python 81.12%

cwl-ica-cli's Introduction

cwl-ica-cli

The CLI used for creating/syncing CWL workflows from GitHub to ICAv1

Please see our wiki page to get started.

cwl-ica-cli's People

Contributors

Stargazers

Watchers

cwl-ica-cli's Issues

tsconfig json comment alignment is slightly off

The alignment of comments is slightly off in tsconfig json file:

{
    "compilerOptions": {
        "target": "es5",                                  /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
        "module": "commonjs",                                /* Specify what module code is generated. */
        "esModuleInterop": true,                             /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables "allowSyntheticDefaultImports" for type compatibility. */
        "forceConsistentCasingInFileNames": true,            /* Ensure that casing is correct in imports. */
        "strict": true,                                      /* Enable all strict type-checking options. */
    }
}

Resolve path prior to checking name-version path in get_name_version_tuple_from_cwl_file_path function

$ cwl-ica create-typescript-interface-from-cwl-schema --schema-path ../cwl-ica/schemas/bclconvert-run-configuration/2.0.0--4.0.3/bclconvert-run-configuration__2.0.0--4.0.3.yaml
Traceback (most recent call last):
  File "/home/alexiswl/mambaforge/envs/cwl-ica/bin/cwl-ica", line 13, in <module>
    sys.exit(main())
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/utils/cli.py", line 603, in main
    _dispatch()
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/utils/cli.py", line 581, in _dispatch
    create_typescript_interface_from_cwl_schema = CreateTypeScriptInterfaceFromCWLSchema(command_argv)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/subcommands/creators/create_typescript_from_schema.py", line 55, in __init__
    self.check_args()
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/subcommands/creators/create_typescript_from_schema.py", line 88, in check_args
    self.name, self.version = get_name_version_tuple_from_cwl_file_path(self.yaml_file_path, items_dir=get_schemas_dir())
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/utils/miscell.py", line 237, in get_name_version_tuple_from_cwl_file_path
    name_version_path = Path(cwl_file_path).absolute().relative_to(items_dir)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/pathlib.py", line 908, in relative_to
    raise ValueError("{!r} does not start with {!r}"
ValueError: '/e/Users/awluc/OneDrive/GitHub/UMCCR/cwl-ica-cli/../cwl-ica/schemas/bclconvert-run-configuration/2.0.0--4.0.3/bclconvert-run-configuration__2.0.0--4.0.3.yaml' does not start with '/e/Users/awluc/OneDrive/GitHub/UMCCR/cwl-ica/schemas

ilmn-tes template is wrong

Rather than ilmn-tes:resources:tier should be ilmn-tes:resoures/tier

Move create tools template to use full ica path name to prevent warnings

Related to:

Resources link is incorrect

https://support-docs.illumina.com/SW/ICA/Content/SW/ICA/RequestResources.htm should be https://illumina.gitbook.io/ica-v1/analysis/a-taskexecution#type-and-size

--ignore-workflow-id-mismatch missing from autocompletion for create-workflow-submission-template

Add function for assistance around yarn

Yarn can be annoying and confusing, causing errors if the top project yarn lock is not up-to-date with a typescript expression directory.

Proposed function assistance could be:

Upgrade yarn lock for cwl-ica repo (which just runs yarn install && yarn upgrade on CWL_ICA_REPO_PATH via conda run)
Clean typescript directory, which reinstalls new yarn lock, package json for a typescript expression directory

Convert "Manually adding in" as a debug log, not at an info level

Add autocompletion to typescript subcommands

Secondary File schema causing issues with yaml

cwl-ica create-tool-submission-template --launch-project development --prefix sambamba-slice --project development_workflows --tool-path /e/Users
/awluc/OneDrive/GitHub/UMCCR/cwl-ica/tools/sambamba-slice-and-index/0.8.0/sambamba-slice-and-index__0.8.0.cwl

Results in error

Traceback (most recent call last):
  File "/home/alexiswl/mambaforge/envs/cwl-ica/bin/cwl-ica", line 13, in <module>
    sys.exit(main())
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/utils/cli.py", line 556, in main
    _dispatch()
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/utils/cli.py", line 486, in _dispatch
    create_tool_template_obj()
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/subcommands/query/create_submission_template.py", line 108, in __call__
    self.write_yaml_file()
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/subcommands/query/create_submission_template.py", line 460, in write_yaml_file
    round_trip_dump(self.input_template_object, yaml_h,
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/main.py", line 1294, in round_trip_dump
    return dump_all(
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/main.py", line 1190, in dump_all
    dumper._representer.represent(data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 85, in represent
    node = self.represent_data(data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 112, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 1173, in represent_dict
    return self.represent_mapping(tag, data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 1018, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 112, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 1173, in represent_dict
    return self.represent_mapping(tag, data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 1018, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 112, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 1173, in represent_dict
    return self.represent_mapping(tag, data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 1018, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 112, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 350, in represent_list
    return self.represent_sequence(u'tag:yaml.org,2002:seq', data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 934, in represent_sequence
    node_item = self.represent_data(item)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 122, in represent_data
    node = self.yaml_representers[None](self, data)
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/site-packages/ruamel/yaml/representer.py", line 393, in represent_undefined
    raise RepresenterError('cannot represent an object: %s' % (data,))
ruamel.yaml.representer.RepresenterError: cannot represent an object: <cwl_utils.parser_v1_1.SecondaryFileSchema object at 0x7f97e6c54880>

cwl-ica nor cwltool checks for outputSource attribute for workflow outputs section

The outputSource can be left empty for any workflow output attribute and cwltool --validate will not inform the user that it's omitted.

Solution: add a check to cwl-ica workflow-validate subcommand that warns user if outputSource does not exist for a workflow

Autocompletion trigger is on PR review rather than push to main

Using -s on installation script causes exit rather than installation

Error for force requirement still says it's updating yaml

2022-09-20 16:55:36,505 - WARNING  - ica_workflow_version      - check_update_okay                        : LineNo. 266  - Cannot update workflow "wfl.bdc024d6eaea42d4ba0f17da380f8a51" version "4.0.3".  It has been modified elsewhere but the modification was not recorded in project.yaml file. Another user may be editing this workflow but has not pushed changes or you have not pulled the changes
2022-09-20 16:55:36,540 - WARNING  - ica_workflow_version      - sync_workflow_version                    : LineNo. 155  - Not updating workflow, definition has changed elsewhere: Showing differences in definitions:
{
    "type_changes": {
        "root['inputs'][9]['type']": {
            "old_type": "list",
            "new_type": "str",
            "old_value": [
                "null",
                "string"
            ],
            "new_value": "string"
        },
        "root['inputs'][17]['type']": {
            "old_type": "list",
            "new_type": "str",
            "old_value": [
                "null",
                "boolean"
            ],
            "new_value": "boolean"
        },
        "root['inputs'][18]['type']": {
            "old_type": "list",
            "new_type": "str",
            "old_value": [
                "null",
                "#samplesheet__2.0.0--4.0.3.yaml/samplesheet"
            ],
            "new_value": "#samplesheet__2.0.0--4.0.3.yaml/samplesheet"
        },
        "root['inputs'][21]['type']": {
            "old_type": "list",
            "new_type": "str",
            "old_value": [
                "null",
                "string"
            ],
            "new_value": "string"
        }
    }
}
2022-09-20 16:55:36,540 - INFO     - sync                      - __call__                                 : LineNo. 129  - Updating project yaml

Solution is to instead throw error after warning

Hardcode yarn version to 3.2.3

Prevents surprise / unwanted changes in new versions
Prevents any merge conflicts that might arise

cwl-ica icav2-launch-pipeline-analysis fails if --create-cwl-analysis-json-output-path not set

Query submission Template requires module from bin directory

Traceback (most recent call last):
  File "/home/alexiswl/mambaforge/envs/cwl-ica/bin/cwl-ica", line 13, in <module>
    sys.exit(main())
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/utils/cli.py", line 642, in main
    _dispatch()
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/utils/cli.py", line 507, in _dispatch
    from subcommands.query.create_workflow_submission_template import CreateWorkflowSubmissionTemplate
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/subcommands/query/create_workflow_submission_template.py", line 8, in <module>
    from subcommands.query.create_submission_template import CreateSubmissionTemplate
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/subcommands/query/create_submission_template.py", line 22, in <module>
    from utils.input_template_utils import create_input_dict
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/utils/input_template_utils.py", line 11, in <module>
    from bin.create_typescript_interface_from_schema import create_packed_schema_json
ModuleNotFoundError: No module named 'bin'

Token recreation shows error message despite successful recreation and completed process

$ cwl-ica workflow-sync --workflow-path workflows/dragen-wts-qc-pipeline/3.9.3/dragen-wts-qc-pipeline__3.9.3.cwl --projects development_workflows
2022-09-14 14:44:44,489 - INFO     - cwl                       - import_cwl_yaml                          : LineNo. 131  - Manually adding in "../../../schemas/fastq-list-row/1.0.0/fastq-list-row__1.0.0.yaml" into namespaces before we import the cwl object
2022-09-14 14:44:57,161 - INFO     - sync                      - __call__                                 : LineNo. 107  - Getting new "workflow" version CWL object
2022-09-14 14:44:57,161 - INFO     - sync                      - __call__                                 : LineNo. 111  - Updating "workflow" yaml.
2022-09-14 14:44:57,203 - INFO     - sync                      - __call__                                 : LineNo. 116  - Updating project definitions on ICA
2022-09-14 14:44:57,203 - INFO     - sync                      - __call__                                 : LineNo. 118  - Updating project "development_workflows"
2022-09-14 14:44:57,204 - ERROR    - ica_utils                 - check_token_expiry                       : LineNo. 134  - This token has expired, please update your token with cwl-ica update-project
2022-09-14 14:44:57,204 - WARNING  - project                   - get_project_token                        : LineNo. 129  - Project access token has expired - attempting to create a new one
2022-09-14 14:45:05,454 - INFO     - sync                      - __call__                                 : LineNo. 129  - Updating project yaml

Error line should just be a debug, also cwl-ica update-project does not exist as a command.

Use tags to generate workflow releases

Tool version specification

With the current cli v1.4.1, we get an error when specifying versions for tools with alphabetic suffix such as STAR https://quay.io/repository/biocontainers/star?tab=tags.

$ cwl-ica create-tool-from-template --tool-name star --tool-version 2.7.10b
2023-01-06 08:19:45,683 - ERROR    - create_from_template      - check_shlex_version_arg                  : LineNo. 124  - Was not able to parse 2.7.10b as a version for parameter --tool-version
Traceback (most recent call last):
  File "/Users/kanwals/miniconda/envs/cwl-ica/bin/cwl-ica", line 13, in <module>
    sys.exit(main())
  File "/Users/kanwals/miniconda/envs/cwl-ica/lib/python3.10/utils/cli.py", line 649, in main
    _dispatch()
  File "/Users/kanwals/miniconda/envs/cwl-ica/lib/python3.10/utils/cli.py", line 305, in _dispatch
    create_tool_obj = CreateToolFromTemplate(command_argv)
  File "/Users/kanwals/miniconda/envs/cwl-ica/lib/python3.10/subcommands/creators/create_tool_from_template.py", line 76, in __init__
    self.check_args()
  File "/Users/kanwals/miniconda/envs/cwl-ica/lib/python3.10/subcommands/creators/create_tool_from_template.py", line 104, in check_args
    self.check_shlex_version_arg("--tool-version", tool_version_arg)
  File "/Users/kanwals/miniconda/envs/cwl-ica/lib/python3.10/subcommands/creators/create_from_template.py", line 128, in check_shlex_version_arg
    raise InvalidVersionError
utils.errors.InvalidVersionError

Adding support for this use case will be useful, IMO.

nested import breaks create-submission-from-template command

$  cwl-ica create-tool-submission-template --launch-project
development --prefix bclconvert-4.0.3-test --project development_workflows --tool-path tools/bcl-convert/4.0.3/bcl-convert__4.0.3.cwl

2022-09-20 15:26:49,715 - WARNING  - input_template_utils      - get_commented_map_for_schema_input_object : LineNo. 226  - Don't know what this is. type has keys <ruamel.yaml.comments.CommentedMapKeysView object at 0x7f45773842b0>
ordereddict([('$import', '../../../schemas/samplesheet-header/2.0.0/samplesheet-header__2.0.0.yaml#samplesheet-header')])
Traceback (most recent call last):
  File "/home/alexiswl/miniforge3/envs/cwl-ica/bin/cwl-ica", line 13, in <module>
    sys.exit(main())
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.8/utils/cli.py", line 596, in main
    _dispatch()
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.8/utils/cli.py", line 496, in _dispatch
    create_tool_template_obj()
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.8/subcommands/query/create_submission_template.py", line 108, in __call__
    self.write_yaml_file()
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.8/subcommands/query/create_submission_template.py", line 455, in write_yaml_file
    self.set_cwl_inputs_as_commented_map()
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.8/subcommands/query/create_submission_template.py", line 442, in set_cwl_inputs_as_commented_map
    self.input_template_object["input"] = create_input_dict(self.cwl_inputs)
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.8/utils/input_template_utils.py", line 63, in create_input_dict
    input_dict = get_commented_map_for_schema_input_object(input_dict,
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.8/utils/input_template_utils.py", line 230, in get_commented_map_for_schema_input_object
    input_dict[input_id] = get_commented_map_for_schema_input_object(input_dict[input_id],
  File "/home/alexiswl/miniforge3/envs/cwl-ica/lib/python3.8/utils/input_template_utils.py", line 202, in get_commented_map_for_schema_input_object
    for field_name in input_object.keys():
AttributeError: 'NoneType' object has no attribute 'keys'

Name clashes causing issues with autocompletion

$ cwl-ica append-typescript-directory-to-cwl-expression-tool --expression-path expressions/get-file-from-directory/1.Traceback (most recent call last):
  File "<stdin>", line 62, in <module>
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/pathlib.py", line 908, in relative_to
    raise ValueError("{!r} does not start with {!r}"
ValueError: 'get-file-from-directory-with-regex/1.0.0/get-file-from-directory-with-regex__1.0.0.cwl' does not start with 'get-file-from-directory'
0.^C

Enum keys in schema need to be quoted when placed in their own TS interface

Catalogue creation has some funny outputs with latest cwlutils

Create inputsjson template and overrides template in markdown document

Schema path autocompletion issue

$ cwl-ica create-typescript-interface-from-cwl-schema --schema-path schemas/samplesheet/Traceback (most recent call last):
  File "<stdin>", line 59, in <module>
  File "/home/alexiswl/mambaforge/envs/cwl-ica/lib/python3.8/pathlib.py", line 908, in relative_to
    raise ValueError("{!r} does not start with {!r}"
ValueError: 'samplesheet-header/1.0.0/samplesheet-header__1.0.0.yaml' does not start with 'samplesheet'

has_conda uses stdout from 'type' command rather than from jq command

cwl-ica-cli container build is failing around yarn npm installation

Yarn / Npm installation may not be required (immediately) for cwl-ica-cli container so might be best to just leave it for now by adding in a parameter that can skip this step

Resources guide in template provides invalid link

# ILMN Resources Guide: https://support-docs.illumina.com/SW/ICA/Content/SW/ICA/RequestResources.htm

should be instead

https://illumina.gitbook.io/ica-v1/analysis/a-taskexecution#type-and-size

Package json warning

warning package.json: No license field

Old version of cwltool is hindering development

In the same way we used a workaround to use the latest version of cwltool to create the visual image, I wonder if there's a way if we can instead ONLY use the two-year-old cwltool version of 3.0.20201203173111 for cwltool validate (and only in the context of ica v1 deployment)

GH Action, add function to automatically run test suite for typescript expressions

cwltool --print-dot failing for more complex workflows

Solution may be to use a later version of cwltool specifically for the graph generation.

We can do this by creating a virtual python environment specifically for this step.

Autocompletion for TS

The auto complete doesn’t work for me when using cwl-ica append-typescript* command using v1.2.5

Autocompletion for miscellaneous files is slowing things down

Using find $PWD -name '*.json' isn't the best method for finding json files since it does a full search even if the json file is in the top directory.

If a user is trying to find a json in a relative directory, like ../launch_jsons/my-launch.json this is also not helpful.

Consider something like find ${CURRENT_WORD-}* -maxdepth 1 -type f -name '*.json' -o type d

Allow icav2-* steps to not require --project-id / --project-name parameter

Use env vars instead (ICAV2_PROJECT_ID) or pull from ~/.icav2/.session.ica.yaml

Useful for the following subcommands

icav2-launch-pipeline-analysis
icav2-list-analysis-steps
icav2-get-analysis-step-logs

round_trip_dump in new version of ruamel doesn't preserve quotes

After upgrading cwltool, we will also need to upgrade usage of ruamel.yaml to cater for new syntax

Wrong autocomplete for add-workflow-to-project for --project parameter

Check-docs shows error when key is not of [a-z0-9_] but should just be warning

Feature request, raise warning if workflow step name is more than 21 chars

Related to

Add options to have prefix in workflow-submission-templates for runs with TEMPLATE fillers

Rather than leave workDirectory and outputDirectory blank, we should allow for the following parameters to assist users to create unique workDirectory and outputDirectory locations per workflow run instance

An idea would be allow users to set one or more of the following parameters on the cli

--run-prefix
- Would set workDirectory to /path/to/run/prefix/work/__DATE__
- Would set outputDirectory to /path/to/run/prefix/output/__DATE__
--output-directory
- Set outputDirectory directly to /path/to/output/directory/
--output-directory-prefix
- Set outputDirectory to /path/to/output/directory/__DATE__
--work-directory
- Set workDirectory to /path/to/work/directory
--work-directory-prefix
- Set workDirectory to /path/to/work/directory/__DATE__

Yarn installation warning

yarn install v1.22.19
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.

get-workflow-step-ids errors with automation for --workflow-path attribute

Use global handler for virtual pyenv, packing, validating can take a very long time

xtrace option should be `-o xtrace` not `-oxtrace`

get-workflow-step-ids errors with accessing workflow from inside non-cwl directory

Migrate to python 3.10

Python 3.10 allows for unions when running isinstance, very helpful in the following settings

from cwl_utils.parser.cwl_v1_0 import \
    ProcessRequirement as ProcessRequirement_v1_0
from cwl_utils.parser.cwl_v1_1 import \
    ProcessRequirement as ProcessRequirement_v1_1
from cwl_utils.parser.cwl_v1_2 import \
    ProcessRequirement as ProcessRequirement_v1_2

ProcessRequirement = Union[
    ProcessRequirement_v1_0,
    ProcessRequirement_v1_1,
    ProcessRequirement_v1_2
]


def get_process_requirement(cwl_obj) -> Optional[ProcessRequirement]:
    try:
        next(
            filter(
                lambda requirement: isinstance(requirement, ProcessRequirement)
            )
        )
    except StopIteration:
        return None