agnostiqhq / covalent-aws-plugins Goto Github PK

Acceptance Criteria

• Implement AWSExecutor class as per Design Details
• Add docstring for AWSExecutor class
• Add unit tests for AWSExecutor class

Design Details (AWSExecutor class)

The AWSExecutor is a type of remote executor, so it inherits from RemoteExecutor. This class contains authentication handling.

• Class attributes:
  • region — Region in which the task will run
  • s3_bucket_name — An alias for the super-class’s remote_cache
  • profile — IAM profile used to run the task
  • execution_role — Name of an IAM role assumed by a task (optional)
  • log_group_name — Name of the log group in Amazon CloudWatch (optional)
• Class methods:
  • _validate_credentials — Validate user credentials (sometimes called _get_aws_account currently) using aws sts get-caller-identity
• Current plugins which can inherit from this class: EC2, ECS, Batch, Lambda, Braket [Note: EC2 inherits from both AWSExecutor and also from SSHExecutor]

In the end, we want to put this class, as well as all AWS plugins, inside a repository called covalent-aws-plugins (i.e. the AWS plugin monorepo). Each plugin should remain individually installable as well.

Acceptance criteria:

• Refer to the fact that now all plugins are included when installing covalent-aws-plugins
• Add introduction copy on what this module does
• Create table that refers to each executor and it's use case
• Add any warnings on external binaries / dependencies needed for all executors to work
• Add a concise example of how the executors are imported and used in electrons.

Acceptance criteria:

• Using a docker environment, pip install covalent-aws-plugins.
• In a python file, check that with only this pip install, you're able to import the following aws executors: batch, ecs, lambda, braket, ec2
• Leave a comment confirming that the bug was reproduced (or not).
• Copy paste the docker file and python script for future use in this issue.

Acceptance criteria:

• Find fix for packaging bug via git submodule tool: https://git-scm.com/book/en/v2/Git-Tools-Submodules
• 2 independent validation that the fix resolves the packaging bug.
• One thing to keep in mind is that when we install covalent-aws-batch for example, it requires covalent-aws-plugins and should NOT install each of the other executors.
• While I won't block the pushing of this fix due to a lack of testing, write down / scope out the addition of a test to ensure that the packaging is working as intended.

Backend logic:

# covalent-awsbatch-plugin/requirements.txt
covalent-aws-plugins[batch]==[version_num]

In covalent-aws-plugins's extras_require={"batch": "covalent-awsbatch-plugin==ver"}

• Validating credentials when an instance of AWSExecutor is created results in a few problems when interacting with covalent self hosted
• If the AWS_PROFILE env variable is set, the client side configuration file gets rendered with profile=<AWS_PROFILE> value. When a user then dispatches a workflow to self-hosted the executor will get populated with the profile information parsed from the configuration file which eventually will cause an error on the self-hosted side as there will be no values for the profile.
• A fix would be to not run credential validation on executor instantiation and rather let boto3 session calls fail downstream if the credentials are expired

Acceptance Criteria

• Ensure tests workflow is passing in develop

Refer to this run for details on error:
https://github.com/AgnostiqHQ/covalent-aws-plugins/actions/runs/3991490209

The version pin on boto3 is incompatible with the braket extras in main Covalent, which requires boto3>=1.28.5.

There are currently no tests for the exec.py file. Since it's a crucial file for this plugin and almost all the AWS plugins rely on it, we should have tests, and subsequently code coverage for it.

Acceptance Criteria

• Move Lambda base executor deployment workflow from core covalent to lambda repo
• Ensure it can be triggered manually or callable from another workflow

Input variables should include:

1. Tag/version of lambda executor to use
2. Version of covalent (to resolve tag for covalent base executor, and also determines which tag to produce for lambda executor (covalent-MAJOR_VERSION))
3. Is Prerelease flag