Babylon azure storage container create command do not support QueryType type

Expected behavior

Argument CONTAINER_NAME should accept QueryType values like : "%deploy%organization_id"

Following a conversation with @johnfolly, we are raising a question about moving the user config folder to the working directory.
I know you put a lot of work and thought on this.

Pros :

• All data relating to a project is in the same location
• Easier to see which config is used for this specific project

Cons :

• Can contain redundant information
• Config will need to be updated for each project if babylon has new requirements (retro-compatibility is thus critical)

Another option is to think about moving keys in a local-settings.json, allowing to override the default user config ?

Needed config keys should be checked before any command is run

API File Output Format

Current Behavior

The following line in the API get commands converts a snake case formatted map into a camel case token map:

converted_content = convert_keys_case(retrieved_solution, underscore_to_camel)

However, it is not working properly, as some tokens are converted while others are not.

Expected Behavior

All snake case formatted maps should be properly converted into camel case formatted maps.

Arguments should be available for macros to run only certain steps

Missing terraform cloud variable default category

Current behavior

Argument VAR_CATEGORY in the command babylon terraform-cloud workspace vars create [OPTIONS] VAR_KEY VAR_VALUE VAR_DESCRIPTION {terraform|env} do not have default value.

Expected behavior

• Argument VAR_CATEGORY should have as default value terraform (the most used category)
• this parameter can be parsed to the command as option and have default value

babylon terraform-cloud workspace vars create -t env VAR_KEY VAR_VALUE VAR_DESCRIPTION

One option is to add link to documentation for each API call to know which rights are needed

Format api scope and resolve suitable value for babylon

@jblanc-cosmotech suggested to auto format api scope from legacy format to a suitable value for babylon api command group.

Current Behavior

• api_scope is a string value in platform config file
• babylon use this value as it is and to not apply any formatting
• common api scopes have /something suffix and cosmotech-api lib do not accept this format

Expected behavior

• format api scope with a function ( or lambda), this function should be parsed to the require_platform_key decorator

@require_platform_key("api_scope", "api_scope", formatter=api_scope_formatter)
```

Deprecated terraform cloud Variables API

Message from hashicorp : The Variables API is deprecated and will be removed in a future release. All existing integrations with this API should transition to the Workspace Variables API.

See : https://developer.hashicorp.com/terraform/cloud-docs/api-docs/variables

We are currently using the terraform cloud Variables API to create terraform cloud Variables through Babylon in the babylon terraform-cloud workspace vars create command.

Suggestion :

    try:
        r = api.workspace_vars.create(workspace_id=workspace_id, payload=var_payload)
    except TFCHTTPUnprocessableEntity as _error:

Screenshots

Expected behavior

try:
        r = api.workspace_vars.create(workspace_id=workspace_id, payload=var_payload)
    except TFCHTTPUnprocessableEntity as _error:

Macro commands can take an input file that sets values initially in the datastore

I think we could load the configuration files only once and store it in the environment or datastore.
It will decrease execution time and probably help with data consistency.

Extend QueryType usage to terraform cloud workspace var value argument

Current behavior

Argument VAR_VALUE in the command babylon terraform-cloud workspace vars create [OPTIONS] VAR_KEY VAR_VALUE VAR_DESCRIPTION {terraform|env} only accept TEXT like values

@argument("var_value") 

Expected behavior

Argument VAR_VALUE should accept QueryType values like : "%platform%resource_group_name"

Adx Cluster name key in platform config is not explicit

Adx cluster name and database name key could be named:

# adx_cluster_name: The name of the Azure Data Explorer Cluster
adx_cluster_name: ""
# adx_database_name : the name of the database used in Azure Data Explorer
adx_database_name: ""

Got traceback after No value found Error for babylon config key

Description of current Behavior

When I tried to invoke a Babylon command which required a missing value from deployment or platform config I got a proper Error log with the error message and below this message I also got an error Traceback.

Screenshots

Steps to Reproduce

• Install Babylon from main brach
• Run an api command without setup the api scope value in config/deployments/deploy.yaml

# Example
babylon api organization get-all

Expected behavior

• When babylon encountered a missing value from only the error message should be displayed.
• The error message should be more explicit, I suggest :

No value provided for the required key api_scope in platform config file.
api commands won't run without it.
Edit ~/babylon_config_path/config/platforms/platform.yaml

babylon azure adx database create
babylon azure adx database delete
babylon azure adx database get-all
babylon azure adx database get

@lalepee

Test:
babylon azure appinsight delete testNibaldo
Result:
App insight testNibaldo has been successfully deleted.

but testNibaldo does not exist.

StaticWebApp deployment fails silently when github branch has not been set.
I think we should go to artifact based deployment, but it implies another way to set variables...

Get-all methods return a lot of data.
We should make use of jmespath and add a decorator that adds a filter feature of the form

babylon api organization get-all -k [name=coucou]

Could you investigate this issue and try to remove the error message? It appears each time some users run Babylon commands.

Hi @johnfolly ,
I will release a v2.0.1 with hotfixes, don't hesitate to propose urgent modifications in this issue.

Merging configuration files in only one babylon_config.yml file with sections

• Move configuration only used in resource creation in working directory
• Investigate using https://www.dynaconf.com/
  example:

 webapp_location: ""
 webapp_repository: ""
 webapp_repository_branch: ""

Separating taks:

• Prepare the skeleton of how a project will look like in terms of configuration files and validate it with CAT and Asset
• Investigate pros and cons of using Dynaconf to handle configuration files
• Replace require_platform and require_deploy by looking up configuration in the Environment singleton
• Change all commands to use new configuration values
  • API
  • Azure
  • PowerBI

Can powerbi deploy-workspace export a the generated set of Report name and report Id ?

The Working Dir could contain an history to follow all actions accomplished for the current project so the deployment state is saved somehow.

Terraform cloud - terraform_version default value

In the terraform cloud workspace template the terraform_version default value is 0.11.1 "terraform_version": "0.11.1".
This version is incompatible with our actual terraform version with is 1.x

Suggestions :

• set the default value to 1.3.1
• do not set default value terraform cloud can auto detect the version

Terraform cloud workspace create command do not write new workspace_id and and workspace_name in terraform workspace config file

Current behavior

• select option do not have default value for the command babylon terraform-cloud workspace create
• the workspace_name is not persist by the select option

Expected behavior

• select option should be boolean option with default value = True
• the workspace_name should be also persist in terraform workspace config file

Should ask @bmoreau for a new ticket

Following discussions with @lalepee and @neomatamune.
This issue will be updated as the project evolves.

Development tools

• autoformatter: yapf
• linter: flake8

Coding style standards

• column limit is 120
• one import per line
• prefer relative imports within the module

Git hooks

You can put the following git hook in a file named .git/hooks/pre-commit to run checks before commit.

#!/bin/sh
flake8 .
yapf -r --diff .
pytest -v

It is possible to run Github actions in a specific environment and set variables that can be used in configuration.

The config would be

• Fork the Sample Web App Repository
• Create the Static Web App
• Update the newly created workflow file in the SWA repository to add env from the github vars
• Update environment variables from the github API

Hello,

At the end of a solution deployment, we maintain a main README that contains all the information related to the deployment. We want to automate the generation and update of this Markdown file. Do you think that it should be a Babylon command or not?

When calling babylon api [solution|organization] --help, I get the following error:

$ babylon api organization --help
AzureCliCredential.get_token failed: ERROR: argument --resource: expected one argument

Examples from AI knowledge base:
az account get-access-token
Get an access token for the current account

az account get-access-token --subscription 00000000-0000-0000-0000-000000000000
Get an access token for a specific subscription

https://docs.microsoft.com/en-US/cli/azure/account#az_account_get_access_token
Read more about the command in reference docs

Traceback (most recent call last):
  File "[...]/babylon/.venv/lib/python3.10/site-packages/azure/identity/_credentials/azure_cli.py", line 141, in _run_command
    return subprocess.check_output(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/bin/sh', '-c', 'az account get-access-token --output json --resource ']' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[...]/bin/babylon", line 33, in <module>
    sys.exit(load_entry_point('Babylon', 'console_scripts', 'babylon')())
  File "[...]/babylon/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "[...]babylon/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "[...]/babylon/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/babylon/.venv/lib/python3.10/site-packages/click/core.py", line 1654, in invoke
    super().invoke(ctx)
  File "[...]/babylon/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/babylon/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "[...]/babylon/.venv/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "[...]/babylon/Babylon/utils/decorators.py", line 152, in wrapper
    func(*args, **kwargs)
  File "[...]/babylon/Babylon/utils/decorators.py", line 152, in wrapper
    func(*args, **kwargs)
  File "[...]/babylon/Babylon/groups/api/__init__.py", line 24, in api
    token = credentials.get_token(api_scope)
  File "[...]/babylon/.venv/lib/python3.10/site-packages/azure/identity/_internal/decorators.py", line 32, in wrapper
    token = fn(*args, **kwargs)
  File "[...]/babylon/.venv/lib/python3.10/site-packages/azure/identity/_credentials/azure_cli.py", line 73, in get_token
    output = _run_command(command)
  File "[...]/babylon/.venv/lib/python3.10/site-packages/azure/identity/_credentials/azure_cli.py", line 154, in _run_command
    raise ClientAuthenticationError(message=message)
azure.core.exceptions.ClientAuthenticationError: ERROR: argument --resource: expected one argument

Examples from AI knowledge base:
az account get-access-token
Get an access token for the current account

az account get-access-token --subscription 00000000-0000-0000-0000-000000000000
Get an access token for a specific subscription

https://docs.microsoft.com/en-US/cli/azure/account#az_account_get_access_token
Read more about the command in reference docs

Ideally consulting the --help option should not require a valid Azure config/setup, or at least not trigger an error.

Generate a macro run report with command return values and which command has failed and succeeded.

We should refactor Babylon to move commands content to classes ie :

from Babylon.commands import AzureCommands
@command
def upload(file_path: str):
   AzureCommands().upload(file_path)
   
class AzureCommands:
  def upload(self, file_path: str):
    pass

We are missing a step in webapp deploy

When the App registration is created, we need to create an associated service principal to add it in powerbi group

az ad sp create-for-rbac --name AzureAssetDeplymentWebApp

• First error : deployment_name got too many values
• Second error : template not found

Folder hierarchy is too deep, we could refactor to

Babylon/
- config/
  - deployment/
    - set-variable.py
- azure
  - acr/
    - storage/
      - create.py

Should be used this way...

babylon powerbi dataset parameters update [dataset_id] -p ADX_CLUSTER https://mycluster.westeurope.kusto.windows.net -p ADX_DATABASE testafos

Missing git on babylon main requirements