Modules about bicep HOT 28 CLOSED

To your first question, we definitely need to keep these types of validations/completions in mind for the language server. I would definitely expect it to show any allowedValues for a given parameter defined in the module. It may be more technically challenging to get the schema values of the actual resource, but that is what we should shoot for.

For the second question, I'm not sure we would allow you to override properties of resources that are not explicitly exposed as a defaultValue of a parameter. So in this case, I would recommend in the module definition to add a parameter like:

parameter agentCount int default 2

(we're still playing around with the defaulting syntax)

If we allow you to override any property, there is no guarantee that the module will continue to work.

from bicep.

For validating "simple" parameter, we can definitely build that improvement later, but I would prefer setting the rules clear up front.

from bicep.

@lwang2016, @majastrz - the spec for modules in in master, should we close this?

from bicep.

1. I like the idea of fileMetadata - seems clean. Would it be required? What happens if I don't include it?
2. I was a little confused by the module consumption syntax. In my head I think of it either as a separate top level keyword or as a resource of type module. Seeing the module name after = is throwing me off for some reason. It is also not clear to me why there is an array syntax (resource[]). The user shouldn't need to know if there are 1 or N number of resource being created by a module. These two make more sense to me:

resource of type module:

resource myModule 'module://[email protected]' = {
  accountName: 'fooAccount'
}

module as keyword:

module myModule '[email protected]' = {
   accountName: 'fooAccount'
}

I prefer the latter because it treats modules as more of a first-class citizen.

3. I personally am not a fan of import, since it feels like we can avoid it and put all the relevant info in the module instantiation, but I think I'm in the minority.

from bicep.

Use a question to answer your first comment 😆. Is module/lib version required?

I like your second option too, and it also allows us to remove import because compiler can search for the module file upon the keyword. Will update proposal accordingly.

from bicep.

type on fileMetadata only has one value right now. There are no libraries (and may never be), so I don't think we need that. (It's also possible for us to automatically figure out the type of file it is based on what is declared inside.)

I don't think we need version either. Version on module declaration and reference implies that there's a repository of modules somewhere and we will get the specified version or fail. For option 3 (relative paths), it's definitely not needed. Probably not needed for other search path options as well because I don't expect users to have multiple versions of the same module checked in as separate files side by side in the same repo.

@lwang2016 on option 2, you said "specify a bicep root". Where is it specified? In the bicep command line? Some config file? (How do we find the config file?) What is the equivalent of that in VS code extension? How easy would it be to keep in sync so the user gets the same errors/experience in command line and VS code?

from bicep.

wrt to "bicep root" I would expect that to be the directory where main.arm is sitting.

@majastrz - we need to have some versioning story for external modules (e.g. a "Bicep Module Registry"). I agree it should not be needed for "local" modules that are in the same repo as main.arm.

from bicep.

What do we expect bicep local development folder structure to be? I can think of several options, and I prefer option 3.

option 1: all files that a main.arm needs live under the same folder. will this make bicep folder messy and lack of logical separation between main files?

/bicep/main1.arm
/bicep/main2.arm
/bicep/module1.arm
/bicep/module2.arm

option 2: each main.arm has its own folder and all files it needs live under the same folder. this means module files have to be duplicated.

/main1/main1.arm
/main1/module1.arm
/main1/module2.arm

/main2/main2.arm
/main2/module1.arm
/main2/module2.arm

option 3: have a bicep root folder and modules can live in a sub-folder that's separarte from main files. In this case, compiler needs to know where to find module files. We can have a bicep settings file (bicep.config in the example below) in which the root folder is specified.

For VS Code, users can do code . under /bicep folder. This opens the bicep workspace, and

For command line, users can specify bicep config file via a compiler option, such as:
bicepc -settings ./bicep.config in which bicepc means bicep compiler.

/bicep/modules/module1.arm
/bicep/modules/module2.arm

/bicep/main1/main1.arm
/bicep/main2/main2.arm
or:
/bicep/main1.arm
/bicep/main2.arm

/bicep/bicep.config

from bicep.

@alex-frankel Totally agree that we need versioning for non-local modules. For local ones, we could either have latest version in the module reference or simply omit the version part.

@lwang2016 I agree on option 3 for file folder structure. I'd expect modules to live in their own folders because they could be shared across different "main" files. The config file could solve it. The search logic for bicep.config file could be just go up the directory hierarchy until you find a bicep.config also. That way the root folder wouldn't matter that much. If we go with relative paths all the way, then we wouldn't need the config file at all.

Regardless, we can make some folder structures easier to use, but we shouldn't block users from using their own structures. (Not saying that any proposal here does that - just explicitly stating a requirement.)

from bicep.

@majastrz, yes, we shouldn't enforce the folder structures.

from bicep.

Updated proposal to incorporate feedback.

from bicep.

We got some feedback from of the DevDiv PM team about aligning resources and modules more closely. There was concern that all of the simplification we will be doing is in modules, but learning modules adds an extra layer of complexity. So we were brainstorming a syntax like the following which is a VM module that creates the VM, NIC, PIP, etc.:

resource myVm 'module://vm' {
    name: 'myvm'
    location: 'eastus'
    os: 'Linux'
}

Personally, I am not a fan as I think this confuses exactly what the resource keyword means, but I think the high level feedback of "how can we make the simple scenarios as easy as possible for people to learn and implement" is fair, so it's something we should consider.

from bicep.

Some things that stood out to me from the conversation. It sounds like modules + a module registry might serve as an abstraction utility to simplify resource declarations.

Will there be a module authoring experience (validation / completions)?

For example, when consuming a module can I discover configurable properties? If I select ctrl + space after os, will the extension return a list of schema derived values? If I add a non-valid value for os will it be flagged as non-valid?

module vm [email protected] = {
    name: 'myvm'
    location: 'eastus'
    os:  // cursor is here 
}

I believe there was also a question about how this would work for nested properties. For example, when changing the agent pool count for an AKS cluster, would this be represented using dot notation?

module aks [email protected] = {
    agentPoolProfiles.count: 2
}

from bicep.

Personally, I am not a fan as I think this confuses exactly what the resource keyword means, but I think the high level feedback of "how can we make the simple scenarios as easy as possible for people to learn and implement" is fair, so it's something we should consider.

Totally agree with the concern here. resource keyword would be very confusing. Curious about the particular part that they felt module adding an extra layer of complexity. Is it yet another keyword? or the extra requirements on the optional params/outputs? or the general concept of module? If the last one, can we test the concept with a few customers with less technical background to see if it's easy for them to take the concept and distinguish resource from module?

from bicep.

@neilpeterson we definitely will have module parameter completions for module references. What's not clear to me yet is how much complexity we will allow in the schema of parameters in module declarations.

We wouldn't allow a notation that overrides the module implementation. If something needs to be a parameter, the module author should expose it as a parameter.

from bicep.

I would expect the declarations to allow for the same set of modifiers as a parameter on a "main" Bicep file (allowedValues, min/max, etc.). I think the interesting scenario to me is something like the following.

Take the same module as Neil's example:

module vm [email protected] = {
    name: 'myvm'
    location: 'eastus'
    os:  // cursor is here 
}

In the declaration, I have no modifiers on my parameter, so something like:

parameter os string

So there is nothing that can be checked other than the resource schema itself, which does have enum values declared of linux and windows. Would the langauge server be smart enough to show those allowed values when instantiating the module?

from bicep.

I see module parameters as a contract that should be thought through and declared upfront. We should avoid the temptation to infer it automagically. Otherwise, a minor expression change (such as assigning a parameter value to a different place) could drastically change the modifiers that we generate in the JSON, which would make readability and module back-compat harder for the user to grok.

If it's done upfront, the module contract is readable and we can validate against module authoring mistakes.

from bicep.

Yes, Marcin and I had a discussion just now. For module parameter validation, we can require authors specifying decorators such as range, allowed values, or default values, etc. and enforce the requirement via GitHub check-in tests when the module is published to BMR. In this way, we don't need resource schemas for the validation and language server can show errors at runtime if invalid values are provided to module parameters.

from bicep.

I see two problems potentially occurring:

• as a module author, I effectively need to duplicate the restrictions specified in the resource schema
• if the module author did not do that duplication, then there is a chance the module will validate but the generated template is invalid. Had I not used the module, and declared the resource directly in the DSL, it would have validated and caught the error. I can imagine this would be a frustrating experience

from bicep.

• as a module author, I effectively need to duplicate the restrictions specified in the resource schema
• if the module author did not do that duplication, then there is a chance the module will validate but the generated template is invalid. Had I not used the module, and declared the resource directly in the DSL, it would have validated and caught the error. I can imagine this would be a frustrating experience

module parameter is not always directly used to create resource, the author can massage the value and create a variable off it or pass it around before actually setting the resource property. So using resource schema for validation is non-trivial.

since parameters and outputs are the contract a module exposes externally, it is a best practice for the author to think through what values could be provided and put in validation up front, and we can promote the practice via BMR check-in tests.

from bicep.

module parameter is not always directly used to create resource, the author can massage the value and create a variable off it or pass it around before actually setting the resource property

Good point - I wonder if there is a way we could do it for "simple" parameter declarations where they are directly used for the resource property.

from bicep.

I see module parameter validation as a "best effort", even deployment validate can't always guarantee its accuracy, many factors could be in play here. We can set user expectation clearly. The open-source nature makes it easier for module users to provide feedback to the author who can continue improve quality. For example, for the frustrating experience you mentioned, maybe the answer is for the author to iterate and improve the validation quality.

from bicep.

How the difference between module and lib will be enforced for example if someone try to write a lib with resources to be deployed? will there be a validation or just leave this to the common sense ?

from bicep.

Good question. We just won't support libraries initially, so we can defer the answer to later.

from bicep.

Good discussions in design meetings today. Notes from Alex:

• no distinction between top level bicep file and a module
• need versioning for a registry, but can defer it for now
• open question on if I am referencing a directory vs referencing a single file
  • if it's a directory for a module, then it should be the same pattern for a top level file
• good with the syntax for instantiating a module (pending closure on versioning & directory v. file reference)

from bicep.

@lwang2016 - should we submit a PR to the /specs directory? Feels like we are closed on enough to do so. We can choose to do directory based modules for now and change it later if we need to. Let's omit versioning until we have more clarity on how the registry is going to work.

from bicep.

@lwang2016 - should we submit a PR to the /specs directory? Feels like we are closed on enough to do so. We can choose to do directory based modules for now and change it later if we need to. Let's omit versioning until we have more clarity on how the registry is going to work.

Yes, will submit today.

Had one question when writing the spec. Is directory still required if a module only has one file? It feels cumbersome.

Bicep compiler should be smart enough to handle both. If file name provided, it's a single-file module. If directory name provided, it's a multi-file module and all files under the directory (non-recursively?) are included. If a file and directory having the same name under the path, compile error.

from bicep.

oh nice - I like the idea of supporting both!

from bicep.