• Idea 1: At the beginning of the IaC setup, an important issue, almost solved is the creation of the ServicePrincipal, which is created using the "ProjectAlias" or suffix variable, which must be unique, alphanumeric and up to 8 digits, all this can be controlled with regular functions [regex] and previously validated, to avoid failures in the inicial deployment of IaC.

• Idea 2: Maybe we could leave the alias as an optional parameter for the setup ps1. We could create the alias randomly (by default) and assign it not only to the service principal, but to all the other resources.... Several people on the BR lab tried to use "dataops" as the alias (our suggestion from the documentation), and they had naming conflicts (some resources need to have unique names)

Describe the bug
Running the following script results in a couple of errors: ./quickstart/scripts/labvm/Deploy-LabVM.ps1:

[DevSquad In a Day] Initializing the lab VM disk

ResourceGroupName : rg-labvm
Location          : eastus
ProvisioningState : Succeeded
Tags              : 
ResourceId        : /subscriptions/<subscriptionID>/resourceGroups/rg-labvm


ResourceGroupName            : rg-labvm
ManagedBy                    : 
ManagedByExtended            : {}
Sku                          : Microsoft.Azure.Management.Compute.Models.DiskSku
Zones                        : 
TimeCreated                  : 4/15/2022 2:20:01 AM
OsType                       : Windows
HyperVGeneration             : 
CreationData                 : Microsoft.Azure.Management.Compute.Models.CreationData
DiskSizeGB                   : 
DiskSizeBytes                : 
UniqueId                     : 9322e375-23bd-4811-b579-0053a7c33ced
EncryptionSettingsCollection : 
ProvisioningState            : Succeeded
DiskIOPSReadWrite            : 500
DiskMBpsReadWrite            : 100
DiskIOPSReadOnly             : 
DiskMBpsReadOnly             : 
DiskState                    : ReadyToUpload
Encryption                   : Microsoft.Azure.Management.Compute.Models.Encryption
MaxShares                    : 
ShareInfo                    : {}
Id                           : /subscriptions/<subscriptionId>/resourceGroups/rg-labvm/providers/Microsoft.Compute/disks/disklabvmeastus
Name                         : disklabvmeastus
Type                         : Microsoft.Compute/disks
Location                     : eastus
ExtendedLocation             : 
Tags                         : {}
NetworkAccessPolicy          : AllowAll
DiskAccessId                 : 
Tier                         : P10
BurstingEnabled              : 
PurchasePlan                 : 
SupportsHibernation          : 
SecurityProfile              : 
PublicNetworkAccess          : Enabled
SupportedCapabilities        : 

[DevSquad In a Day] Copying the lab VM snapshot to your subscription
INFO: Scanning...
INFO: azcopy: A newer version 10.14.1 is available to download


failed to perform copy command due to error: no SAS token or OAuth token is present and the resource is not public
Revoke-AzDiskAccess: /home/<path>/<project>/quickstart/scripts/labvm/Deploy-LabVM.ps1:33
Line |
  33 |  Revoke-AzDiskAccess -ResourceGroupName $resourceGroupName -DiskName $ …
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The specified cookie value in VHD footer indicates that disk 'abcd' with blob https://md-impexp-v0b5nhzmddqq.z21.blob.storage.azure.net:8443/z321kgskk1ml/abcd is not a supported VHD. Disk is expected to have cookie value 'conectix'. ErrorCode:
     | InvalidVhd ErrorMessage: The specified cookie value in VHD footer indicates that disk 'abcd' with blob https://md-impexp-v0b5nhzmddqq.z21.blob.storage.azure.net:8443/z321kgskk1ml/abcd is not a supported VHD. Disk is expected to have cookie value
     | 'conectix'. ErrorTarget:  StatusCode: 400 ReasonPhrase: Bad Request OperationID : 854c65f5-2a08-4ebb-8e9a-e85f30d60ffa

[DevSquad In a Day] Creating the lab VM associated resources
WARNING: Upcoming breaking changes in the cmdlet 'New-AzVirtualNetworkSubnetConfig' :
Update Property Name
Cmdlet invocation changes :
    Old Way : -ResourceId
    New Way : -NatGatewayId
Update Property Name
Cmdlet invocation changes :
    Old Way : -InputObject
    New Way : -NatGateway
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.
WARNING: Upcoming breaking changes in the cmdlet 'New-AzPublicIpAddress' :
Default behaviour of Zone will be changed
Cmdlet invocation changes :
    Old Way : Sku = Standard means the Standard Public IP is zone-redundant.
    New Way : Sku = Standard and Zone = {} means the Standard Public IP has no zones. If you want to create a zone-redundant Public IP address, please specify all the zones in the region. For example, Zone = ['1', '2', '3'].
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.
[DevSquad In a Day] Creating the lab VM
New-AzVM: /home/<path>/<project>/quickstart/scripts/labvm/Deploy-LabVM.ps1:86
Line |
  86 |  New-AzVM -ResourceGroupName $resourceGroupName -Location $location -V …
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Disk 'disklabvmeastus' is created for Upload. Call BeginGetAccess to upload data and EndGetAccess before copying or attaching the disk. ErrorCode: OperationNotAllowed ErrorMessage: Disk 'disklabvmeastus' is created for Upload. Call BeginGetAccess to
     | upload data and EndGetAccess before copying or attaching the disk. ErrorTarget: /subscriptions/<subscriptionID>/resourceGroups/rg-labvm/providers/Microsoft.Compute/disks/disklabvmeastus StatusCode: 409 ReasonPhrase: Conflict
     | OperationID : 9228f6dc-0fd9-4873-9a9c-f9f59e215869

[DevSquad In a Day] Done!

To Reproduce
Steps to reproduce the behavior:

1. Follow the Hands-on tutorial setup instructions until step 5 (create the hands-on lab VM)
2. Execute the script to create a VM.

Expected behavior
The script should succeed without any errors, and the lab vm should be created under the resource group.

Screenshots

Desktop (please complete the following information):

• OS: MacOS Monterey
• Browser Edge
• Version 100.0.1185.39 (Official build) (x86_64)

Additional context
N/A

./quickstart/scripts/dataops/Deploy-AzureDevOps.ps1 -ConfigurationFile "./quickstart/outputs/hol.json" -UsePAT $true]

First error- Variables MAJOR, MINOR, PATCH and VERSION do not exist
Second error and Third error
there were no build definitions matching name "we2f5fd1-iac-ci " = "alias-iac-ci"

fourth error
New- item : an item with the specified name

....
8 tokens replace sucesssfully
;....
5 token replaced sucessfully
quicksart-Step4 - .pdf

• Without permission on the Azure Active Directory to grant SP into Application administrator role the pipeline will return an error when try to create a new secret on the SP to store it on the Key Vault to databricks purpouses.

As solution we can store the first secret created into quickstart scripts execution and use it on the pipleline task if the user doesnt have the AAD permission.

Describe the bug
On the last part of Step 2 for the lab, there is a broken link under the following text:

(OPTIONAL) If you are using this project as a Hands-On Lab, feel free to proceed to the next step of the lab setup. If you are using this project as a template for dataops, check [this additional documentation](https://github.com/microsoft/devsquad-in-a-day/blob/main/quickstart/1b-create-prereqs-azure-advanced.md) that explains advanced configuration options.

It looks like this link may have been broken since this line was introduced, according to the following commit:

d871af8

To Reproduce
Go through the quickstart setup instructions

1. At the end of step two, click on the link for checking additional documentation.

Expected behavior
The link should send you to a place where you can get advanced configuration options

Screenshots
N/A

Desktop (please complete the following information):

• OS: All OS
• Browser: All browsers
• Version All versions

Additional context
N/A

We need to put the Service Principal that is created at quickstart scripts into Application administrator role due pipeline permition executions.

Is your feature request related to a problem? Please describe.
Yes. Throughout the Hands-on lab, I encountered several issues. Some I was able to figure out on my own. I would have appreciated for us to have a troubleshooting section where we can discuss these, and provide more context as to how to solve them. This section could grow over time as we find more people having problems following along with the product.

Describe the solution you'd like
These are a handful of topics that I would like to discuss (and I can also create a PR on a fork with a Markdown document detailing the solutions for the items I came across with):

• Connecting DevOps to AAD, when your devops subscription and your target directory do not match. In this case, my Azure Infrastructure was wired up to the default directory under the subscription I have for my work account. My DevOps account defaulted to the "Microsoft Account" directory, which prevented me from linking them together.
• Deleting the project without deleting the artifact feed. Here I did the cleanup of a previous run of the tutorial without going through the cleanup instructions on the guide. When I tried recreating the artifact feed for lib-packages, it wouldn't let me as there was a reference to an existing feed I wasn't able to get to because the project that had a link to it had been removed.
• Setting up a hosted build agent for a project. Here there is an expectation that the lab will be able to run successfully by using a hosted VM. This is currently not working as a new project needs to file a request to get a hosted agent.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
Insert in the quickstart/README.md
First topic:
Requirements before to execute the kickstart
New user : Acquired VisualStudioOnline -
Basic + Test Plans - https://go.microsoft.com/fwlink/?linkid=2092282&clcid=0x409
Use the Trial version to 30 days

You can set up billing, change your subscription or pay for more users and resources within Azure DevOps. Learn more

• Resolve the error that appears when trying creating the same service principal n times

• Remove any references and dependencies to the env variable (PAT) AZURE_DEVOPS_EXT_PAT_TEMPLATE

The current version of the Databricks 7.5 is deprecated and we need to change for a new version.

As sugested by @nansravn we can use the 9.1 LTS that is valid for 2 years

There is some part of the documentation tha need some adjustments

• Enviornments creation step
  • Since recentlly project version updates the enviornment creation is now automatically in the quickstart scripts
• PowerShell script sintax
  • There is some scripts that have "comma" into their instruction and its invalid when running it
• Databricks Scope creation
  • Before run the scope creation script its need to access the workbrench for first time to create the user profile and this instruction isnt present into documentation.

To Reproduce
Steps to reproduce the behavior:

1. Go to hands-on lab documentation
2. Check those steps mencioned before
3. See error

Expected behavior
Documentation adjusted as reality

Screenshots
N/A

Desktop (please complete the following information):
N/A

Smartphone (please complete the following information):
N/A

Additional context
N/A

Following the script bellow on the lab process - Exercice 3 - Mid Task 1 - there are some upgrades that could be interesting for the lab like:

$clientSecret = ConvertTo-SecureString -AsPlainText
./DatabricksSecrets.ps1
-ClientID "<client_id>"
-ClientSecret $clientSecret
-DataResourceGroup "<data_resource_group_name>"
-ComputeResourceGroup "<compute_resource_group_name>"
-KeyVaultName "<kv_name>"
-DataLakeName "<adls_name>"
-DatabricksName "<databricks_name>"

1. Adjust the PS script DatabricksSecrets with the follow changes:

• Get the <client_id> info dinamically using the current context and accessing the Service Principal associated
• Auto generate a new secret inside the SP founded on previous step
• All others parameters will be used as variables inside the pipe due change bellow;

2. Make the execution of this script inside of the pipeline before the task Publish Outputs in the current CI yaml file:

/azure-pipelines/iac/templates/stage.deploy.yml

Describe the bug
A clear and concise description of what the bug is.
run pipeline xxxx-iac-cd

deploy to dev
deploy to DBW

ERROR
Set-AzKeyVaultAccessPolicy: /home/vsts/work/1/s/infrastructure-as-code/scripts/DatabricksSecrets.ps1:62
Line |
62 | Set-AzKeyVaultAccessPolicy -VaultName $KeyVaultName -ObjectId …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Vault 'kv-k0ca8827-lkecvm-dev' does not exist in current
| subscription. If this vault exists in your tenant, please
| switch to the correct subscription in order to modify the
| Access Policies of this vault.

##[error]PowerShell exited with code '1'.

Screenshots
If applicable, add screenshots to help explain your problem.

When run pipeline "alias-iac-cd"
Error-

System.Management.Automation.ItemNotFoundException:
Cannot find path /home/vsts/work/1/t because does not exist
PS - All script run and the resource into de rg-alias-compute-dev and rg-alias-data-dev were create

hands-on-exercise3 -runpaipeline-errro.pdf

Some images are not showing as expected on the main documentation

• Improve the documentation regarding databricks secret scope configuration (system env variable configuration and databricks configure --token).

• By now the deployment of the Lab VM depends on reading a public .vhd from Azure Storage with the VM image. We should investigate other approaches to avoid sharing the whole .vhd publicly.

In present day we have the csv's files into a public blob storage and time-to-time Azure IT block public-access to this storage and a error occur when run the CD pipeline.

A change is needed to store those file compressed into the source repo and instead of copy thos file from the storage we need to de-compress those files and upload it from the repo at runtime of the pipeline execution.

• Clone this repo instead of the private Azure DevOps repo used to create this material
• Remove any references and dependencies to the env variable (PAT) AZURE_DEVOPS_EXT_PAT_TEMPLATE.

When try to create a databricks scope using databricks cli on Task 1: Executing CI/CD Pipeline IaC in the session Databricks Secrets Scope the user will have a 403 permission error.

To Reproduce
Steps to reproduce the behavior:

1. Execute all lab until Task 1: Executing CI/CD Pipeline IaC / Databricks Secrets Scope
2. Try to execute powershell script command showing in the guid regardless scope creation
3. See error

Expected behavior
Execute the script without error

Screenshots
N/A

Desktop (please complete the following information):
N/A

Smartphone (please complete the following information):
N/A

Additional context
For work-arround the error the user must have a first access into the Workbrench inside Databricks platform (its seens due user profile creation).

Describe the bug
In the quickstart
Step 4: Prepare your Azure DevOps project

item 2 :
On the Azure PowerShell terminal: run the following script to clone the hol repo, create the pipelines and service connections inside your new Azure DevOps.

In this command
./quickstart/scripts/dataops/Deploy-AzureDevOps.ps1 -ConfigurationFile "./quickstart/outputs/hol.json" -UsePAT $true

After
devops ran the pipeline automaticaly
k0ca8827-iac-cd
Error
#20220303.1 Update template parameters
##[error]No hosted parallelism has been purchased or granted. To request a free parallelism grant, please fill out the following form https://aka.ms/azpipelines-parallelism-request

To Reproduce*
Steps to reproduce the behavior:

1. Go to 'excute the quickstart
2. Copy the command -./quickstart/scripts/dataops/Deploy-AzureDevOps.ps1 -ConfigurationFile "./quickstart/outputs/hol.json" -UsePAT $true
3. Go to devops and go to pipeline
4. See error
   5

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

• On day 1, at the end when the VM is created, it should be stopped, because the creation script leaves it running, and it only use the next day.