While running this workshop. We found that people were attempting to do the 4th and 5th workshop as well. Would it be possible to split the notebooks into 2 different dbc files?

Not a big one but when I follow the guide there are a few instances where the actual Portal Screen does not match the guide images. Not sure if this is due to location or user differences.

1\ To add the container to the blob store my Portal does not show "Containers", it shows "Browse Blobs" (see image attached).

2\ When creating the Experimentation Service the screen and instructions say "AI + Cognitive Services" where I am seeing "AI and Machine Learning"

The Error appears to be when running az ml account modelmanagement set -n bigdataholmlexpModelMgmt -g big-data-hol

            {
              "code": "BadRequest",
              "message": "SubDeployment: OperationId=D8F9B9524A432C4D, ProvisioningState=Failed, StatusCode=BadRequest, StatusMessage={\n  \"code\": \"BadRequest\",\n  \"message\": \"An error has occurred in subscription df8428d4-bc25-4601-b458-1c8533ceec0b, resourceGroup: big-data-hol-azureml-fee3b request: OrchestratorProfile has unknown orchestrator version: 1.7.7.\"\n}\n"
            },

I believe this is due to the Kubernetes version 1.7.7 no longer being valid.

c:\HOL\FlightDelays>az ml account modelmanagement set -n bigdataholmlexpModelMgmt -g big-data-hol
The behavior of this command has been altered by the following extension: azure-cli-ml
{
  "created_on": "2018-10-29T18:53:59.357547Z",
  "description": "",
  "id": "/subscriptions/df8428d4-bc25-4601-b458-1c8533ceec0b/resourceGroups/big-data-hol/providers/Microsoft.MachineLearningModelManagement/accounts/bigdataholmlexpModelMgmt",
  "location": "westcentralus",
  "model_management_swagger_location": "https://westcentralus.modelmanagement.azureml.net/api/subscriptions/df8428d4-bc25-4601-b458-1c8533ceec0b/resourceGroups/big-data-hol/accounts/bigdataholmlexpModelMgmt/swagger.json?api-version=2017-09-01-preview",
  "modified_on": "2018-10-29T18:53:59.357547Z",
  "name": "bigdataholmlexpModelMgmt",
  "resource_group": "big-data-hol",
  "sku": {
    "capacity": 1,
    "name": "DevTest"
  },
  "subscription": "df8428d4-bc25-4601-b458-1c8533ceec0b",
  "tags": null,
  "type": "Microsoft.MachineLearningModelManagement/accounts"
}

c:\HOL\FlightDelays>az ml env list
The behavior of this command has been altered by the following extension: azure-cli-ml
[
  {
    "Cluster Name": "flightdelays",
    "Cluster Size": 2,
    "Created On": "2018-10-29T22:21:32.883Z",
    "Current Mode": "cluster",
    "Location": "westcentralus",
    "Provisioning Errors": [
      {
        "error": {
          "code": "BadRequest",
          "details": [
            {
              "code": "OK",
              "message": "SubDeployment: OperationId=C9CA9D729418060B, ProvisioningState=Succeeded, StatusCode=OK, StatusMessage=\n"
            },
            {
              "code": "OK",
              "message": "SubDeployment: OperationId=929E883E9C6C3F4E, ProvisioningState=Succeeded, StatusCode=OK, StatusMessage=\n"
            },
            {
              "code": "OK",
              "message": "SubDeployment: OperationId=FC5C74D11979371E, ProvisioningState=Succeeded, StatusCode=OK, StatusMessage=\n"
            },
            {
              "code": "OK",
              "message": "SubDeployment: OperationId=048F9B1A3F6D131E, ProvisioningState=Succeeded, StatusCode=OK, StatusMessage=\n"
            },
            {
              "code": "BadRequest",
              "message": "SubDeployment: OperationId=D8F9B9524A432C4D, ProvisioningState=Failed, StatusCode=BadRequest, StatusMessage={\n  \"code\": \"BadRequest\",\n  \"message\": \"An error has occurred in subscription df8428d4-bc25-4601-b458-1c8533ceec0b, resourceGroup: big-data-hol-azureml-fee3b request: OrchestratorProfile has unknown orchestrator version: 1.7.7.\"\n}\n"
            },
            {
              "code": "OK",
              "message": "SubDeployment: OperationId=AC4B60FE0F7564A7, ProvisioningState=Succeeded, StatusCode=OK, StatusMessage=\n"
            },
            {
              "code": "OK",
              "message": "SubDeployment: OperationId=3FE3F9847F8B6FFB, ProvisioningState=Succeeded, StatusCode=OK, StatusMessage=\n"
            },
            {
              "code": "Conflict",
              "message": "SubDeployment: OperationId=08586607550878056481, ProvisioningState=Failed, StatusCode=Conflict, StatusMessage=Template output evaluation skipped: at least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.\n"
            }
          ],
          "message": "Deployment failed with one or more errors. Please look at the inner error details"
        }
      }
    ],
    "Provisioning State": "Failed",
    "Resource Group": "big-data-hol",
    "Subscription": "df8428d4-bc25-4601-b458-1c8533ceec0b"
  }
]
c:\HOL\FlightDelays>az ml env set -n flightdelays -g big-data-hol
The behavior of this command has been altered by the following extension: azure-cli-ml
{
    "Azure-cli-ml Version": null,
    "Error": "Resource with group big-data-hol and name flightdelays cannot be set, as its provisioning state is Failed. Provisioning state succeeded is required."
}

Every one at my table ran into the same problem when setting up the machine learning environment for the Machine Learning Workbench.
The solution we found eventually was to go to the Azure Portal and for your subscription in the "Resource Providers" register the following providers:

Microsoft.ContainerInstance
Microsoft.ContainerRegistry
Microsoft.ContainerService
Microsoft.MachineLearningServices

Note: This happened to users with only 1 Azure subscription as well as to the ones with more than 1 subscription

Ex.1: Task.2.Step.6.- 3rd Notebook command 10 is giving error: Unable to create the workspace.

Abstracts on the ReadMe page do not match the abstracts used in the WDS PPT, trainer/student guides, or the HOLs.

We should create Lab VM in Before hands on Lab. Can someone look into this?

WDS Trainer guide asks and answers 8 customer objections - PPT only has 7

Hi @joelhulen

In Exercise 8: Deploy intelligent web app Task 2 Deploy web app from GitHub When we click on provided URL: https://github.com/Microsoft/MCW-Big-data-and-visualization/blob/master/Hands-on%20lab/lab-files/BigDataTravel/README.md to deploy the web, it re-directs to the Azure portal and opens the GUI as show in below image:

In this GUI, we have two options for Resource Group - 1. We can select Existing Resource Group 2. We can create a new Resource Group.

Issue: When we're selecting Existing Resource Group even after that it asks for the permissions to create a new Reaource Group.

As, we're using shared subscription pool for this workshop we can't give subscription level permission to users to create new Resource Group. We're proving them a pre-created Resource Group and it works for the lab fine. Only we're having issue while deploying this web app - as it is asking subscription level permissions to create a new resource group.

Could you please take a look and see if this can be fixed ( selecting existing Resource Group can work as expected) ?

I can provide you one lab environment, if needed.

Thanks in Advance!
Abhishek Pathania

there is a syntax error on CMD 15 line 2 in the Predict Flights dbc file. The 36L should become 36

The august-2018-test-fix branch is ready for review and QC.

Hi there, the links to both hands-on-labs (step-by-step and unguided) are not working "Page 404 - not found". Can you please fix this?

Thanks

We should have a "Deploy to Azure" button to automate as much of setup as possible.

Followed the Lab steps to created the DSVM and encountered the below error when RDP to the DSVM.

"CredSSP encryption oracle remediation" error when RDP to a Windows VM in Azure

The below KB describes the issue and workaround.
https://support.microsoft.com/en-us/help/4295591/credssp-encryption-oracle-remediation-error-when-to-rdp-to-azure-vm

I had followed the workaround options and still unable to RDP the DSVM.

We may need to address this issue prior to the workshop or will not able to proceed with the lab.

With client who would like to complete this lab -- the mismatch of the ever-changing interface needs to be resynchronized, thanks

hi,

there are some broken links in CONTRIBUTING.md file. Below are 3 broken links which require fixing:

• Before you create a new issue, please do a search in open issues to see if the issue or feature request has already been filed.
• Be sure to scan through the most popular feature requests.
• If you are interested in writing code to fix issues, please see How to Contribute in the wiki.

When I hover over the images I'm not seeing the alt-text titles - only links popping up at the bottom of the screen.

This step starts successfully but when you run "az ml env show..." command you may get an error due to an unregistered resource provider Microsoft.ContainerService.

To resolve this issue in my subscription I had to run the command "az provider register --namespace Microsoft.ContainerService"

Please review & fix in scheduled test/fix.

Remediation
Upgrade microsoft.identitymodel.clients.activedirectory to version 5.2.0 or later. For example:

Always verify the validity and compatibility of suggestions with your codebase.

Details
CVE-2019-1258
More information
moderate severity
Vulnerable versions: < 5.2.0
Patched version: 5.2.0
An elevation of privilege vulnerability exists in Azure Active Directory Authentication Library On-Behalf-Of flow, in the way the library caches tokens, aka 'Azure Active Directory Authentication Library Elevation of Privilege Vulnerability'.

In Exercise 4 Task 2: Trigger Workflow I tried out the lab and found, Trigger is failing with error message: Activity BatchScore failed: Databricks execution failed with error message: . Run page url: https://eastus2.azuredatabricks.net/?o=2487023948884853#job/1/run/1. This URL is pointing to Exercise 4 of Notebook, but according to lab guide we don't need to run this notebook here.

Can someone please look into this on urgent bases. Thanks
Abhishek Pathania

While deploying the ARM Template in exercise 7, we got an error: Deployment Error - The current api-version doesn't support RBAC users. Please use a newer version instead. . By looking closer to the deployment details we could see that's coming from Microsoft.Web/sites/sourcecontrols.

In Exercise 7, the web app is always displaying 0% confidence for the Delay Prediction.

In Exercise 8, task 2 (deploying the application) the notes refer to "Finally, enter the ML API Primary key (we got that from Azure databricks Notebook #3, remember?) and Weather API information." ... I am struggling to find where this key actually came from?

This is in preparation for running a workshop next week. While not critical or a show stopper the revised content of the workshop makes it much more likely attendees will get all the way to the end and this point will be reached.

In Before Hands on lab Step 8. Creating Azure Resources Machine Learning Experimentation and Machine Learning Model Management
But the current status of these resources in Azure portal is Retiring

Showing error, no such file or directory, it was all good last week.

cat: /databricks/driver/aml_config/config.json: No such file or directory

Please look into this ASAP, we've workshop today itself.

Thanks
Abhishek Pathania

1. I had raised a issue earlier for Lab VM but author replied we have removed Lab VM in latest update, but I see use of Lab Vm is still there in Exercise 2. Task 2

• Here are some screenshots:

1. Exercise 2 Task 2: Install and configure Azure Data Factory Integration Runtime on the Lab VM
   
2. Task 3: Configure Azure Data Factory Step 13.
   

I am confused due to this, can anyone look into this ASAP, we have one workshop scheduled next week. Thanks

Folder and document names have been updated. Please check your HOL documents for links that use folder names in their path and make sure they are still valid and working.

Hello,
This workshop is scheduled for a content update. Please review the workshop and current open issues and give update suggestions.

Please include field request to add Azure Data Lakes as a storage solution to the workshop.

Thanks,
Dawnmarie

Please update https://github.com/Microsoft/MCW-Big-data-and-visualization/blob/master/Hands-on%20lab/HOL%20step-by-step%20-%20Big%20data%20and%20visualization.md#exercise-8-deploy-intelligent-web-app-optional-lab to be "Exercise 7" vs "Exercise 8"

May update QC'd and merged, ready for updated HTML

Cmd 43: DeprecationWarning: ContainerImage class has been deprecated and will be removed in a future release. Please migrate to using Environments. https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-environments

Cmd 45: DeprecationWarning: deploy_from_model has been deprecated and will be removed in a future release. Please migrate to using Environments. https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-environments

Optional Exercise 8, can't be performed because new signup's are not possible for Dark Sky API
Signup option is no more available on Dark Sky API.

Exercise 1 Task 2: Open Azure Databricks and complete lab nitebook
Step 1. Download: BigDatavVis.dbc , in HTML guide this link is broken. Got blob not found error.

Could anyone please look into this and fix ASAP, we have a workshop scheduled tomorrow we needed this to be fixed ASAP.

Thanks,
Abhishek Pathania

Adventure Works Travel is not on the list of approved fictitious names. Adventure Works Cycles is, but it doesn't make sense to have a cycle company in the example without a bunch of changes to the travel company example.

While running the Azure Data Factory pipeline in Exercise 4 in the Notebook 3 for this cell:

dfDelays = spark.read.csv("wasbs://" + containerName + "@" + accountName + ".blob.core.windows.net/FlightsAndWeather/*/*/FlightsAndWeather.csv",
                    schema=data_schema,
                    sep=",",
                    header=True)

We got this error:

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container sparkcontainer in account storagex6uezrh5wfar2.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration.

Even if we have properly setup the Azure Blob storage output step. Is there anything missing in this lab to have this working?

Notes:

• I needed to update Notebook 3 by replacing STORAGE-ACCOUNT-NAME by my appropriate accountName - it's not explicitly mentioned in this lab to do so, we need to update the lab instructions in that regards.
• If I change the default access level policy of my blob storage container to public, it's working, it's a temporary workaround but not an ideal solution.

Hi,
in Exercise 2: 01 Data Preparation Cmd 57, the first line of code has a wrong syntax issue:
df = dfWeather.withColumn('Hour', f.floor(dfWeather['Time']/100)) throws an NameError: name 'f' is not defined. Instead, it must be:
df = dfWeather.withColumn('Hour', F.floor(dfWeather['Time']/100)). Either change the Cmd 57 line 1 or change the Cmd 3 line 5: from pyspark.sql import functions as f

Hi, when I try to run the Databricks Notebook "01 Deploy for Batch Scoring" in Exercise 5, the Cmd 15 fails with a Py4JJavaError error:

`
(1) Spark Jobs
Job 293 View(Stages: 0/0, 1 skipped)

org.apache.spark.SparkException: Job aborted.

Py4JJavaError Traceback (most recent call last)
in ()
----> 1 prediction.write.mode("overwrite").saveAsTable("scoredflights")

/databricks/spark/python/pyspark/sql/readwriter.py in saveAsTable(self, name, format, mode, partitionBy, **options)
773 if format is not None:
774 self.format(format)
--> 775 self._jwrite.saveAsTable(name)
776
777 @SInCE(1.4)

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in call(self, *args)
1255 answer = self.gateway_client.send_command(command)
1256 return_value = get_return_value(
-> 1257 answer, self.gateway_client, self.target_id, self.name)
1258
1259 for temp_arg in temp_args:

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
61 def deco(*a, **kw):`

The shortcut for running notebook cells as mentioned in the HOL step-by-step guide should be Ctrl+Enter instead of Ctrl+Shift.

https://docs.databricks.com/user-guide/notebooks/notebook-use.html

Hi there,
in HOL unguided - Big data and visualization.md in cell 32 there is an instruction saying that we should use the spark command SaveTableAs(). This function is not existing. Instead it should be SaveAsTable().

dfFlightDelays_Clean.write.saveAsTable("flight_delays_clean")

Hello,
Please check formatting. I added the TOC to the document but the links inside the table aren't working, not sure what I did wrong.

Instruction to install the PyPi librairie – azureml.core needs to be documented otherwise the first cell of Notebook 3 will fail.

We have found that in Notebook 3 and 4 there are places where participants need to change the subscription ID and storage-account name. This is not called out very clearly. Can we explicitly fail when someone doesn't enter/replace the subscription ID and other required variables

Github is issuing a potential security issue because the packages.config file is referencing an older version of Bootstrap that has a known vulnerability issue. recommend to update to 3.4.1

This workshop is scheduled for an update. Please review the contents of this repo and comment on this issue with suggested updates. All suggestions must be added to this issue by EOD 11/9/18.

Slide 21 is missing alt text.

It would be nice to highlight dbutils.fs.mount() functionality

The directions specify creating a Databricks 3.4 cluster specifically due to the spark version installed on the data science vm; however the lowest cluster version we can create is 3.5 LTS.

The documentation should be updated to reflect either a higher version of the Databricks cluster, and if necessary update the DSVM to a higher version of Spark to match.

In the step by step guide, step 9 says "Change the Table Name to "flight_weather_with_airport_code" and select the checkmark for First row is header. Select Create Table."

My guess is the "flight_weather_with_airport_code" should be "flight_delays_with_airport_code" as per the screen shot below it.

In Exercise 1, Task 4, Step 8, users are receiving an error if they have multiple Azure subscriptions and the default is not the subscription they are using for the lab.