There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Leaving an issue for Gene, but with the Inisghts module, the way in which data is landed in stage1np is not the OEA standard.

• If possible, can we change "M365" parent folder to "Insights"?
• Also, if possible, can we change the folder structure of the Insights data being landed to be:
  • parent_folder/tables/date-timestamp/*.csv rather than parent_folder/activity-and-roster/tables/date-timestamp/*.csv

These issues are relatively minor and do not inhibit the ability to use the module, but are not in agreement with the OEA standard.

The notebook Insights_module_ingestion runs another notebook that is missing or call needs to be renamed

"%run /NEW_Insights_py" This notebook is not in the branch.

The only thing similar is Insights_py.

"name": "CanvasSynchronisationOrchestrator",
"instanceId": "1311ca23e7864c10a8e07cf65af4f5c6",
"runtimeStatus": "Failed",
"input": "{}",
"customStatus": null,
"output": "Orchestrator function 'CanvasSynchronisationOrchestrator' failed: Activity function 'GetFileChangeList' failed: NameError: name 'JsonHmacApi' is not defined \n {"$type":"System.Exception, System.Private.CoreLib","ClassName":"System.Exception","Message":" NameError: name 'JsonHmacApi' is not defined","Data":null,"InnerException":{"$type":"Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException, Microsoft.Azure.WebJobs.Script","StackTrace":" at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 96\n at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82\n at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task1 src) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225\\n at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync(Object instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs:line 52\n at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 581\n

Received following error message running setup.sh at revision 05d0a4e8563b92e15314f795d84af74a98c96b1b

--> Creating storage account containers: stage1, stage2, stage3, synapse
{
  "created": true
}
{
  "created": true
}
{
  "created": true
}
{
  "created": true
}
{
  "created": true
}
--> Creating Synapse Workspace: syn-oea-dmoea001
Command group 'synapse' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
ValidationFailed: Workspace request validation failed, check error details for more information
WARNING: Command group 'synapse' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
ERROR: (ResourceNotFound) The Resource 'Microsoft.Synapse/workspaces/syn-oea-dmoea001' under resource group 'rg-oea-dmoea001' was not found. For more details please goto https://aka.ms/ARMResourceNotFoundFix
argument --assignee: expected one argument

TRY THIS:
az role assignment create --assignee 00000000-0000-0000-0000-000000000000 --role "Storage Account Key Operator Service Role" --scope $id
Create a new role assignment for a user, group, or service principal. (autogenerated)

az role assignment create --assignee sp_name --role a_role
Create role assignment for an assignee.

az aks show --name MyManagedCluster --resource-group MyResourceGroup
Show the details for a managed Kubernetes cluster

https://docs.microsoft.com/en-US/cli/azure/role/assignment#az_role_assignment_create
Read more about the command in reference docs

Saw a comment from Paul DeSante on Teams chat that he might have run into the same thing. He said "I also needed to register the Microsoft.sql resource provider so that the Synapse workspace could be setup." I haven't checked to see if that addressed my actual issue.

There needs to be a way to specify custom resource names for customers that have an established naming convention that differs from the standard.

Description:
A clear and concise description of what the bug is.

Action version:
Specify the action version

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Repro steps:
A description with steps to reproduce the issue. If your have a public example or repo to share, please provide the link.

Expected behavior:
A description of what you expected to happen.

Actual behavior:
A description of what is actually happening.

Originally posted by @GCC1111 in actions/stale#1104

https://github.com/web-platform-tests/wpt/blob/master/fetch/api/crashtests/body-window-destroy.html

We have been using the IPEDS Dashboard and the link below, where we had previously gone in and worked with the Trivium Dashboard. It is still up and running but I can't get to it through this link anymore to edit the file. Can you provide an update?
https://github.com/microsoft/OpenEduAnalytics/tree/main/modules/IPEDS

Need to add tags to resources for better resource labeling and management.
Use tags to organize your Azure resources and management hierarchy

This also helps to satisfy policy requirements used in some orgs: What is Azure Policy

setup.sh fails to create all resources if az-cli-ml extension has not been added prior to the run.
mitigation error message is helpful, but would be great to state this in prereqs.

Hi,
I'm having trouble opening and working with the PowerBI-reports from https://github.com/microsoft/OpenEduAnalytics/tree/main/modules/module_catalog/Microsoft_Education_Insights
both for K12 and HED. The old reports can be opened, but the new reports should already be prepared for Lake-DB.

When I open the new reports, I get a message that it couldn't verify the sensitivity label of the file.
The errormessage is exactly, though sadly in swedish:
Det går inte att verifiera filens känslighetsetikett. Försök igen senare
My english translation:
Cannot verify the sensitivity label of the file. Try again later.

Trying to open it online gives me the following error:
This file is restricted. Your organization has protected the file with a sensitivity label.
Error code: InformationProtectionInsufficientUsageRights

Could anyone please verify that the file is indeed not protected by any Sensitivity Label?

I've tried opening it on multiple devices.

If a user wants to use Windows Subsystem for Linux (WSL) to execute setup.sh, the installation will script will fail if the user just executes a Git clone operation against the OpenEduAnalytics repo. The reason for this is that the default git client behavior on Windows is to convert Linux line endings to Windows line endings, as controlled by the following git config parameter:
Configuring Git to handle line endings

While it's true that this parameter is under end-user control and will need to be adjusted on their end, it might be worth adding a brief note to the Setup section about this default behavior when cloning the code to Windows machines. The git clone command can be invoked with the following parameters to override the system defaults with regards to line endings:
git clone --config core.autocrlf=false https://github.com/microsoft/OpenEduAnalytics

Need to update the naming to align with this guidance: Develop your naming and tagging strategy for Azure resources

          012 180 015538371447

Originally posted by @Jessuss21 in #290 (review)

I ran through the guide this morning. In the documentation, please update the database name in Power BI to: test_s2_contosoisd.

There has been an update to explicitly do the diff acquisition secured by spark streaming as commented below, but is this unnecessary?

2a82954#r112823984

.sh has all the logic to create all the components to set up the integral solution; it starts creating different resource groups, Analytikus has policies created to force add tags. It means that an automatic process that creates components into azure can return errors.

Observation: We suggest adding some JSON file with defaults tags to be attached to the setup of the new components, for example:

[ { "tag_name":"CreatedBy", "tag_value":"Uayeb Caballero" } ]

and finally, we run our command doing something like:

./setup.sh <unique_suffix> <tags.json>

This repository is currently missing a LICENSE file.

A license helps users understand how to use your project in a compliant manner. You can find the standard MIT license Microsoft uses at: https://github.com/microsoft/repo-templates/blob/main/shared/LICENSE.

If you would like to learn more about open source licenses, please visit the document at https://aka.ms/license (Microsoft-internal guidance).

Fails Synapse Workspace deployment.

    "statusMessage": "{\"error\":{\"code\":\"ValidationFailed\",\"message\":\"Workspace request validation failed, check error details for more information\",\"details\":[{\"code\":\"InvalidSqlServerPassword\",\"message\":\"Sql Server password not complex. The password must be atleast 8 characters long and contain characters from three of the following four categories: (uppercase  letters, lowercase letters, digits (0-9), Non-alphanumeric characters such as: !, $, #, or %).\"}]}}",

I have the latest version of Power BI Desktop installed. After downloading it, when I try to open the report I get a this error message

I was wondering should we have a function below get_secret in OEA() that gets the SALT from key vault? Something like:

 def getSALT(self):
        self.SALT = self.get_secret('kv-oea-SALT')

There are open compliance tasks that need to be reviewed for your OpenEduAnalytics repo.

Action required: 3 compliance tasks

To bring this repository to the standard required for 2021, we require Microsoft administrators of this GitHub repository to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your GitHub repo.

Microsoft repo admins: Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/microsoft/repos/OpenEduAnalytics/compliance

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

AS instructed in Step 3.1 in the document "Tutorial for ML pipeline.docx" here, unable to see the command option for the "+" --> Import resources from support files". Hence cannot proceed, please guide to resolve this issue.

We are running into some issues trying to run the bash setup.sh script in cloudshell. It appears that a folder path might be missing. Could you please let me know if I am missing something?

This is less of an issue, as a consideration, but it might be worth setting the "Allow Blob public access" parameter for the OEA Storage Account to False by default. Doing so would add one more layer of risk mitigation, by forcing end-users to explicitly enable this feature if they really do have a use case for anonymous access in their own containers within their Storage Account.

Configuring the OEA Storage Account defaults in this way would also align with our recommendations in the following document, Security recommendations for Blob storage where we outline the following:

Anonymous public read access to a container and its blobs grants read-only access to those resources to any client. Avoid enabling public read access unless your scenario requires it.

To be clear though, I did see that ALL OEA blob containers themselves are set to Private, which is great.