azure / iotedge-eflow Goto Github PK
View Code? Open in Web Editor NEWAzure IoT Edge for Linux on Windows
Home Page: https://aka.ms/azeflow-docs
License: MIT License
Azure IoT Edge for Linux on Windows
Home Page: https://aka.ms/azeflow-docs
License: MIT License
Question Summary
The Iothub allows devices to use it's File Upload API handshake to upload files to Blob storage using SAS Tokens. Can the EFLOW device leverage this api workflow too? And if so Could I think have a module leverage it through EFLOW?
Just wondering
Not sure if this a bug or not, but we noticed a quirk of the hosts file on the VM that is causing us trouble in connecting to the edgeHub module. Here's the content of /etc/hosts in the EFLOW VM:
# Begin /etc/hosts (network card version)
::1 ipv6-localhost ipv6-loopback
127.0.0.1 localhost.localdomain
127.0.0.1 localhost
127.0.0.1 efl-vm
# End /etc/hosts (network card version)
Notably, there's no entry there that contains the actual host name of the EFLOW VM, in our case W10IOT-EFLOW-458b3451. On the standard Ubuntu and Raspbian systems we've tested with, there's been an entry included in the hosts file like "127.0.1.1 DeviceHostname"
Is this intentional? By the way, looking at my Preview Preview EFLOW VM, it looks like there was such an entry included.
Not having this entry there seems to result in a failure to connect to the edgeHub. We are using Microsoft.Azure.Devices.Client.ModuleClient.CreateFromEnvironmentAsync(TransportSettings) and TransportSettings are created as new Microsoft.Azure.Devices.Client.ITransportSettings[] { new AmqpTransportSettings(Microsoft.Azure.Devices.Client.TransportType.Amqp_Tcp_Only) }
which are doing based on the example here
Once we manually add the missing entry to the hosts file, we're able to connect to the edgeHub successfully.
Hi
I had a look in the internet and I could not find a clear answer in the following question:
Does Azure IoT Edge for Linux on Windows (eflow) supports Nvidia GPU for deploying CNN models for inference?
And if yes can you please guide if there is somewhere relevant documentation?
Many thanks
Is it possible to change the switch the EFLOW VM is using from the Default Switch to an External Switch? Typically I'd just update a VM's settings in Hyper-V Manager, but as of the GA release, EFLOW VM's are no longer included in Hyper-V Manager.
Our IoT Edge modules send and receive UDP broadcasts to communicate with devices on the host machine's network, so we want the VM to get an IP address on that network, rather than the 172.x.x.x IP address it gets from a default switch.
Connect-EflowVM results in following error:
ssh : GetConsoleMode on hOutputConsole failed with 6
At C:\Program Files\WindowsPowerShell\Modules\AzureEFLOW\AzureEFLOW.psm1:2213 char:17
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Expected to be able to connect to the EFLOW virtual machine.
Troubleshooting documentation is scarce.
Get-EflowLogs runs for a while but I have to stop it.
PS C:\WINDOWS\system32> Get-EflowLogs
[07/15/2021 20:48:12] Collecting Logs from Deployment...
[07/15/2021 20:48:12] Collecting configuration
[07/15/2021 20:48:12] Collecting 'Azure IoT Edge for Linux on Windows' event logs
[07/15/2021 20:48:12] Collecting virtual machine logs
-- stopped it here because it hung up ---
How do I find out where the problem is in connecting to the EflowVM?
Describe the bug
We've done several deployments which are all went well. The only thing is that we don't see the VM in the Hyper-V manager as a virtual machine.
To Reproduce
Steps to reproduce the behavior:
Deploy-Eflow
Go to Hyper-V manager and see an empty screen.
Expected behavior
We expected to see a virtual machine there. Eflow is working perfectly fine, no issue's there.
Windows Host OS (please complete the following information):
Additional context
Eflow is working (except for the CPU usage bug which will be fixed in about 6 days from now)
The first time I deployed Azure IoT Edge on my Windows PC was failed because of time out.
I cannot deploy it again as the error: 'Unsupported: Device not clean, Azure IoT Edge for Linux on Windows components detected (Azure IoT Edge PS module)'
There seems to be a problem when checking the SSH connection during running of Deploy-Eflow
Seems to be the same as reported in a comment for issue #41
Running (as admin):
Windows 10 Enterprise 1903 (18362)
AzureIoTEdge LTS 1.1.2106.1.msi
[07/09/2021 13:33:49] Querying IP and MAC addresses from virtual machine (XXXXXXX-EFLOW)
[07/09/2021 13:33:51] Testing SSH connection... [07/09/2021 13:35:57] SSH channel cannot be initialized due to fail to create known hosts file
[07/09/2021 13:35:57] Exception caught!!!
Getting a similar issue when calling Get-EflowLogs
[07/09/2021 13:53:48] Collecting virtual machine logs - Retrieving CloudInit logs from: "XXXXXXX-EFLOW" [07/09/2021 13:53:51] Exception caught!!!
What can be done for this?
First off... I'm not an expert with TPMs in general. so if any of my questions are truly ignorant please forgive me...
I'm installing the new EFLOW release and wanted to provision it using the DPSTpm. And I'm getting lost trying to follow the documentation with respect to this provisioning path. I have my laptop that I want to setup the EFLOW install as an Edge device in the hub.
I've installed Eflow and can connect to it. But now I want Provision. And the documentation points to "simulating a TPM For your device". My laptop has a real tpm. the Eflow instance should have a pass through to it. So shouldn't I be able to get an endorsement key/Registration key for Eflow with out using a simulated TPM? Is there a linux command that will show me this info?
Or am I thinking about how this should work all wrong?
Thanks!
I am facing '“Signer not trusted” error while deploying the EFLOW VM.
This issue is happening with both LTS (#AzureIoTEdge LTS 1.1.2106.1) and previous version(AzureIoTEdge).
Steps to Reproduce the error:
• Install AzureIoTEdge (any version).
• Connect the physical device to the internet.
• Enable Hyper-V and Containers.
• Enable the required services (like ICS, Windows Update, Server, Hyper-v..).
• Make sure windows update service is enabled and running.
• Deploy the Eflow VM using “Deploy-Eflow" command.
• Even though Windows Update service is enabled, there is warning message saying that “WARNING: Microsoft Update is not enabled. Please enable manually to ensure Azure IoT Edge for Linux on Windows stays up to date."
• Then deployment failed with error “Signer is not trusted (file: D:\AzureIoTEdge\wssdagent.exe, certificate Subject: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US) at line 4294"
Describe the Problem
It seems that step 7 is not valid anymore and documentation need to be updated: https://github.com/Azure/iotedge-eflow/blob/main/samples/interop-textmsg-consoleapp/Documentation/Configuring%20the%20IoT%20Edge%20Device.MD
Expected behavior
Customer reported an issue when following the steps, it seems that id_rsa is missing, instead they should use: Copy-EflowVmFile that replaced the scp procedure.
This error usually occurs when I try to removed the vm and redeploy it. This is not something I can consistently reproduce, but time seems to be a factor here. So I install eflow, use it as normal (I do quite a few Remove-EflowVm and Deply-Eflow), and then after some days this error starts coming up. Only way I have discovered to fix it is to reinstall Eflow.
Funny enough, it doesn't seem to be affecting anything. If I ignore it, it is still capable of registering to the iot hub, and even deploying correctly. No error logs from either edgeAgent or edgeHub, everything is coming back ok.
I will try this Friday to install Eflow and try to see how long it takes before it starts failing again.
[04/15/2021 08:51:33] Step 3: Configuring directories and virtual machine
[04/15/2021 08:51:33] Configuring directories
[04/15/2021 08:51:33] Associating wssdagent service with nodectl
- C:\iotedge\nodectl.exe security login --loginpath c:\programdata\wssdagent\nodelogin.yaml --identity failed to execute [Error: rpc error: code = Unauthenticated desc = Valid Token Required.]
False
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I don't expect it to fail.
Windows Host OS (please complete the following information):
With previous/current versions of iotedge, I was able to support passing physical comports from the host machine to a running module in a windows host/windows container or linux host/linux container setup.
I understand that iotedge linux-on-windows will be the recommended configuration moving forward.
Is it or will it be possible to pass a physical comport from the windows host to a running linux container?
If this is currently possible, how is this set up?
Thanks in advance!
Describe the Problem
With the latest version of AzureIoTEdge (AzureIoTEdge LTS 1.1.2106.0) Deploy-Eflow is failing with error “Failed to acquire IP address for the VM”.
This version of EFLOW is using Default-Switch.
We are using Windows 10 Version 1809. Issue is happening after applying Baseline security policies (Windows 10 Version 1809 and Windows Server 2019 Security Baseline).
Issue is not seen if we did not apply Baseline security policies.
Note: Earlier version of AzureIoTEdge (#1.0.x) is working fine even with Baseline security policies in place. This version is using External Switch.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Successful EFLOW deployment
Windows Host OS (please complete the following information):
Describe the bug
Deploy-Eflow seems to always append “-EFLOW” to Windows hostname. In my case, I have my Windows hostname in lowercase and the provisioned EflowVm actually has my Windows hostname changed to uppercase with “-EFLOW” appended to it!?
To Reproduce
Steps to reproduce the behavior:
$ProgressPreference = 'SilentlyContinue'
Invoke-WebRequest "https://aka.ms/AzEflowMSI" -OutFile $msiPath
Start-Process -Wait msiexec -ArgumentList "/i","$([io.Path]::Combine($env:TEMP, 'AzureIoTEdge.msi'))","/qn"
Deploy-Eflow
Expected behavior
Assuming a Windows 10 with hostname 'mywindows', after running Deploy-Eflow command, the provisioned EFLOW VM, when using Connect-EflowVm, should have the hostname as 'mywindows-eflow', instead of what it is - MYWINDOWS-EFLOW.
Screenshots
If applicable, add screenshots to help explain your problem.
Windows Host OS (please complete the following information):
Additional context
Add any other context about the problem here.
Question Summary
We are getting below error while running "iotedge check"
Detailed background
We have installed iotedge-eflow on azure windows VM. hostname is "VMName-EFLOW" in config.yaml, if we use IP address of Eflow VM still we get same error.
Are we missing any network settings?
here is iotedge check --verbose
Describe the bug
Whevener I try to run Deploy-Eflow, it fails with the following error:
set-acl : The process does not possess the 'SeSecurityPrivilege' privilege which is required for this operation.
I believe that this is an issue because it works whenever I try to create a deployment through Windows Admin Center without any issue.
This fails in step 3, during "Configuring directories". The process then continues and fails in step 4, during "Setting dynamically expanding virtual hard disk maximum size to 16 GB", and fails when it tries to instantiate the vm.
[03/10/2021 11:14:46] Setting dynamically expanding virtual hard disk maximum size to 16 GB
- Creating storage vhd (file: AzureIoTEdgeForLinux-v1-EFLOW)
- Creating vnic (name: XXX-EFLOWInterface)
- Instantiating virtual machine (name: XXX-EFLOW)
[03/10/2021 11:14:52] Error: Virtual machine creation failed!
[03/10/2021 11:14:52] Cleaning up...
- Virtual machine (name: XXX-EFLOW) does not exist or could not be removed, see error:
- C:\Program Files\Azure IoT Edge\nodectl.exe compute virtualmachine delete --name XXX-EFLOW failed to execute [Error: rpc error: code = NotFound desc = [Store] unable to filter entity from store for type[VirtualMachineInternal], FilterName[Name], FilterValue[XXX-EFLOW]: Not Found]
- Trying to remove vnic...
- Removing vnic (name: XXX-EFLOWInterface)
- Removing storage vhd (file: AzureIoTEdgeForLinux-v1-EFLOW)
[03/10/2021 11:14:52] Exception caught!!!
- C:\Program Files\Azure IoT Edge\nodectl.exe compute virtualmachine create --config "C:\Program Files\Azure IoT Edge\\yaml\cloudvm.yaml" failed to execute [Error: rpc error: code = Unknown desc = executePowershell failed with error new-vm : 'XXX-EFLOW' failed to add resources to 'XXX-EFLOW'. Failed to set folder permission. Verify that the host machine account has read/write access to the share. Failed to set security info. 'XXX-EFLOW' failed to add resources. (Virtual machine ID 6012C19A-0A28-4DD7-A926-10D482CE53D7) 'XXX-EFLOW' failed to set folder permission for C:\Program Files: Access is denied. (0x80070005). Verify that the 'XXX' machine account has read/write access to the share. (Virtual machine ID 6012C19A-0A28-4DD7-A926-10D482CE53 D7) Failed to set security info for 'C:\Program Files': 'Access is denied.'('0x80070005'). At line:24 char:10 + ... $vm = new-vm -Name $vmName -MemoryStartupBytes $memory -BootDev ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (:) [New-VM], VirtualizationException + FullyQualifiedErrorId : InvalidParameter,Microsoft.HyperV.PowerShell.Commands.NewVM : exit status 1] at line 2289
To Reproduce
Install Azure EFLOW
Execute the following command
Deploy-Eflow -acceptEula 'yes' -acceptOptionalTelemetry 'yes'
Expected behavior
I expect it to successfully create the linux vm, just like the "Windows admin center" does.
Windows Host OS (please complete the following information):
Question Summary
In using EFLOW (which we know is IOT Edge 1.1) are we constrained to using .net core 3.1 for our modules for now? Or can I spin my own docker file to point to the 5.0 sdks? or if I do that am I asking for more trouble than it's worth?
Detailed background
We are developing some modules and may need some of the SDK support in 5.0. But before I commit to using them I'm wondering if I shouldn't go there. As IOT Edge itself runs on .net core, does it matter how we configure our modules' dockerfiles?
Thanks!
Describe the bug
Hi, we try to setup EFLOW with Windows Server 2019 Datacenter on a Azure VM (Standard D8ds_v4 (8 vcpus, 32 GiB memory))
Based on https://docs.microsoft.com/en-us/azure/virtual-machines/acu this VM should have Hyper-threaded and capable of running nested virtualization. We also make a try with Standard_D4s_v4, with the same result.
on the step 7 (Windows Admin Center) in https://docs.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-on-windows?view=iotedge-2018-06&tabs=windowsadmincenter#create-a-new-deployment we get a
"Unsupported: Required feature 'Microsoft-Hyper-V-Hypervisor' is not available in this Windows edition. (Windows Server 2019 Datacenter as Azure VM)"
To Reproduce
we follow https://docs.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-on-windows
all steps in detail. Including to add Hyper-V
We check the Hyper-V Network Adapter. At that point the documentation is not clear.
We did not any changes on Virtual Machine Migration and Default Stores page.
The server runs including Hyper-V
Result of(Get-WmiObject Win32_OperatingSystem).OperatingSystemSKU = 8
Result of Get-WindowsOptionalFeature -Online -FeatureName \*Hyper-V\*
FeatureName : Microsoft-Hyper-V
DisplayName : Hyper-V
Description : Hyper-V
RestartRequired : Possible
State : Enabled
CustomProperties :
ServerComponent\Description : Hyper-V provides the services that you can use to create and manage
virtual machines and their resources. Each virtual machine is a virtualized computer system that
operates in an isolated execution environment. This allows you to run multiple operating systems
simultaneously.
ServerComponent\DisplayName : Hyper-V
ServerComponent\EventQuery : Virtualization.Events.xml
ServerComponent\Id : 20
ServerComponent\Type : Role
ServerComponent\UniqueName : Hyper-V
ServerComponent\Version\Major : 1
ServerComponent\Version\Minor : 0
ServerComponent\BestPractices\Model\Id : Microsoft/Windows/Hyper-V
ServerComponent\Deploys\Update\Name : Microsoft-Hyper-V-Offline
ServerComponent\Deploys\Update\Name : Microsoft-Hyper-V-Online
ServerComponent\Configuration\RootClassMofDefinition : class ServerComponent_HyperV
{ String VirtualSwitchNetworkAdapters[]; String
DefaultVirtualMachinePath; String DefaultVirtualHardDiskPath; BOOLEAN
EnableVirtualMachineMigration; String VirtualMachineMigrationAuthenticationType;
};
ServerComponent\SystemServices\SystemService\DefaultMonitoring : true
ServerComponent\SystemServices\SystemService\Name : vmmsFeatureName : Microsoft-Hyper-V-Offline
DisplayName : Hyper-V Offline
Description : Hyper-V Offline
RestartRequired : Possible
State : Enabled
CustomProperties :FeatureName : Microsoft-Hyper-V-Online
DisplayName : Hyper-V Online
Description : Hyper-V Online
RestartRequired : Possible
State : Enabled
CustomProperties :FeatureName : RSAT-Hyper-V-Tools-Feature
DisplayName :
Description :
RestartRequired : Possible
State : Enabled
CustomProperties :
mum2:customInformation\ServerComponent\Description : Hyper-V Management Tools includes GUI and
command-line tools for managing Hyper-V.
mum2:customInformation\ServerComponent\DisplayName : Hyper-V Management Tools
mum2:customInformation\ServerComponent\Id : 464
mum2:customInformation\ServerComponent\InstallWithParentByDefault : true
mum2:customInformation\ServerComponent\Parent : RSAT-Role-Tools
mum2:customInformation\ServerComponent\Type : Feature
mum2:customInformation\ServerComponent\UniqueName : RSAT-Hyper-V-ToolsFeatureName : Microsoft-Hyper-V-Management-Clients
DisplayName : Hyper-V Management Console
Description : Hyper-V Management Console
RestartRequired : Possible
State : Enabled
CustomProperties :
ServerComponent\Description : Hyper-V GUI Management Tools includes the Hyper-V Manager snap-in and
Virtual Machine Connection tool.
ServerComponent\DisplayName : Hyper-V GUI Management Tools
ServerComponent\Id : 301
ServerComponent\InstallWithParentByDefault : true
ServerComponent\Parent : RSAT-Hyper-V-Tools
ServerComponent\Type : Feature
ServerComponent\UniqueName : Hyper-V-Tools
ServerComponent\Version\Major : 1
ServerComponent\Version\Minor : 0
ServerComponent\Deploys\Update\Name : Microsoft-Hyper-V-Management-Clients
ServerComponent\Relationships\OptionalCompanionFor\Prerequisite : Server-Gui-Mgmt
ServerComponent\Relationships\OptionalCompanionFor\Type : RSAT
ServerComponent\Relationships\OptionalCompanionFor\UniqueName : Hyper-VFeatureName : Microsoft-Hyper-V-Management-PowerShell
DisplayName : Hyper-V PowerShell cmdlets
Description : Hyper-V PowerShell cmdlets
RestartRequired : Possible
State : Enabled
CustomProperties :
ServerComponent\Description : Hyper-V Module for Windows PowerShell includes Windows PowerShell
cmdlets for managing Hyper-V.
ServerComponent\DisplayName : Hyper-V Module for Windows PowerShell
ServerComponent\Id : 457
ServerComponent\InstallWithParentByDefault : true
ServerComponent\Parent : RSAT-Hyper-V-Tools
ServerComponent\Type : Feature
ServerComponent\UniqueName : Hyper-V-PowerShell
ServerComponent\Version\Major : 1
ServerComponent\Version\Minor : 0
ServerComponent\Deploys\Update\Name : Microsoft-Hyper-V-Management-PowerShell
ServerComponent\Relationships\OptionalCompanionFor\Type : RSAT
ServerComponent\Relationships\OptionalCompanionFor\UniqueName : Hyper-V
Windows Admin Center / Version 2103 / Build 1.3.2103.01006
Windows Host OS (please complete the following information):
Describe the Problem
I had no issue installing several version of EFLOW on a test system but when I try it at a customer, in a plant, on a test device but in a production environment (OT network), I receive this error: Failed to acquire IP address for virtual machine
I had the issue on the GUI install (through Windows Admin Center and using the PowerShell commands.
To Reproduce
Steps to reproduce the behaviour:
Expected behaviour
EFLOW installs successfully without error.
Windows Host OS (please complete the following information):
PowerShell output
`
[03/15/2021 12:16:51] Successfully created virtual machine
[03/15/2021 12:16:52] Querying IP and MAC addresses from virtual machine (BEMOED002-EFLOW)
[03/15/2021 12:17:46] Done.
[03/15/2021 12:17:46] Virtual machine hostname: BEMOED002-EFLOW-b283610e
[03/15/2021 12:17:46] ERROR: Could not retrieve MAC or IP address for virtual machine
[03/15/2021 12:17:46] Virtual machine may have failed to acquire an IP address.
Confirmation Required
Do you want to try a fallback to the internal default switch?
Note: This will not destroy the external switch. You can get to back to using the external switch with the command 'Connect-VMNetworkAdapter -SwitchName External -VMname BEMOED002-EFLOW' at any time
later
[Y] Yes [N] No [?] Help (default is "Y"):
[03/15/2021 12:21:03] Trying to associate virtual machine with the internal default switch
[03/15/2021 12:21:04] Waiting 5 seconds to allow virtual machine to acquire an IP address
[03/15/2021 12:21:09] Querying IP and MAC addresses from virtual machine (BEMOED002-EFLOW)
[03/15/2021 12:22:03] Done.
[03/15/2021 12:22:03] ERROR: Retry could not retrieve MAC or IP address for virtual machine (headless)
`
Just in case others run into this... AFter following the directions on installing EFLOW things were working great. Then "randomly" (or so I thought) I couldn't SSH into the EFLOW vm. After some investigation the VM wasn't getting an IP Address anymore. It turns out that I had setup the EFLOW vm with my laptop plugged in to a network.. then when I went wireless, the VM wasn't getting an ip address anmore. I went to Hypervisor and changed teh External adaptor (from the Switch Manager) to use my wireless and was back up and running again.
Not sure there is a better way to manage this but if anyone runs into this problem I thought I'd post this experience here in the hopes of saving someone from deciding they must reinstall EFLOW
Question Summary
How can you EXPOSE a PORT for a module that such that it is tied to the Windows host machines IP Address instead of the EFlowVM's IP address
Detailed background
It seems like under certain conditions the EFLOW VM IP address changes, this does not provide a reliable address to access modules running in the EFLOW VM.
Hello all,
I'm facing a problem with the SSH key management in Eflow VM
Sometime, the SSh-Eflow-Vm command asks me the password
I also tried with ssh command with the rsa file in argument with same result.
The VM seems to be well started (without any error)
Does it exists a workflowa that manage the SSH key between the host and the VM ?
Regards
Maxime
A windows IOT enterprise (1809) device needs to have data processed by a module before upload to the cloud. What are the options for getting that data to the module.
I have a Windows 10 IOT enterprise device (medical device). We are looking to start managing these devices via IOT Hub and are coming up to speed and trying to uncover best practices. We would like to be able to create modules for performing various activities. One such activity is to collect certain data from the main os. (e.g. log files of different types) In looking at EFLOW it seems that this is the future for IOT Edge running on a Windows 10 device.
I would like to understand how I pass these logfiles from teh Windows 10 OS where my software runs to control hardware etc to a module that will pre-process them before sending them up to the cloud
My initial going in was:
If I could collocate my log data on the Windows OS in a particular directory maybe I could mount this particular folder into my module's container and the module could periodically process and upload. However with EFLOW, it looks like I would have to first mount the folder from windows into the EFLOW VM before it could then be bound or mounted to a module container. In looking around I'm not sure this is supported? (I can think of good reasons why it wouldn't be but I just want to be sure)
So this leads me to "module->device adn device->module" messaging, I'd either have to send from the device (and write some code that uses the DeviceClient SDK) and define messages (as I found in teh Windows-iot-samples) to ask for (or send) the log data periodically, for requirement based reasons I don't have to do this constantly so the data becomes "batchy" and fairly large. but is this really the best/only approach?
We do have SQL Server installed in the Windows 10 os. Are modules in EFLOW allowed to talk to that?
Thanks for helping me learn.. this stuff is all very cool
Describe the Problem
Get 404 Using Windows Admin Center to create a new Azure IoT Edge for Linux on Windows deployment.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Expected a successful deployment.
Screenshots
If applicable, add screenshots to help explain your problem.
Windows Host OS
Additional context
Have successfully deployed EFLOW previously on this same machine. Completely uninstalled Windows Admin Center and Azure IoT Edge for Linux on Windows more than once before with no issues.
Describe the Problem
Cannot connect to IoT Edge via Windows Admin Center
To Reproduce
Steps to reproduce the behavior:
Connection error
RemoteException: The 'Verify-EflowVm' command was found in the module 'AzureEFLOW', but the module could not be loaded. For more information, run 'Import-Module AzureEFLOW'.
Windows Host OS (please complete the following information):
Additional context
I can connect to VM using Connect-EflowVm command from elevated PowerShell and see that IoT Edge works fine.
Before VM reboot WAC worked fine.
I have been exploring Azure IoT Edge for Linux on Windows, a feature that is still in preview.
Following the documentation I realized the device provisioning via DPS using TPM is not included.
Does anyone know why it is not included? Does it mean that TPM attestation mechanism is not supported?
I'm trying to copy a file to the Eflow VM... The new copy command allows me to copy the file.
My deployment.template.yaml is setup to bind a location from Eflow to the Module container
When I run my module I get permission denied on reading the file.
If I manually go into eflow (connect-eflowvm) and change the permissions of the file I created to 666 I my module can now read the file.
Is this something I shoudl be able to address in teh deployment yaml ? (Maybe I'm missing something in teh binding) or in how the module is launched? or is there something else?
Prior to GA this wasn't a problem so I'm not sure where to start
Thanks!
Describe the bug
When running sudo iotedge check --iothub-hostname "xxx", got error:
Socket Error - ScoketErroCode (ConnectionRefused) Connection refused : /var/lib/iotedge/mgmt.sock
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Windows Host OS (please complete the following information):
Additional context
Tried to change owner on this file with chown iotedge:iotedge /var/lib/iotedge/mgmt.sock but no success
Question Summary
I'm running the Windows10IOT samples example (console version) and getting a RemoteCertificateValidationCallback exception when I call the deviceClient.SetMethodHandlerAsync(...) call. How can I track down this problem?
Detailed background
I've followed ever step in the demo. I've generated the certs and SCP'd them to eflow.
Along the way I notice some things like :
Step 6: Configuring the edge device.
the instructions say to upload all the local files in the edgeCertificates\certs folder yet there is also a private folder and the instructions in the editiing the config.yaml point to needing the key.pem
When replacing the eflow hostname with the IP address.. the iotedge check command reports an error about the ip address but that seems not to be a causal problem (that I can tell as the device console client seems to make some sort of connection, if I use the hostname from my os I get a timeout on connect error)
Here is the error I see in the check command:
config.yaml has correct hostname - Error
config.yaml has hostname 10.0.13.80 but device reports hostname USWE-944TPV2-L-EFLOW-8cc59340.
Hostname in config.yaml must either be identical to the device hostname or be a fully-qualified domain name that has the device hostname as the first component.
I generated the certificate material by just following the instructions here (although you have to replace openssl.dockerfile with dockerfile):
https://github.com/Azure-Samples/IoTEdgeAndMlSample/tree/master/CreateCertificates
I guess the sample's cert generation instructions could benefit from more clarity. Like if I don't want to use a separate VM to just generate keys... a direct pointer to a couple of scripts would go a long way to eliminating ambiguity.
But overall any pointers on debugging this problem?
Thanks a ton!
In following the example instructions I ended up using .net core 5 (I think these samples were 3.1 originially... not sure that matters)
Ideally it would be nice to get down to the AmqpTransport layer and get more detailed logging as to what the problem is.
also when I am ssh-eflowvm and in the EFLOW Linux os... when I run sudo iotedge version I get iotedge 1.1.0 ... is 1.2 out? Should I be updating?
Thanks!
When running the Powershell commands for EFLOW (like Copy-EflowVmFile) if I'm running in Powershell 7 I see the following type of output:
PS C:\Temp> Copy-EflowVmFile -fromFile C:\Temp\packaged_metadata.json -toFile /home/iotedge-user/storage -pushFile
Write-EventLog: C:\Program Files\WindowsPowerShell\Modules\AzureEFLOW\AzureEFLOW.psm1:4611
Line |
4611 | Write-EventLog -LogName $LogName -Source $AppName -EventID $E …
| ~~~~~~~~~~~~~~
| The term 'Write-EventLog' is not recognized as a name of a cmdlet, function, script file, or
| executable program. Check the spelling of the name, or if a path was included, verify that the path is
| correct and try again.
I think I get it that Write-EventLog isn't available in PS7... due to it being cross platform... it seems this error doesn't cause the command to fail.. but should I always be able to count on that? Is there a way to get this error to go away? (either by pulling in some sort of extension or some flag?)
I'm writing a .net program (using .net 5 adn pulled the Powershell SDK in which is also based on PS7 and this same error shows up there too... Seeing errors always gives be agita ;)
Thanks!
There are many scenarios to consider for performance but it would be helpful to provide guidelines for general use cases. Such as migrating an app from a Windows container to an Eflow container. Intuitively one might think that linux interop overheads are higher compared to "pure" windows. Or general interop performance between a Windows UI and a backend workload running on windows vs Eflow.
A good place to mention or link to performance considerations might be here:
https://docs.microsoft.com/en-us/azure/iot-edge/iot-edge-for-linux-on-windows?view=iotedge-2018-06
Describe the bug
AFter rebooting our device (hardwired into network). (Device was provisioned with DpsTpm) we installed a module on it and were seeing the following log output:
nhandled exception. System.AggregateException: One or more errors occurred. (Error calling SignAsync: Module not found
caused by: Signing error occurred.
caused by: An error occurred in the key store.
caused by: HSM failure
caused by: HSM API failure occurred: 478, StatusCode: 404)
---> Microsoft.Azure.Devices.Client.HsmAuthentication.HttpHsmComunicationException: Error calling SignAsync: Module not found
caused by: Signing error occurred.
caused by: An error occurred in the key store.
caused by: HSM failure
caused by: HSM API failure occurred: 478, StatusCode: 404
at Microsoft.Azure.Devices.Client.HsmAuthentication.HttpHsmSignatureProvider.SignAsync(String moduleId, String generationId, String data)
at Microsoft.Azure.Devices.Client.HsmAuthentication.ModuleAuthenticationWithHsm.SafeCreateNewToken(String iotHub, Int32 suggestedTimeToLive)
at Microsoft.Azure.Devices.Client.AuthenticationWithTokenRefresh.GetTokenAsync(String iotHub)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttIotHubAdapter.ConnectAsync(IChannelHandlerContext context)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenInternalAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.Mqtt.MqttTransportHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass38_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(CancellationToken cancellationToken)
at Microsoft.Azure.Devices.Client.InternalClient.OpenAsync()
at LogModule.ModuleClientHandler.Init() in /app/LogModule/ModuleClientHandler.cs:line 31
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
did a Stop-EflowVm and Start-eflowvm and things started working.
This is on a windows 10 IOT Enterprise 1809 OS with the latest eflow installed.
What does this stacktrace mean?
Describe the bug
Whenever I try to Deploy Eflow through Azure Pipelines, I get the following error:
[03/24/2021 10:56:36] Step 5: Installing and verifying virtual machine software
- Installing and verifying required virtual machine features (username: iotedge-user)
[03/24/2021 10:56:38] Exception caught!!!
- Importing GPG key 0x3135CE90: at line 2324
[03/24/2021 10:56:38] Exception caught!!!
- Installing software in virtual machine failed (L762)
[03/24/2021 10:56:38] In order to attempt another deployment, please uninstall the Azure IoT Edge msi and start from fresh
To Reproduce
Deploy-Eflow -acceptEula 'yes' -acceptOptionalTelemetry 'no' -enableVtpm -vnetType 'ICS' -vnetName 'Default Switch'
Expected behavior
I would expect it to deploy it correctly.
Windows Host OS (please complete the following information):
Edition Windows 10 Enterprise
Version 20H2
Installed on 27-02-2021
OS build 19042.868
Experience Windows Feature Experience Pack 120.2212.551.0
Additional context
The pipeline agent is running as NT Authority\System. After it fails, I can actually use the pipeline to remove the VM successfully. I can't log in to the machine and remove it though. Whenever I try to do that, I get the following error:
C:\iotedge\nodectl.exe compute vm list -o tsv --query "[*].name" failed to execute [Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"] at line 2336
I have to remove the vm, uninstall Eflow, and then reinstall it to get it into a valid state again.
If I logon to the machine and install it with an admin user, it successfully installs.
If I then run the Verify-EflowVm in the pipeline, it fails with the following error:
- Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"NodeAgent\")" at line 2324
This is the same error that I get after trying to deploy with pipeline, and then logging into the machine and running verify-eflowvm (or any other command associated with eflow, for that matter).
I am guessing there are issues using eflow from different users, but what can I do about the original error? (Importing GPG key 0x3135CE90)
Describe the bug
When I try to run custom vision UWP application, the same error always happen around _mediaCapture.StartPreviewAsync().
My Windows 10 environment is "Windows 10 version Dev (OS Build 21376.1)".
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Preview of video capture is shown.
Screenshots
The error message is following.
System.Exception
HResult=0xC00D5212
Message=No suitable transform was found to encode or decode.
No suitable transform was found to encode or decode.
Source=System.Private.CoreLib
Stack Trace:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DeviceCustomVisionUWPApp.MainPage.d__42.MoveNext() in C:\customvision\DeviceCustomVisionUWPApp\MainPage.xaml.cs:line 435
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DeviceCustomVisionUWPApp.MainPage.d__40.MoveNext() in C:\customvision-test\customvision\DeviceCustomVisionUWPApp\MainPage.xaml.cs:line 403
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DeviceCustomVisionUWPApp.MainPage.<>c__DisplayClass47_0.<b__0>d.MoveNext() in C:\customvision\DeviceCustomVisionUWPApp\MainPage.xaml.cs:line 607
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DeviceCustomVisionUWPApp.MainPage.d__47.MoveNext() in C:\customvision\DeviceCustomVisionUWPApp\MainPage.xaml.cs:line 618
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DeviceCustomVisionUWPApp.MainPage.<Window_VisibilityChanged>d__36.MoveNext() in C:\customvision\DeviceCustomVisionUWPApp\MainPage.xaml.cs:line 289
Windows Host OS (please complete the following information):
Additional context
Other functions included in application, such as connecting to IoT Edge device, work correctly.
Question Summary
In previous versions of eflow, I was able to fetch the TPM endorsement key and registration id assigned to the VM by running the following command:
Get-EflowVmTpmProvisioningInfo
That command no longer exists - how can I fetch that information in GA?
Detailed background
I need this information in order to be able to create an individual enrollment in the DPS service.
Describe the bug
When using Windows Admin Center, the Command Line does not appear as a tool. Access to the Edge VM command line was very useful for development.
To Reproduce
Expected behavior
Left hand list displays Overview and Command Line
Windows Host OS:
Additional context
This problem occurs more often than not. Re-installing Azure IoT Edge via Windows Admin Center sometimes resolves the issue, sometimes not.
This issue was replicated on another Machine with the following Host OS:
Describe the bug
Got an error saying Could not resolve hostname file : no such host is known
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Should have copied files... the powershell command should be aware of the created VM.
Windows Host OS (please complete the following information):
Additional context
Add any other context about the problem here.
Describe the Problem
I am trying to follow the documentation for setting up auto-provisioning using TPM attestation. I have installed IoT Edge for Linux on Windows already using the Deploy-Eflow
command from the previous steps, but I am getting an error when I try to run the command Provision-EflowVM -provisioningType "DpsTpm" -scopeId "<scope id>"
with my scope id.
To Reproduce
Steps to reproduce the behavior:
Provision-EflowVM -provisioningType "DpsTpm" -scopeId "<scope id>"
with scope id from DPSExpected behavior
Expect to run successfully with no errors.
Windows Host OS
Describe the bug
Executing PS command Provision-EflowVM with parameter DpsX509 for provisionning type, passing certificate in arguments, cause error :
ssh failed to execute [] at line 3101
To Reproduce
Steps to reproduce the behavior:
Provision-EflowVm -provisioningType DpsX509 -scopeId "xxxx" -registrationId "xxxxx" -identityCertPath "C:\Certs\xxxx.pem" -identityPrivKeyPath "C:\Certs\xxxx.key"
Expected behavior
Successful provisionning with X.509 certificates
Screenshots
Windows Host OS (please complete the following information):
Additional context
Deploying Eflow with Windows Admin Console never worked but was successful first time with Powershell commands,
We are doing some testing with Eflow and somehow Eflow go into a state where:
I try to ssh into the VM in powershell and got a"The authenticity of host '172.18.13.248 (172.18.13.248)' can't be established.ECDSA key fingerprint is SHA256:XyRA7zu7SXgRR5hKs4S91qwlN2Cz/zIpHaYlDhNvoqs.
Are you sure you want to continue connecting (yes/no)?" prompt. when using the Connect-EflowVm command.
We had been just using the Copy-EflowVmFIle powershell command to move some files into EFlow.
Any thoughts of what could cause this state? and what one should do about it?
Question Summary
I would like to be able to select another internal vmswitch other then 'Default switch'. Whenever I do so, it tells me that the only supported switch is 'Default switch'.
Running
Deploy-Eflow -acceptEula 'yes' -acceptOptionalTelemetry 'no' -enableVtpm -vnetType 'ICS' -vnetName 'IoT EFLOW Nat Switch'
gives me
- C:\iotedge\nodectl.exe network vnet create --config "C:\iotedge\\yaml\cloudvnet.yaml" failed to execute [Error: rpc error: code = Unknown desc = VMMS only supports the 'Default Switch' ICS network
Unfortunately, for 'reasons', the 'Default switch' doesn't work for me. It is not capable of providing internet access to any of the other vm's in my setup. I've tried a lot of different things, but nothing seems to work. As soon though as I create my own NAT network, (see https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/setup-nat-network) and get my windows vm's to use it, it works.
Issue here though is that the 'Default switch' provides DHCP like functionality, and my own NAT network doesn't. That means that I have to open up the vm and manually assign a ip address, gateway address and DNS server.
I see two issues here:
I will be attempting to setup a small DHCP server in a seperate vm, and see if that works. If you have any other suggestions, I would really appreciate it.
I'm using latest version of WAC and its extensions all installed on the latest version of Windows Server 2019. Hyper-V has been installed and is running. When I deploy a new Azure IoT Edge for Linux on Windows on host (same PC) using default setting recommended by WAC, I ran into an error of "Error: Installation of virtual switch failed".
I already checked event viewer and below is what it shows:
Log Name: Microsoft-ServerManagementExperience
Source: UI
Date: 7/26/2021 12:35:46 PM
Event ID: 2
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: XXX
Description:
sessionId : 52a7f5bc-872c-4e95-ae8f-de9faa105039
eventId : Trace
level : Error
sourceName : Microsoft.AzureIoTEdge
source : EflowCreateWizard
timestamp : 1627295715557
message : DeployStepComponent
stack :
moreEventData :
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="UI" />
<EventID Qualifiers="0">2</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-07-26T10:35:46.173052100Z" />
<EventRecordID>2590</EventRecordID>
<Channel>Microsoft-ServerManagementExperience</Channel>
<Computer>XXX</Computer>
<Security />
</System>
<EventData>
<Data>sessionId : 52a7f5bc-872c-4e95-ae8f-de9faa105039
eventId : Trace
level : Error
sourceName : Microsoft.AzureIoTEdge
source : EflowCreateWizard
timestamp : 1627295715557
message : DeployStepComponent
stack :
moreEventData : </Data>
</EventData>
</Event>
When I check Hyper-V manager, I can see that a virtual switch has already been made after I installed Hyper-V. So I don't see any error with Hyper-V. But clearly WAC cannot create a virtual switch that it apparently needs. Would you please help me find the solution?
Hello,
I'm testing the new Eflow version of IotEdge.
I'm facing a subject. I need to share hosts informations with module.
Does exist a powershell script or a documentation to share a disk with the VM then with the docker.
Is it plan ?
If it doesn't exist, I will follow the standard way to share host disk with Hyper-V VM.
Thanks for your feedback,
Regards,
Maxime.
Describe the bug
Installing EFLOW on Windows 10 Pro Build 19041 is unsuccessful due to wssagent being unreachable.
There is no service called WSS Agent installed in WIndows
To Reproduce
Install EFLOW on said build
Expected behavior
Installation should be successful
Windows Host OS (please complete the following information):
Question Summary
I would like to give the eflow VM (the Linux machine hosted on Hyper-V) a fixed IP address to make it "play nice" with our network structure. Alternatively, is there a way to configure it so that it works as a kind of "NAT" and share the host machine's IP address?
Detailed background
Our IT service wants us to fix the IP address of the eflow VM so that it can have a correct access to the required resources and to the internet. By default the machine gets a DHCP address but it doesn't work correctly with out internal network structure. We therefore need to fix the IP for the VM.
Describe the bug
When I use the -installPackages
parameter on the deployment command, the IoTEdge installation fails. Moby-engine installation happens without problems.
To Reproduce
Steps to reproduce the behavior:
Deploy-Eflow -acceptEula yes -acceptOptionalTelemetry yes -mobyPackageVersion 19.03.11 -iotedgePackageVersion 1.1.0 -installPackages
[03/17/2021 04:33:27] Exception caught!!!
ssh -o LogLevel=ERROR -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i "C:\Program Files\Azure IoT Edge\id_rsa" [email protected] sudo dnf -y install azure-iotedge-1.1.0* failed to execute [Error: Unable to find a match: azure-iotedge-1.1.0*] at line 2336
[03/17/2021 04:33:27] Exception caught!!!
Installing software in virtual machine failed (L762)
[03/17/2021 04:33:27] In order to attempt another deployment, please uninstall the Azure IoT Edge msi and start from fresh
Expected behavior
Successfully install the iotedgePackageVersion 1.1.0
Windows Host OS (please complete the following information):
After about 20 minutes the CPU and Disk usage climbs very high and IoT Edge stops sending telemetry and PowerShell, Linux connection, etc. stop responding.
This issue of excessive CPU and Disk usage causing things to stop streaming and stop responding repeats even with a fresh installation of Windows 10 IoT Enterprise 2019 LTSC and EFLOW. The only thing that stays the same is the Azure DPS X509 enrollment with IoT Hub group deployment of 2 modules (simulated temperature sensor and streaming analytics reset module). This same deployment has worked on four other Ubuntu Linux computers and 2 Raspberry PIs for three months without issues.
Rebooting the computer restarts telemetry and all is calm for another 20 minutes or so until the excessive CPU and DIsk usage starts back up again.
WSSD Agent Service CPU usage 25% initially. Then 28% after 20 minutes. Not much change.
Vmmem CPU usage <1% initially. Then 25% after 20 minutes. Big change.
AzureIoT EdgeForLinux-v1-EFLOW.vhdx 48Kbps initially. Then 14 Mbps after 20 minutes then 160 Mbps after 25 minutes. Big change.
Is there a leak or something building up that is not properly managed behind the scenes?
I am not sure how to find out what is causing the high disk and memory usage and the effective shutting down of telemetry.
The Disk will keep running at nearly full capacity after an hour or more until rebooted even after telemetry stops sending. The CPU and Disk keep operating at very high capacity with the SSD LED on the front panel on solid.
What steps could be taken to troubleshoot? Since I cannot see logs when the computer is not responding, are we able to store logs on the computer to view when booting up and stopping iotedge from working? Would it be possible to eliminate the deployment, delete the containers and start with just two modules, the edgeAgent and the edgHub and see if the problem is related to the containers?
I will post screen clippings after submitting this issue.
Describe the Problem
After using DpsTpm as provisioning type a first time, I wasn't able to re-provision the same device.
Use cases: change the hub device name or change the hub device type
To Reproduce
Steps to reproduce the behavior:
Provision-EflowVM -provisioningType "DpsTpm" -scopeId "scopId"
Provisioning successful. iotedge service running.
Expected behavior
New IotEdge/IoT device created on the targeted hub
Windows Host OS (please complete the following information):
Additional context
It is happening in 2 different devices
Describe the bug
As per title, following the powershell instructions here: https://docs.microsoft.com/en-us/azure/iot-edge/how-to-install-iot-edge-on-windows?view=iotedge-2018-06&tabs=powershell#create-a-new-deployment
During the "Verifying expected Linux VM image" phase the following error occurs:
[06/24/2021 14:48:30] Verifying installation
- Verifying expected Windows host binaries
- Verifying expected Linux VM image
[06/24/2021 14:48:57] Exception caught!!!
- Signer is not trusted (file: C:\Program Files\Azure IoT Edge\AzureIoTEdgeForLinux-v1.cab, certificate thumprint: ) at line 4284
[06/24/2021 14:48:57] Exception caught!!!
- Verification of Azure IoT Edge for Linux on Windows installation failed (L897)
[06/24/2021 14:48:57] In order to attempt another deployment, please uninstall the Azure IoT Edge msi and start from fresh
Verification of Azure IoT Edge for Linux on Windows installation failed
Windows Host OS
Attempted on two machines with same result:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.