aws-iot-builder-tools / aws-greengrass-provisioner Goto Github PK
View Code? Open in Web Editor NEWSimplifies provisioning Greengrass Cores and building Greengrass Lambda functions
License: Apache License 2.0
Simplifies provisioning Greengrass Cores and building Greengrass Lambda functions
License: Apache License 2.0
CI will be nice once we get some pull requests in
Python users prefer requirements.txt to specifying the dependencies in the function.conf. This is a standard way of declaring dependencies and GGP needs to switch to this.
See exception below, looks like there may need to be retries here.
software.amazon.awssdk.services.ec2.model.Ec2Exception: The instance ID 'i-05071b42256ca7e1d' does not exist (Service: Ec2, Status Code: 400, Request ID: e49fc004-c7c1-4bf8-b810-b8e4e69b229e)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleErrorResponse(HandleResponseStage.java:115)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleResponse(HandleResponseStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:58)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:113)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:86)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:62)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240)
at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96)
at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120)
at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.ec2.DefaultEc2Client.describeInstances(DefaultEc2Client.java:9429)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicDeploymentHelper.execute(BasicDeploymentHelper.java:718)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicDeploymentHelper.execute(BasicDeploymentHelper.java:58)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.executeOrDisplayHelp(Operation.java:34)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.lambda$execute$1(Operation.java:22)
at io.vavr.control.Try.onSuccess(Try.java:648)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.execute(Operation.java:22)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$null$2(AwsGreengrassProvisioner.java:83)
at java.util.Optional.map(Optional.java:215)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$run$da6d192b$1(AwsGreengrassProvisioner.java:83)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.run(AwsGreengrassProvisioner.java:78)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.runProvisioner(AwsGreengrassProvisioner.java:56)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$main$249538b4$1(AwsGreengrassProvisioner.java:30)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.main(AwsGreengrassProvisioner.java:30)
Need to add pip to the container
Greengrass support in CloudFormation was recently launched
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-reference-greengrass.html
The provisioner should be able to output CloudFormation templates that use these features.
Before the project gets much (any?) larger we should switch to the most recent Java SDKs
The root CA in the docs is root.ca.pem. We used root-ca.pem. We cannot stand for an egregious discrepancy like this.
Detection of running Docker containers reports that a container is running if it exists, rather than if it is actually running. This needs to be fixed.
This exception is preventing Java functions from starting in Greengrass 1.7.1 today:
Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/services/lambda/runtime/LambdaLogger
1.8.0 is out, we need to support it
When launching java functions with --docker-launch
the deployment fails because of seemingly missing java8 executable.
java -jar ../aws-greengrass-provisioner/build/libs/AwsGreengrassProvisioner.jar -g xx -d deployments/cdd-skeleton.conf --docker-launch
docker exec -it xx cat /greengrass/ggc/var/log/system/runtime.log
[2019-01-14T14:27:29.675Z][INFO]-Starting worker arn:aws:lambda:eu-west-1:xx:function:xx-CDDSkeletonJava:19
[2019-01-14T14:27:29.675Z][ERROR]-Deployment xx of type NewDeployment for group xx failed error: unable to create worker process for arn:aws:lambda:eu-west-1:xx:function:xx-CDDSkeletonJava:19. cannot find executable java8 under any of the provided paths [/usr/bin /usr/local/bin]
[2019-01-14T14:27:29.675Z][INFO]-Updating status to Failure of deployment xx for group xx
However java 8 is actually present in the container, just not in the path where ggc is looking for it. A quick fix for example is to create a symlink ln -s /etc/alternative/java /usr/local/bin/java8
and then redeploying the function.
Add support for connectors, there appears to be a generic structure to the connector configuration. This needs to be a generic conf with a connector ARN and connector params so that you don't have to build something for each connector. All connectors have an ARN, I suspect many more will be coming.
EC2 launch only launches X86 instances, it should support ARM at least so we can do more testing
--ec2-launch
saves a lot of time when working with EC2 instances. This needs to support local systems too so that bootstrapping and rebuilding systems is faster.
After creating a container with java -jar ../aws-greengrass-provisioner/build/libs/AwsGreengrassProvisioner.jar -g xx -d deployments/cdd-skeleton.conf -c -a X86_64
and then trying to start it with docker run -p 8883:8883 greengrass:xx
it fails with:
Setting up greengrass daemon
Validating hardlink/softlink protection
Found cgroup subsystem: name=systemd
Found cgroup subsystem: pids
Found cgroup subsystem: hugetlb
Found cgroup subsystem: net_prio
Found cgroup subsystem: perf_event
Found cgroup subsystem: net_cls
Found cgroup subsystem: freezer
Found cgroup subsystem: devices
Found cgroup subsystem: memory
Found cgroup subsystem: blkio
Found cgroup subsystem: cpuacct
Found cgroup subsystem: cpu
Found cgroup subsystem: cpuset
failed to create overlay fs for container nosysRootfs operation not permitted
The Greengrass daemon process with [pid = 62] died
Maybe I'm just missing something from the startup command?
Currently it is not immediately clear how the provisioner should be used with maven based functions. Looking at BasicMavenBuilder, it seems to be very opinionated but gives little information in the way of javadocs or other documentation.
I'm sure we can live with the hard coded values for now, but it should be clear what files are expected where. Maybe just a simple sample directory structure with explanations?
.
├── deployments
│ ├── deployment.defaults.conf
│ ├── function.defaults.conf
│ └── my-function.conf
├── foundation
│ ├── 1-I-wonder
│ ├── 2-what-should
│ └── 3-be-here
└── functions
└── my-function
└── function.conf
While testing on Cloud9 I noticed that pip dependency installations fail. It appears that the system has the --user
option hard coded so it needs to be disabled.
Device Tester for GG doesn't support 1.8.1 as of today so this is on hold.
Adding a device to a Greengrass group no longer works because the list returned from the latest SDK is unmodifiable. It throws an unsupported operation exception.
On systems that need to use the no isolation the default isolation mode must be set. Otherwise the core will fail to start.
The provisioner's bootstrap script checks to see if node is present by executing which node
. If that executable is present it doesn't check to see if the nodejs6.10 symlink is present.
Instead of which node
it should really do which nodejs6.10
.
There are multiple scenarios that need to be tested with integration tests. They are a combination of the two ways GGP itself it can be run and the kinds of builds it can perform.
Ways to run GGP:
Kinds of builds it can perform:
For the first pass that gives us 12 combinations of builds we need to test. The first step is to do this with a simple script and then move it to something more formal if necessary.
The bootstrap script should detect when cgroups are not enabled and enable them.
It appears that a check to see if a reboot was required was put in the wrong place which broke this test.
Add support for secrets, also defining the secrets via secrets manager.
Released 2019-05-01
For example, if Gradle fails to download due to CDN issues an exception is thrown but the provisioner hangs forever.
If no SSH keys are found it doesn't make sense for GGP to launch an EC2 instance. It should detect that the keys are missing and then refuse to launch the instance.
While there exists support for creating a Greengrass Docker container it is not utilizing the official AWS Greengrass Docker containers. Those containers can be found in the Running AWS IoT Greengrass in a Docker Container documentation.
We should migrate to these Docker containers so we can support running Greengrass on Linux, Mac OS, and Windows.
For testing purposes it would be helpful to not have to manually launch EC2 instances. An option like —ec2-launch could build the configuration, launch an instance, scp the files to it, and run the bootstrap script.
Dagger just isn't worth it anymore
If the SSH connection to an EC2 instance fails due to the wrong SSH key being used an infinite loop occurs where the provisioner just reports "Auth fail" forever. This needs to bail out in the normal timeout period or it needs to fail immediately.
Being OCD I would like my account cleaned up of any resources that were created during the lab(s) that are not cleaned up when deleting the cloud formation stack(s).
See https://docs.aws.amazon.com/greengrass/latest/apireference/createfunctiondefinitionversion-post.html
Partial snippet of function configuration:
{
"Environment": {
"Execution": {
"IsolationMode": "GreengrassContainer|NoContainer",
"RunAs": {
"Uid": "integer",
"Gid": "integer"
}
}
}
}
Add support for Greengrass HSI and that support should work with Device Tester integration as well
The provisioner creates the group and installation package, fails to obtain SSH keys to setup the ec2 instance.
[INFO] BasicDeploymentHelper: Deployment successful
[ERROR] AwsGreengrassProvisioner: Is a directory
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Is a directory
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.handleProvisionerFailure(AwsGreengrassProvisioner.java:45)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$main$0(AwsGreengrassProvisioner.java:31)
at io.vavr.control.Try.onFailure(Try.java:633)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.main(AwsGreengrassProvisioner.java:31)
Caused by: java.io.IOException: Is a directory
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at java.nio.file.Files.read(Files.java:3105)
at java.nio.file.Files.readAllBytes(Files.java:3158)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.IoHelper.lambda$readFile$c72e3cae$1(IoHelper.java:64)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.IoHelper.readFile(IoHelper.java:64)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.IoHelper.readFileAsString(IoHelper.java:58)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.lambda$getPrivateKeyFilesForSsh$0(BasicIoHelper.java:45)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.getPrivateKeyFilesForSsh(BasicIoHelper.java:47)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.lambda$getJschWithPrivateKeysLoaded$8d6a4564$1(BasicIoHelper.java:113)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.getJschWithPrivateKeysLoaded(BasicIoHelper.java:113)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicIoHelper.getSshSessionTask(BasicIoHelper.java:143)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicDeploymentHelper.execute(BasicDeploymentHelper.java:742)
at com.awslabs.aws.greengrass.provisioner.implementations.helpers.BasicDeploymentHelper.execute(BasicDeploymentHelper.java:55)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.executeOrDisplayHelp(Operation.java:34)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.lambda$execute$1(Operation.java:22)
at io.vavr.control.Try.onSuccess(Try.java:648)
at com.awslabs.aws.greengrass.provisioner.interfaces.helpers.Operation.execute(Operation.java:22)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$null$2(AwsGreengrassProvisioner.java:80)
at java.util.Optional.map(Optional.java:215)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$run$da6d192b$1(AwsGreengrassProvisioner.java:80)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.run(AwsGreengrassProvisioner.java:75)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.runProvisioner(AwsGreengrassProvisioner.java:53)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.lambda$main$249538b4$1(AwsGreengrassProvisioner.java:30)
at io.vavr.control.Try.of(Try.java:75)
at com.awslabs.aws.greengrass.provisioner.AwsGreengrassProvisioner.main(AwsGreengrassProvisioner.java:30)
Need to add NodeJS and npm to the container
It can take a long time to pull down Gradle depending on network conditions. This could be sped up in Docker by:
curl -s "https://get.sdkman.io" | bash
sdk install gradle 5.4.1
1.8.0 has been superseded by 1.8.1
Device Tester support doesn't work in Docker. Errors indicate missing credentials.
Add support for running Device Tester
No dependencies get installed, throws a bunch of exceptions and even Python doesn't get installed so the core fails to start.
Some example functions have an environment variable called PORT
that is used to tell a pinned Greengrass Lambda function what port to listen on. When running in EC2 we should update the security groups so this function is accessible externally. When running in Docker we should map this port to the host.
Error looks like this with --ec2-launch
install: unrecognized option '--upgrade'
Try 'install --help' for more information.
install: missing destination file operand after 'AWSIoTPythonSDK'
Try 'install --help' for more information.
install: missing destination file operand after 'mxnet'
Try 'install --help' for more information.
install: missing destination file operand after 'opencv-python'
Try 'install --help' for more information.
Greengrass SDKs are still at 1.2 even though we've switched to Greengrass Core version 1.7.0. These should be updated soon.
The library that we are using (docker-java) is showing signs of dying. We should switch to a library that is more actively maintained like Spotify's Docker library.
Some operating systems obviously don't support systemd. There must be an option to turn it off.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.