swift-serverless / aws-lambda-swift-sprinter Goto Github PK
View Code? Open in Web Editor NEWAWS Lambda Custom Runtime for Swift with swift-nio 2.0 support
License: Apache License 2.0
AWS Lambda Custom Runtime for Swift with swift-nio 2.0 support
License: Apache License 2.0
Update async-http-client to version 1.0.0
Improve the Lambda Handler for NIO Implementation
Add new lambda handlers to rely directly on SwiftNIO 2.0 EventLoop.
public typealias SyncCodableNIOLambda<Event: Decodable, Response: Encodable> = (Event, Context) throws -> EventLoopFuture<Response>
public typealias SyncDictionaryNIOLambda = ([String: Any], Context) throws -> EventLoopFuture<[String: Any]>
public typealias AsyncCodableNIOLambda<Event: Decodable, Response: Encodable> = (Event, Context, @escaping (Result<Response, Error>) -> Void) -> Void
public typealias AsyncDictionaryNIOLambda = ([String: Any], Context, @escaping (DictionaryResult) -> Void) -> Void
Describe the bug
make create_lambda doesn't behave correctly.
To Reproduce
make create_lambda
Expected behaviour
Evaluate the performance of this library by comparing it with different languages
We need to compare the same example 'HelloWorld' by running it with Lambda using the same configuration but with different languages.
Swift, C#, Java, Go, NodeJS, Python, Ruby, PowerShell
Wanted solution
We need a CloudFormation or SAM template to build the test environment.
We need the code to run the performance test and obtain a csv or a JSON file with the results.
We need a tool to show the graphics of the result.
We should be able to change the example to evaluate the performances with a more complex one.
The solution should be automated.
Provide a Docker file to build Swift 5.0.3 on Docker amazonlinux:2018.03
The AWS Lambda Custom Runtime for Swift should be built on Docker amazonlinux:2018.03, the current solution uses the Swift docker images based on Ubuntu.
Solving this could help to:
The solution is a fix to this broken Dockerfile:
Dockerfile
:
FROM amazonlinux:2018.03 as builder
RUN yum update && yum -y group install "development tools" && \
yum -y install ninja-build clang python \
libuuid-devel libicu-devel libicu-devel libstdc++-devel \
libedit-devel libxml2-devel sqlite-devel swig python26-devel ncurses-devel \
pkgconfig openssl-devel systemtap-sdt-devel tzdata rsync git wget
#cmake
WORKDIR /tmp
RUN wget https://cmake.org/files/v3.15/cmake-3.15.0.tar.gz
RUN tar -xvf cmake-3.15.0.tar.gz
WORKDIR /tmp/cmake-3.15.0
RUN ./bootstrap
RUN make
RUN make install
RUN mkdir /swift-source
WORKDIR /swift-source
RUN git clone https://github.com/apple/swift.git
RUN ./swift/utils/update-checkout --clone
RUN ./swift/utils/update-checkout --tag swift-5.0.3-RELEASE
RUN ./swift/utils/build-script --preset=buildbot_linux installable_package=$ROOT_DIR/swift-package.tar.gz install_destdir=/tmp/swift
Error:
clang: error: unable to execute command: Killed
clang: error: linker command failed due to signal (use -v to see invocation)
[1683/3385] Building CXX object tools/llvm-lto2/CMakeFiles/llvm-lto2.dir/llvm-lto2.cpp.o
ninja: build stopped: subcommand failed.
./swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
Pull request details
The Docker file must be under the path docker/amazonlinux-2018.03
Is your feature request related to a problem? Please describe.
Support the new release of swift 5.1
Describe the solution you'd like
The solution must be compatible with swift 5.*
When the connection is slow it's hard to upload the package and the lambda
make upload_lambda_layer
could fail because the connection is slow.
TODO:
upload_lambda_layer_with_s3
to use the s3 bucket to update the Lambda layercreate_lambda_with_s3
to create the lambda with the code and layer on s3update_lambda_with_s3
to update the lambda from S3upload_build_to_s3
to upload only required files.Add support for Travis CI
The minimal support requires to test that each example builds correctly.
Add support for Visual Studio Code Debugging
The minimal support requires:
Note the documentation for this will be added later
We want to provide some context on the internal performance when the Lambda is Warm
HelloWorld example:
import AsyncHTTPClient
import Foundation
#if canImport(FoundationNetworking)
import FoundationNetworking
#endif
import LambdaSwiftSprinter
import LambdaSwiftSprinterNioPlugin
enum MyError: Error {
case unableToConvertString
}
class Lambda: LambdaHandler {
func commonHandler(event: Data, context: Context) -> LambdaResult {
guard let data = "Hello World!".data(using: .utf8) else {
return .failure(MyError.unableToConvertString)
}
return .success(data)
}
}
let lambda = Lambda()
public func log(_ object: Any, flush: Bool = false) {
fputs("\(object)\n", stderr)
if flush {
fflush(stderr)
}
}
do {
let sprinter = try SprinterNIO()
sprinter.register(handler: "helloWorld", lambda: lambda)
try sprinter.run()
} catch {
log(String(describing: error))
}
Note Data and Context are not used, but are being passed to the lambda function.
Using AWS Lambda with 256MB the warm performance is around ~1.19ms.
We want to evaluate where the execution time is spent
The main loop when the lambda is warm is contained in the following file:
https://github.com/swift-sprinter/aws-lambda-swift-sprinter-core/blob/master/Sources/LambdaSwiftSprinter/Sprinter.swift
public func run() throws {
while !cancel {
let (event, responseHeaders) = try apiClient.getNextInvocation()
counter += 1
if let lambdaRuntimeTraceId = responseHeaders.rhk(key: .runtimeTraceId) {
setenv(Context.AWSEnvironmentKey.xAmznTraceId.rawValue, lambdaRuntimeTraceId, 0)
}
guard let lambda = lambdas[handlerName] else {
try apiClient.postInitializationError(error: SprinterError.missingEnvironmentVariables(.handler))
return
}
let context = try Context(environment: environment, responseHeaders: responseHeaders)
let result = lambda.commonHandler(event: event, context: context)
switch result {
case .success(let outputData):
try apiClient.postInvocationResponse(for: context.awsRequestId, httpBody: outputData)
case .failure(let error):
try apiClient.postInvocationError(for: context.awsRequestId, error: error)
}
}
}
The getNextInvocation() is implemented in https://github.com/swift-sprinter/aws-lambda-swift-sprinter-nio-plugin/blob/master/Sources/LambdaSwiftSprinterNioPlugin/LambdaApiNIO.swift
public func getNextInvocation() throws -> (event: Data, responseHeaders: [AnyHashable: Any]) {
let result = try httpClient.execute(
request: _nextInvocationRequest,
deadline: nil
).wait()
let httpHeaders = result.headers
guard result.status.isValid() else {
throw SprinterNIOError.invalidResponse(result.status)
}
if let body = result.body,
let data = body.getData(at: 0,
length: body.readableBytes,
byteTransferStrategy: .noCopy) {
return (event: data, responseHeaders: httpHeaders.dictionary)
} else {
throw SprinterNIOError.invalidBuffer
}
}
Note that the Lambda remains in waiting for the next event while calling getNextInvocation() in the run loop until is killed or the timeout of 3600s is triggered.
We want understand if some optimisation can improve the performance
Reference Golang with a performance at warm of ~0.44 ms.
package main
import "github.com/aws/aws-lambda-go/lambda"
type Response struct {
StatusCode int `json:"statusCode"`
Body string `json:"body"`
IsBase64Encoded bool `json:"isBase64Encoded"`
}
func hello() (Response, error) {
return Response{
StatusCode: 200,
Body: "Hello world",
IsBase64Encoded: false,
}, nil
}
func main() {
lambda.Start(hello)
}
Investigation
It's required to better understand, in terms of performance, the AWS Lambda Custom Runtime with Swift.
It's required to provide solution to reduce the Initalization
time and the Overhead
time that you can monitor when X-Ray is enabled.
It could be helpful if you can comment this with ideas.
Provide a Docker file to build Swift 5.1 on Docker amazonlinux:2018.03
The AWS Lambda Custom Runtime for Swift should be built on Docker amazonlinux:2018.03, the current solution uses the Swift docker images based on Ubuntu.
Solving this could help to:
The solution is a fix to this broken Dockerfile:
Dockerfile
:
FROM amazonlinux:2018.03 as builder
RUN yum update && yum -y group install "development tools" && \
yum -y install ninja-build clang python \
libuuid-devel libicu-devel libicu-devel libstdc++-devel \
libedit-devel libxml2-devel sqlite-devel swig python26-devel ncurses-devel \
pkgconfig openssl-devel systemtap-sdt-devel tzdata rsync git wget
#cmake
WORKDIR /tmp
RUN wget https://cmake.org/files/v3.15/cmake-3.15.0.tar.gz
RUN tar -xvf cmake-3.15.0.tar.gz
WORKDIR /tmp/cmake-3.15.0
RUN ./bootstrap
RUN make
RUN make install
RUN mkdir /swift-source
WORKDIR /swift-source
RUN git clone https://github.com/apple/swift.git
RUN ./swift/utils/update-checkout --clone
RUN ./swift/utils/update-checkout --tag swift-5.1-RELEASE
RUN ./swift/utils/build-script --preset=buildbot_linux installable_package=$ROOT_DIR/swift-package.tar.gz install_destdir=/tmp/swift
Pull request details
The Docker file must be under the path docker/amazonlinux-2018.03
Test Lambda invocation with Lambci and LocalStack
Add an automated test to check that the code works correctly with amazonlinux.
Add commands to test code locally
make build_lambda_local
: Build the lambda and copy the artefacts under $(LOCAL_LAMBDA_PATH)
make invoke_lambda_local_once
: Invoke the Lambda locally once with Docker and LambCI on port 9001
make start_lambda_local_env
: Start the local environment with Docker and LambCI on port 9001
make invoke_lambda_local
: Invoke the lambda locally using the endpoint `http://localhost:9001'
Use of Docker Compose
Add command in Makefile to build and test the code locally by using docker-compose on port 9001.
Add commands for local environment
make start_docker_compose_env
: Start the docker-compose test environment
make stop_docker_compose_env
: Stop the docker-compose test environment
make test_lambda_local_output
: Verify the the output of the local invoke is the expected one.
Add the tests to Travis CI
Add tests for each Example
Describe the bug
While sample code of syncLambda runs properly. Sample code for asyncLambda: AsyncCodableLambda throws following error
aws lambda invoke --function-name HelloWorld --profile default --payload "fileb://Examples/HelloWorld/event.json" ./.build/tmp/outfile && echo "\nResult:" && cat ./.build/tmp/outfile && echo "\n"
{
"FunctionError": "Unhandled",
"ExecutedVersion": "$LATEST",
"StatusCode": 200
}
Result:
{"errorType":"Runtime.ExitError","errorMessage":"RequestId: 3278a4df-68cb-45d0-afe9-8870a675dae5 Error: Runtime exited without providing a reason"}
To Reproduce
Steps to reproduce the behavior:
Clone this repository and change following line
// sprinter.register(handler: "helloWorld", lambda: syncLambda)
// sprinter.register(handler: "helloWorld2", lambda: syncDictLambda)
// sprinter.register(handler: "helloWorld3", lambda: asyncLambda)
sprinter.register(handler: "helloWorld4", lambda: asyncDictLambda)
Inside Example/HelloWorld/Sources/HelloWorld/Main.swift
Follow all the steps to make docker build, layer, lambda from terminal
Run make invoke_lambda
Expected behavior
Sample code given in asyncDictLambda: AsyncDictionaryLambda or asyncLambda: AsyncCodableLambda should run without any error
Desktop (please complete the following information):
The Makefile
should allow to pass parameters from command line.
@Andrea-Scuderi - came over to your repo from your earlier update from tonisuter/aws-lambda-swift.
Your doc says AWS Linux does not support Swift ... but Ubuntu does.
Do you have a quick start to upload the Layer into Lambda to start using this without all the building required before use ?
Security Update
https://support.apple.com/en-us/HT210647
I am using this example aws server-less environment.
I am able to deploy the code with given example on aws lambda. Now to use this in our actual project I need to use postgres and redis with swift.
I am unable to find example where i can use standalone postgresql with server-side swift without using other server framework like Vapor.
Can you please help us in finding the right-way to integrate swift example with postgres and redis.
Thank you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.