Giter VIP home page Giter VIP logo

microgateway's Introduction

Microgateway Action for common microgateway patterns

Build Status godoc

Microgateway Action

This is a Microgateway Action which supports several common microgateway patterns , key ones highlighted below:

๐Ÿ”€ Conditional, content-based routing
๐Ÿ” JWT validation
โณ Rate limiting
๐Ÿ”Œ Circuit breaking

Quick Start

With Flogo CLI

The Flogo CLI takes a Flogo application defined in JSON and produces an executable application. The Flogo CLI can be installed from here. Next, follow the instructions here to build your first Microgateway Flogo application.

With Flogo API

The Flogo API allows developers to define Flogo applications in the Go programming language without using the Flogo CLI. You can get started by cloning this repo:

git clone https://github.com/project-flogo/microgateway.git

and then following the instructions here to build your first Microgateway Flogo application. Documentation for the Flogo Microgateway API can be found here.

Development

Testing

The commands supported include:

  • clean
  • build
  • generate
  • test
  • test-short

To run tests issue the following command in the root of the project:

go run build/build.go test

It cleans the cache, and runs all the tests in your directory. The tests should take ~2 mintues. To re-run the tests first run the following:

To skip the integration tests use the test-short command:

go run build/build.go test-short

To know the USAGE and the list of commands supported:

go run build/build.go help

Resource

Schema

The JSON Schema for the Microgateway resource can be found here.

Sections

Services

A service defines a function or activity of some sort that will be utilized in a step within an execution flow. Services have names, refs, and settings. Any Flogo activity works as a service. Services that are specific to a microgateway can be found here. Services may call external endpoints like HTTP servers or may stay within the context of the gateway, like the js activity. Once a service is defined it can be used as many times as needed within your routes and steps.

A service definition looks like:

{
  "name": "PetStorePets",
  "description": "Get pets by ID from the petstore",
  "ref": "github.com/project-flogo/contrib/activity/rest",
  "settings": {
    "uri": "http://petstore.swagger.io/v2/pet/:petId",
    "method": "GET"
  }
}

Steps

Each microgateway is composed of a number of steps. Each step is evaluated in the order in which it is defined via an optional if condition. If the condition is true, that step is executed. If that condition is false the execution context moves onto the next step in the process and evaluates that one. A blank or omitted if condition always evaluates to true.

A simple step looks like:

{
  "if": "$.payload.pathParams.petId == 9",
  "service": "PetStorePets",
  "input": {
    "method": "GET",
    "pathParams.id": "=$payload.pathParams.petId"
  }
}

As you can see above, a step consists of a simple condition, a service reference, input parameters, and (not shown) output parameters. The service must map to a service defined in the services array that is defined in the microgateway resource. Input key and value pairs are translated and handed off to the service execution. Output key value pairs are translated and retained after the service has executed. Values starting with = are evaluated as variables within the context of the execution. An optional halt condition is supported for steps. When the halt condition is true the execution of the steps is halted.

Responses

Each microgateway has an optional set of responses that can be evaluated and returned to the invoking trigger. Much like routes, the first response with an if condition evaluating to true is the response that gets executed and returned. A response contains an if condition, an error boolean, a code value, and a data object. The error boolean dictates whether or not an error should be returned to the engine. The code is the status code returned to the trigger. The data object is evaluated within the context of the execution and then sent back to the trigger as well.

A simple response looks like:

{
  "if": "$.PetStorePets.outputs.data.status == 'available'",
  "error": false,
  "code": 200,
  "data": {
    "body.pet": "=$.PetStorePets.outputs.data",
    "body.inventory": "=$.PetStoreInventory.outputs.data"
  }
}

Example Flogo JSON Usage of a Microgateway Action

An example of a basic gateway can be found here.

Example Flogo API Usage of a Microgateway Action

An API example can be found here.

microgateway's People

Contributors

akshaygadikar avatar lakshmimekala avatar lixingwang avatar mellistibco avatar nareshkumarthota avatar pointlander avatar raji-narayanan avatar rameshpolishetti avatar ykalidin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

microgateway's Issues

Make the DefaultHttpPattern more configurable

The DefaultHttpPattern needs more configuration options such as circuit breaker options. The pattern can be found here: https://github.com/project-flogo/microgateway/blob/master/internal/pattern/DefaultHttpPattern.json After changing this file the assets need to be generated with 'go generate' in the pattern directory. After adding more configuration options this code needs to be updated to use them:
https://github.com/project-flogo/microgateway/tree/master/examples/api/default-http-pattern
and
https://github.com/project-flogo/microgateway/tree/master/examples/json/default-http-pattern

Error seen on using flogo build command

Example Recipe Link: https://github.com/project-flogo/microgateway/tree/master/examples/json/basic-gateway
Steps to produce:
1)Clone the microgateway Repository:
git clone https://github.com/project-flogo/microgateway
cd microgateway/examples/json/basic-gateway
2)Create the flogo App:
flogo create -f flogo.json
cd MyProxy
3)Try the build command:
flogo build
Below error is seen:

Error in building /home/mashling/micro/src
Error: # github.com/project-flogo/flow/definition
../../go/pkg/mod/github.com/project-flogo/[email protected]/definition/definition_ser.go:87:52: not enough arguments in call to mapper.ConvertLegacyMappings
have (*mapper.LegacyMappings)
want (*mapper.LegacyMappings, resolve.CompositeResolver)

Unable to run the Microgateway API examples

Example recipe link: https://github.com/project-flogo/microgateway/tree/master/examples/api/async-gateway

Steps to reproduce:
1)Take the latest Flogo.
2)Clone the microgateway project
git clone https://github.com/project-flogo/microgateway
cd microgateway/examples/api/async-gateway
3)Run the main.go
go run main.go
Below error is seen:
github.com/project-flogo/microgateway
../../../../action.go:116:15: config.Data undefined (type *action.Config has no field or method Data)
../../../../action.go:117:31: config.Data undefined (type *action.Config has no field or method Data)

The same is observed with all API examples.

Move pattern.go outside of internal directory hierarchy

Other projects can't use custom pattern because pattern.go is under the internal directory hierarchy:
https://github.com/project-flogo/microgateway/blob/master/internal/pattern/pattern.go

pattern.go needs to be moved outside of internal/ to a new directory pattern/ that is in the project root. The files DefaultChannelPattern.json, DefaultHttpPattern.json, and assets.go can remain in internal/pattern/. Create a generate.go file in internal/pattern/ for the go generate directive that is in pattern.go, and remove the go generate directive that is in pattern.go

Unable to install custom-pattern

Example Recipe Link: https://github.com/project-flogo/microgateway/tree/master/examples/json/custom-pattern
Steps to produce:
1)Clone the microgateway Repository:
git clone https://github.com/project-flogo/microgateway
cd microgateway/examples/json/custom-pattern
2)Create the flogo App:
flogo create -f flogo.json
cd MyProxy
3)Try installing the custom-pattern
flogo install github.com/project-flogo/microgateway/examples/json/custom-pattern
Below error is seen:
panic: runtime error: index out of range

Support http:// and file:/// URIs in microgateway settings

Currently, only flogo resources are supported in the microgateway URI setting:
https://github.com/project-flogo/microgateway/blob/master/examples/json/basic-gateway/flogo.json#L75

'http://' and 'file:///' should be added as supported URI types. If a URI doesn't have a 'http://' or 'file:///' prefix then it is assumed to be a flogo resource. When the URI is 'http://' the microgateway definition will be fetched using http, and when the URI is 'file:///' the microgateway will be loaded from the local file system. Some form of caching should be used for the microgateway definitions.

Create channel pattern example

Channels allow one action to call another action. Create an example using this pattern:
https://github.com/project-flogo/microgateway/blob/master/internal/pattern/DefaultChannelPattern.json
The example will call another microgateway action (in the same flogo application/json file/go file) using channels. The called microgateway action can be as simple as a call to the log activity. Two triggers will be needed: a http trigger for calling the channel pattern, and a channel trigger for calling the second microgateway:
https://github.com/project-flogo/contrib/tree/master/trigger/channel

Async microgateway example and test

In theory we support the async execution of a microgateway action:
https://github.com/project-flogo/microgateway/blob/master/metadata.go#L7
https://github.com/project-flogo/microgateway/blob/master/internal/core/core.go#L78

Create an example and test for the async execution of a microgateway action. The microgateway action can be as simple as having a single call to the log service/activity. Fix any bugs related to the async execution of a microgateway action.

Error while creating microgateway app

The below error is seen while creating the microgateway app:
flogo create -f flogo.json

Error in installing github.com/project-flogo/microgateway@master
Warning: unable to install github.com/project-flogo/microgateway

While running the generated binary we are getting below error:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x6bec03]

goroutine 1 [running]:
github.com/project-flogo/core/app.New(0xc0000a9380, 0x80cb20, 0xc0000d4a00, 0xc0000b40d8, 0x1, 0x1, 0x0, 0x1, 0xc0001bbd40)
/home/ykalidin/go/pkg/mod/github.com/project-flogo/[email protected]/app/app.go:55 +0x413
github.com/project-flogo/core/engine.New(0xc0000a9380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f295cd996c0, 0x0)
/home/ykalidin/go/pkg/mod/github.com/project-flogo/[email protected]/engine/engineimpl.go:113 +0x398
main.main()
/home/ykalidin/go/src/github.com/project-flogo/microgateway/examples/json/basic-gateway/MyProxygomod/src/main.go:41 +0xa7

Different behaviour observed for Sqld Json Model

Sqld Example - Json model
1)Run the microgateway
2)Run the below curl command.
curl http://localhost:9096/pets --upload-file payload.json
3)The below response is seen always.
{"code":415,"message":"com.sun.jersey.api.MessageException: A message body reader for Java class io.swagger.sample.model.Pet, and Java type class io.swagger.sample.model.Pet, and MIME media type application/octet-stream was not found.\nThe registered message body readers compatible with the MIME media type are:\n*/* -\u003e\n com.sun.jersey.core.impl.provider.entity.FormProvider\n com.sun.jersey.core.impl.provider.entity.StringProvider\n com.sun.jersey.core.impl.provider.entity.ByteArrayProvider\n com.sun.jersey.core.impl.provider.entity.FileProvider\n com.sun.jersey.core.impl.provider.entity.InputStreamProvider\n com.sun.jersey.core.impl.provider.entity.DataSourceProvider\n com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$General\n com.sun.jersey.core.impl.provider.entity.ReaderProvider\n com.sun.jersey.core.impl.provider.entity.DocumentProvider\n com.sun.jersey.core.impl.provider.entity.SourceProvider$StreamSourceReader\n com.sun.jersey.core.impl.provider.entity.SourceProvider$SAXSourceReader\n com.sun.jersey.core.impl.provider.entity.SourceProvider$DOMSourceReader\n com.sun.jersey.json.impl.provider.entity.JSONJAXBElementProvider$General\n com.sun.jersey.json.impl.provider.entity.JSONArrayProvider$General\n com.sun.jersey.json.impl.provider.entity.JSONObjectProvider$General\n com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider\n com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider$General\n com.sun.jersey.core.impl.provider.entity.XMLListElementProvider$General\n com.sun.jersey.core.impl.provider.entity.XMLRootObjectProvider$General\n com.sun.jersey.core.impl.provider.entity.EntityHolderReader\n com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider$General\n com.sun.jersey.json.impl.provider.entity.JSONListElementProvider$General\n com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy\napplication/octet-stream -\u003e\n com.sun.jersey.core.impl.provider.entity.ByteArrayProvider\n com.sun.jersey.core.impl.provider.entity.FileProvider\n com.sun.jersey.core.impl.provider.entity.InputStreamProvider\n com.sun.jersey.core.impl.provider.entity.DataSourceProvider\n com.sun.jersey.core.impl.provider.entity.RenderedImageProvider\n","type":"unknown"}

Error log observed in Circuit breaker on tripping

When the circuit breaker trips, below error log is seen:
1547804715539743741 ERROR [flogo.microgateway] - error executing route: circuit breaker tripped
github.com/project-flogo/core/support/log.(*zapLoggerImpl).Error
/home/yogitha/go/src/github.com/project-flogo/core/support/log/zap.go:43
github.com/project-flogo/microgateway/internal/core.Execute
/home/yogitha/go/src/github.com/project-flogo/microgateway/internal/core/core.go:85
github.com/project-flogo/microgateway.(*Action).Run
/home/yogitha/go/src/github.com/project-flogo/microgateway/action.go:321
github.com/project-flogo/core/engine/runner.ActionWorker.Start.func1
/home/yogitha/go/src/github.com/project-flogo/core/engine/runner/worker.go:100

Circuit Breaker getting Tripped is expected and should not be treated as Error.
The log type should be Info and the above lines should not be logged.

Error log observed in Jwt for invalid token

For an invalid Jwt the below error log is seen:
ERROR [flogo.microgateway] - error executing route: token should be longer
github.com/project-flogo/core/support/log.(*zapLoggerImpl).Error
/home/yogitha/go/src/github.com/project-flogo/core/support/log/zap.go:43
github.com/project-flogo/microgateway/internal/core.Execute
/home/yogitha/go/src/github.com/project-flogo/microgateway/internal/core/core.go:85
github.com/project-flogo/microgateway.(*Action).Run
/home/yogitha/go/src/github.com/project-flogo/microgateway/action.go:321
github.com/project-flogo/core/engine/runner.ActionWorker.Start.func1
/home/yogitha/go/src/github.com/project-flogo/core/engine/runner/worker.go:100

When an invalid Jwt is sent the log type should be Info instead of Error.Also the above lines should not be logged.

Dependencies are broken

Hi,

github.com/project-flogo/microgateway depends on github.com/project-flogo/[email protected] which has been removed and replaced by github.com/project-flogo/[email protected].

Moreover when creating the application with latest Flogo CLI, github.com/project-flogo/microgateway is not imported and the application fails at runtime when loading the resource loader for the type microgateway. When adding manually the import, it is not resolved as described above.

No error is displayed with the attack payload in sqld

Sqld Example -API Model
1)Run the main.go file
2)Inject the attack payload with the below command
curl http://localhost:9096/pets --upload-file attack-payload.json
No error is displayed in the Response.Below is the response:
{"category":{"id":0,"name":"string"},"id":1,"name":" or 1=1 ","photoUrls":["string"],"status":"available","tags":[{"id":0,"name":"string"}]}

Additional tests

@ykalidin recommendations:

Circuit breaker Recipe:
1)Testcases with other test modes(b and c).
2)Testcases for changing the parameters(Period, Timeout, Threshold)

JWT Recipe:
1)A test case to verify invalid Issuer, subject and audience can be added.

Rate Limiter Recipe:
1)Modifying the rate limit parameters(3-M) testcase can be added.
2)Adding a Global token Testcase.

Not able to hit more than 5 clients simultaneously to gRPC-to-gRPC Streaming gateway

For gRPC-to-gRPC Streaming, when multiple clients hit the gateway only 5 client connections successful and are able to send and receive messages through gateway.The gateway assigns worker threads to first 5 client calls. The other clients send messages to gateway but are not assigned a worker request to proceed with streaming. These clients are waiting till the other client closes connections.

Steps to Reproduce
1)git clone https://github.com/project-flogo/grpc
cd grpc/examples/api/grpc-to-grpc
go build
2)Start proxy gateway:
./grpc-to-grpc
3)Start sample gRPC server.
./grpc-to-grpc -server
4)Run 6 clients in different terminals
./grpc-to-grpc -client -port 9096 -method bulkusers
5)It is seen that the 5 clients are assigned workers but 6th client is not assigned a worker and is kept in wait state until a previous client is killed.

Note:
It is observed that FLOGO_RUNNER_WORKERS default value is 5. The clients connecting is directly proportional to FLOGO_RUNNER_WORKERS value.
Also the same is not observed using the old mashling code.
https://github.com/TIBCOSoftware/mashling/
https://github.com/TIBCOSoftware/mashling-recipes

Add additional unit tests for circuit breaker

The current circuit breaker recipe testcases are not covering functionality of all modes.
The testcases should be modified specific to mode to check the exact functionality.

Below testcases can be added:
1)A test case for Tripping and not tripping the circuit breaker for each mode.Eg: Testcase for b mode need not have continuous connection failure calls for the circuit to be tripped.A success call can be added in between.
2)Default value check for Threshold and Period Testcase.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.