A service backed by Ktor, writing incoming HTTP requests to one or serveral topics of one or more message brokers (Kafka, Redis or Cloud Pub/Sub).
Sometimes a call to a webhook is only received once. Writing a HTTP request to a semi-permanent message broker log allows for repeated processing.
Defining a route with the same syntax used for Ktor routing
Example
title = "My routes config"
[[routes]]
path = "/status/mail"
topic = "mail"
verify = false
format = "proto"
methods = [ "POST", "PUT", "HEAD", "GET" ]
[[routes]]
path = "/api/{id}"
topic = "api_calls"
format = "raw_body"
verify = true
validateJson = true
jsonSchema = """
<put your JSON schema here>
"""
Routing is saved in files with .toml extension, and their location is set with the environment varaiable CONFIG_DIRECTORY. If this is not set it will default to /etc/config/.
Configuration consists of few tables:
Configuration is loaded on startup and then refreshed every 1 minute.
It can be retrieved either from a file or from kubernetes custom resources.
Configuration in a file is stored in TOML file format. See example configuration here. You can have several configuration files stored in /etc/config/ or in path defined in CONFIG_DIRECTORY environment variable.
Configuration in kubernetes requires creating custom resource defitnions first:
kubectl create -f leiaroute_crd.yaml
kubectl create -f leiasinkprov_crd.yaml
Then add your objects:
kubectl create -f object1.yaml
...
Custom resource definitions and sample object definitions YAML files can be found here.
Route defines where given request should be sent to.
path- is mandatory, HTTP URL path to matchtopic- is mandatory, topic where the message will be sentsink- is optional, name of sink where message will be sent, if not provided uses default sink.verify- is optional, default isfalse. Decides whether only request with a verified json web token is allowed. Requires ktor-jwt.format- is optional, default isprotoa format using protobuf defined in zensum/webhook-proto. Also available israw_body- writes HTTP body as is.methods- is optional, default is all seven HTTP verbs.response- is optional, HTTP response code which should be sent to the sender on success, default is204.validateJson- is optional, wether to validate body of the request as JSON before sending, default isfalse.204code is returned on success,400code on error.jsonSchema- is optional, validates request body against this JSON schema ifvalidateJsonis set totruecors- is optional, list of hosts to check incoming request's origin against for CORS. By default all traffic is allowed. If contains*then all hosts are allowed. When non empty,OPTIONSmethod is allowed implicitly.hosts- is optional, list of hosts to check incoming request's host field against.authenticateUsing- is optional, list of auth providers to verify request against
Leia needs at least one sink configured and marked as default.
name- is mandatory, name to identify the sink in routes configurationsinkfieldtype- is optional, one of these types:kafka,redis,gpubsub,null, default iskafkaisDefault- is optional, must be set totruefor one sink, default isfalseoptions- is optional, additional options to configure sink provider, it is a list of key/value pairs
For testing purposes you can use null sink type which does not forward messages and only logs them.
To set kafka hostname and port use following option in configuration:
host = "<hostname>:<port>"
To set up hostname and port for redis Pub/Sub use following options in configuration:
host = "<hostname>"
port = "<port>"
To use other project than the default for Cloud Pub/Sub one use following option in configuration:
projectId = "<your-project-id>"
Before writing to sinks in Cloud Pub/Sub you need to create them first:
gcloud pubsub topics list-subscriptions --project <your-project-id> <topic>
GOOGLE_APPLICATION_CREDENTIALS environment variable needs to point to your cloud service account key in json file. Create service account key.
This table is optional in configuration.
name- is mandatory, name to identify the auth provider in routes configurationauthenticateUsingfieldtype- is optional, one of these types:jwk,basic_auth,no_auth, default isno_authoptions- is optional, additional options to configure auth provider, it is a list of key/value pairs
Option basic_auth_users is a map of users and passwords.
Option jwk_configis mandatory, it is a map of key/values. The map must contain jwk_url and jwk_issuer keys.
CONFIG_DIRECTORY- location of toml files (default/etc/config/)PORT- port on which leia listens for requests (default80)KUBERNETES_SERVICE_HOST- hostname of kubernetes API (defaultlocalhost)KUBERNETES_SERVICE_PORT- port of kubernetes API (default8080)KUBERNETES_ENABLE- whether to read configuration from Kubernetes custom resources (defaulttrue). To disable set it tofalse.PROMETHEUS_ENABLE- if set totrueenables/metricsendpoint on port9090exposing metrics for prometheus (uses zensum/ktor-prometheus-feature), defaultfalseLOG_REMOTE_HOST- if set totrueenables logging of remote host (client address or host name), defaultfalse. Note the value may come from headers and could be falsified.
To check status of leia make request to http://leiahost/leia/health. Sample output:
sink redis1: ERROR
sink kafka1: OK
leia: ERROR
Error messages will be written to logs. You can also get them using verbose parameter in the request. Here is sample output for http://leiahost/leia/health?verbose:
sink redis1: ERROR redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host redis:6379
sink kafka1: OK
leia: ERROR
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent