Giter VIP home page Giter VIP logo

serverless-offline-msk's Introduction

serverless-offline-msk

A serverless offline plugin that enables AWS MSK events.

serverless npm version License: MIT

Note

Something not working? Open an issue. Pull requests welcome.

Getting Started

Example

See the ./examples folder for fully built out examples.

Installation

Install the plugin:

# NPM or Yarn examples below:

npm install --save-dev serverless-offline-msk
yarn add -D serverless-offline-msk

In your serverless.yml we'll want to add the plugin after serverless-offline:

plugins:
  - serverless-offline
  - serverless-offline-msk

Plugin Configuration

custom:
  serverless-offline-msk:
    allowAutoTopicCreation: true # If we can create topics in kafka automatically
    brokers: # list of brokers to use for kafka
      - "kafka:19092" # this example here is from a docker compose example in examples

Function Event Configuration

The supported options are the same that are supported by serverless and MSK. See the serverless docs for more information.

functions:
  consumerName:
    events:
      - msk: # list of MSK events.
          arn: 'arn:*' # requried! in AWS you would use a real MSK ARN, but this can be anything locally
          topic: some_topic_name # required! whatever the topic name is.
          batchSize: 1
          startingPosition: LATEST
          enabled: false # this will turn it off

LICENSE

See LICENSE for more info.

serverless-offline-msk's People

Contributors

lukevers avatar

Stargazers

avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

brianmaldo

serverless-offline-msk's Issues

Consumer example in python

Hello,
I am looking for an consumer example in Python using that plugin.
Could you please give me an example?
Thanks

WARNING: Never try to work with this plugin

I tried to integrate AWS MSK with serverless using this plugin and after trying 3+ days, I got nothing.
NOTE to author: I highly appreciate creating such plugins but if its not maintained at least give a heads-up saying its not man. Totally frustrated using this plugin

My serverless.yml file

EventsProcessor:
  handler: src/handlers/events.handler
  layers:
    - {Ref: PythonRequirementsLambdaLayer}
  timeout: 300
  events:
    - msk: 
        arn: '<arn of AWS MSK>'
        topic: <topic name>
        batchSize: 100
        startingPosition: LATEST
        enabled: true # this will turn it off

custom.yml

serverless-offline-msk:
  allowAutoTopicCreation: true # If we can create topics in kafka automatically
  brokers: # list of brokers to use for kafka
    - "AWS MSK broker 1"
    - "AWS MSK broker 2"

Error if lambdaPort differs from the default 3002

If you have are using lambdaPort that differs from the default 3002 value then it will error out with the following error message

INVOKE ERR UnknownEndpoint: Inaccessible host: `localhost' at port `3002'. This service may not be available in the `us-east-1' region.

It is caused by this line here

Perhaps it should get this value dynamically?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.