I'm George, a software developer from ๐ง๐ท
My main interests are web development using the Python ecosystem, Open-source and Generative Art / Creative coding.
Check my ๐ personal website for more.
Asynchronous message dispatcher - Currently using asyncio and amazon SQS
License: MIT License
I'm George, a software developer from ๐ง๐ท
My main interests are web development using the Python ecosystem, Open-source and Generative Art / Creative coding.
Check my ๐ personal website for more.
Force handlers to receive at least 2 parameters: message, meta
"meta" wll be a dictionary containing the message metadata (initially, source name and/or route name)
Another option is to create a fixed message structure like:
message = {'content': message. 'meta': {}}
First approach seems a little better but will break existing implementations.
Create an isolated pipeline of chainable/reusable "translators"
https://github.com/rholder/retrying or https://github.com/danielfm/pybreaker
brainstorming
Requires python 3.5.2:
https://docs.python.org/3/library/asyncio-eventloop.html#error-handling-api
The old version of aiobotore returns directly to the client when you call the create_client
. Now with this last version the client needs to be instantiated with an async context manager, broken some pieces because they receive a context generator, not the botocore client:
old version:
https://github.com/aio-libs/aiobotocore/blob/904332b2ab0a3cf44c49af13ac04173a71f9e647/aiobotocore/session.py#L19
new version:
https://github.com/aio-libs/aiobotocore/blob/b86e8ce76c8282fd71b78770b00b7e23a6cf545a/aiobotocore/session.py#L43
Find out how to setup tox and execute the tests against multiple python versions without duplicating entries in circleci config.
It should change LoaferDIspatcher and LoaferRunner creation
The library supports sending errors to Sentry, but only using the deprecated Raven, and not the new sentry_sdk.
I am getting the following error:
exception (<class 'ValueError'>, ValueError("<loafer.ext.aws.message_translators.SQSMessageTranslator object at 0x10faf1610> failed to translate message={'MessageId': 'd20819f2-406f-4f96-a618-544b477b1a3d', 'ReceiptHandle': 'AQEBFK5WHeaLkDQ2nOLhsr7BBeyVIAsSGManlBfE8i8g5gI5Rh16QkTONbtE+cMgZSzlTnOlLAgzxab3ZSABycM9lQ2HOQtx/x6fc1c3y7VL9f9jwD0jxtU992sPUxtk2uf8I1wACULQh6TbnnK6hXbCB/4yZnWuP09xVlbQhyZTWOPjB//JKsjx6LXphHohmguw++kgzndVIL8FzW8alQACGxu3kXN5nMgFBLSNZYFXEkOJeS/0QaTS7qgK9uIqmFFSEZRGImrMv7CwhBRgFRhKPmrAQAFt3DJHlhyz27WXEuQZm36Jxfb/MMrFaD3rhjpf4p5ZvcnQ7WTA+0+fZ6fTs5nr7TsFCkZfoJS130OjIqxCUoHmo7npWum2Xyt0k4kZQcrwhKBrwlab72D7nnYrkQ==', 'MD5OfBody': '8ca1da029d8479102410a867ed661515', 'Body': '36|0|123456789'}"), <traceback object at 0x111d87870>) received
It seems we are not using prettyconf correctly with .env files, requiring the file to exists in the package root (loafer package).
It should work with .env files from the user project root.
Add enabled
boolean attribute to route, so we can restrict the route execution using the flag.
There's a change in aiobotocore
's version 0.12.0
that removes a loop
parameter from a get_session
method and it makes loafer
stop working.
Also related with #29
We should consider something like this:
Check the thread-pool executor for study and reference.
We should be able to configure the number of threads globally or by route ?
Line 24 in e878f27
My pytest shows a deprecation on line 24 (It isn't supposed to fix it immediatelly):
DeprecationWarning: 'with await lock' is deprecated use 'async with lock' instead
๐
fix instantiation of https://github.com/georgeyk/loafer/blob/master/loafer/ext/aws/bases.py#L21
Hello, I have a problem with the worker stopping receiving messages.
I will try to explain the situation. I have two SQS Queues. The loafer is v1.3.2.
The loafer, as well as queues, is set to receive every 10 messages (maximum) in short polling mode (WaitTimeSeconds = 0).
At AWS, at ECS, I have 6 consumers for each queue. About 60,000 messages are processed every 5 minutes when all consumers are fully operational.
The first worker with loafer usually receives a good amount of messages within minutes (approximately 80,000 messages for 5 minutes).
The handler function processes and can send between 1 and 12 messages to the second queue (using SQSHandler, publish method).
The second worker, also with loafer, is responsible for consuming the messages produced by the first worker handler.
After a while, only the consumers from queue 1 stop receiving messages. The strange thing is that no errors are logged, and the containers are not dropped. The cpu usage is 0% (from docker stats).
Has anyone had this problem?
Does the SQS have a limit?
To get around the problem, I am monitoring the ECS logs, and when there is no further message processing, I kill the task so that ECS uploads a new task and continues processing.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.