Comments (8)
@jeffhollan So the default behavior would be to checkpoint as if the batch had completed even if there was a failure in the middle of the batch?
Is this because checkpointing after every message is expensive?
from azure-functions-kafka-extension.
@jeffhollan what happens if the batch fails even after the retries, do we have some concept of a dead-letter? worried about “losing data” here if we just discard the batch
from azure-functions-kafka-extension.
May help to see how we do in event hubs today. Deadlettering isn’t really a feature of event streams by design. My thinking here is we’d at least have a basic retry policy (e.g. 5xs) and just make sure we pass in the retry count to the active execution. I think we’d have to leave up to user to decide what they want to do if they hit an exception on the 5th time. A bit hesitant to automatically “deadletter” in another stream - at least for the initial version
https://hackernoon.com/reliable-event-processing-in-azure-functions-37054dc2d0fc
from azure-functions-kafka-extension.
Dead letter queue and circuit breaker pattern should be the 2 patterns we should look at to ensure we have a high scalability stream processing, isn't it?
from azure-functions-kafka-extension.
I don't think we'll pull off circuit breaker at the host level alone as it requires some coordination at the site level which the instance code wouldn't have. As mentioned above I'm a bit hesistant to automatically deadletter, but potentially an optional setting where you can specify a deadletter Kafka stream in your config somewhere and it will deadletter to that for you? But definitely not a must have. In fact this entire issue should be noted as a "should have" or even "nice to have," so ideally we don't spend too many cycles coding for it until other pieces that are more critical are completed.
from azure-functions-kafka-extension.
Marked as P1 and changed title to "should have" to reflect lower urgency.
from azure-functions-kafka-extension.
Opened new issue as P2 for "nice to have" to deal with dead lettering
from azure-functions-kafka-extension.
Close this issue, since we have
#122
Please discuss on it! :)
from azure-functions-kafka-extension.
Related Issues (20)
- Target scaler is not working (.NET) HOT 1
- Inconsistency between consumer and producer config HOT 1
- How to do exception handling in kafka output trigger if trigger fails to write to kafka HOT 2
- Distributed Tracing and kafka trigger
- Kafka trigger parameterization issues HOT 2
- Confluent Packages out of date causing runtime error
- Support for Bring your own certificates (.pfx)
- SchemaRegistryUrl attribute does not work with Java Azure Functions
- Enable support for rich datatypes in dotnet - isolated based apps using Kafka extension
- Kafka Trigger, Exception in Kafka subscriber, System.ObjectDisposedException at Confluent.Kafka.Impl.SafeKafkaHandle.ThrowIfHandleClosed
- Combine Key, Value, Header, TimeStamp in the same .Net object
- Make Kafka Output a first class typed citizen
- Support for librdkafka compression.codec and compression.type
- Kafka extension V3.9.0 issue HOT 1
- A lot of redundant logs produced HOT 1
- Azure Function (Java ) to Confluent Kafka output binding error
- Typescript Support for Functions v4?
- Upgrading .NET version for Extension Projects
- Confluent SDK upgrade
- Scale to Zero Support for Kafka Extension
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-functions-kafka-extension.