Comments (8)
@stephanos It looks like that setting is only relevant if we also used offset_commit_policy: commit_to_kafka_v2
. Currently, Kaffe commits offsets after the handle_messages
function is returned. I don't think it would be possible to support that option without other changes to Kaffe.
Am I understanding that correctly?
I'm looking at the Brod code here.
from kaffe.
Outside of this specific question, though, we should see which of those properties we could support (assuming I'm reading things right in this case).
from kaffe.
🤔I don't know enough about brod or kaffe to answer that question.
What I found was that in an integration test that sent a message to Kafka and waited for it to be acked in Kafka always took exactly 5s. I could see that the message was acked in our handler code but it was only sent to Kafka every 5s. This matches the default time for offset_commit_interval_seconds
exactly, so my naive assumption was that I can just lower that setting to 100ms during testing. But, to my surprise, it didn't do anything. It still took 5s. So - based on what I observed - I think this setting is useful and is being used right now.
Please note that this is happening when using supervisor(Kaffe.GroupMemberSupervisor, [])
. Based on what I understand from the source code, it would work correctly when using worker(Kaffe.Consumer, [])
(since the group_config
setting is passed on to brod). But we need the group member supervisor in our code.
from kaffe.
Very interesting! Do you want to try to pass it in and see if you get the results you expect?
Looking at the definition of consumer_group_config
here, I do think you're on to something.
I think you'd just need to grab the config like this:
defp group_config do
Kaffe.Config.Consumer.configuration.group_config
end
Having it verified in your test would ensure that it would do the right thing.
from kaffe.
I just tried it: I can see that my test went from ~5s to 1s! The ack is definitely sent faster now.
This is all I changed:
{:ok, pid} = group_coordinator().start_link(subscriber_name, consumer_group,
[topic], group_config(), __MODULE__, self())
and
defp group_config do
Kaffe.Config.Consumer.configuration.group_config
end
as well as
config :kaffe,
consumer: [
rebalance_delay_ms: 100,
offset_commit_interval_seconds: 1,
endpoints: [localhost: 9092]
],
...
from kaffe.
@stephanos Excellent! Would you be interested in submitting a PR? I think this is a nice addition.
from kaffe.
I will!
from kaffe.
closed, continued in #70
from kaffe.
Related Issues (20)
- Defining multiple handlers HOT 1
- worker_per_topic_partition with multiple topics HOT 1
- Examples not compatible with Elixir 1.10 or 1.11 HOT 2
- extract_der is giving error with SSL HOT 2
- Undefined function exponential_backoff HOT 10
- Offset doesn't get updated between runs and runs crash with OOM errors HOT 4
- async ack - lots of duplicate messages until I restart the application HOT 2
- Kaffe.Producer.produce_sync raises on timeout
- How to set kafka headers when publishing message HOT 1
- Invalid call to raise/reraise on brod/kpro error
- Add support for SCRAM mechanism in SASL authentication. HOT 1
- Module to help write ExUnit tests
- It's impossible to create 2 separate consumers for different topics
- Running mix with kaffe deps fails to download pc package from hex
- Wrong place for configuration
- Config dump on error leaks credentials HOT 1
- Endpoints configuration is weird for using ENV vars
- Missing documentation HOT 8
- Connecting to a TLS-based Kafka instance under AWS MSK? HOT 18
- Receives notification when rebalance in progress/assignments revoked HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kaffe.