Comments (7)
rwdaigle
commented on July 21, 2024
@mrezentes can you provide evidence of the first message that you're not seeing? E.g., with another tool like kafkacat have you confirmed there are messages that haven't expired that you're not getting?
from kaffe.
sdball
commented on July 21, 2024
Checking locally I can't reproduce this.
starting at latest offset
$ curl http://core.spreedly.dev/v1/gateways/$CORE_DEV_GATEWAY/purchase.json \
-i \
-u $CORE_DEV_CREDENTIALS \
-H 'Content-Type: application/json' \
-d '{
"transaction": {
"credit_card": {
"first_name": "Joey",
"last_name": "Jones",
"number": "4111111111111111",
"verification_value": "423",
"month": "3",
"year": "2020"
},
"amount": "100",
"currency_code": "USD"
}
kafkacat
$ kafkacat -b kafka:9092 -t whitelist -o end -q -f "dev-services kafka %t topic, partition %p, offset %o: %k\n"
dev-services kafka whitelist topic, partition 8, offset 19: RG1twVbbnrIvmANtbrj3gDun5X6
dev-services kafka whitelist topic, partition 2, offset 20: 2eCLeFjj3xCkAMOv6TtenEL4VaM
dev-services kafka whitelist topic, partition 8, offset 20: RG1twVbbnrIvmANtbrj3gDun5X6
dev-services kafka whitelist topic, partition 2, offset 21: 2eCLeFjj3xCkAMOv6TtenEL4VaM
dev-services kafka whitelist topic, partition 2, offset 22: 2eCLeFjj3xCkAMOv6TtenEL4VaM
kaffe
start_with_earliest_message: false # default falsedev-services kaffe whitelist topic, partition 2, offset: 20: 2eCLeFjj3xCkAMOv6TtenEL4VaM
dev-services kaffe whitelist topic, partition 2, offset: 21: 2eCLeFjj3xCkAMOv6TtenEL4VaM
dev-services kaffe whitelist topic, partition 2, offset: 22: 2eCLeFjj3xCkAMOv6TtenEL4VaM
dev-services kaffe whitelist topic, partition 8, offset: 19: RG1twVbbnrIvmANtbrj3gDun5X6
dev-services kaffe whitelist topic, partition 8, offset: 20: RG1twVbbnrIvmANtbrj3gDun5X6
starting at earliest offset (first 10)
kafkacat
Limited to partition 0 to more easily match up with the kaffe output. Kaffe gets the messages in batches from each partition counting up, Kafkacat looks like it randomly pulls partitions to start with.
$ kafkacat -b kafka:9092 -t whitelist -o start -q -c 10 -e -p 0 -f "dev-services kafka %t topic, partition %p, offset %o: %k\n"
dev-services kafka whitelist topic, partition 0, offset 0: HpMp7uXXZ3AIWnAgQuscJR0WyET
dev-services kafka whitelist topic, partition 0, offset 1: aD4LNmnobrJhyMyopPYZsNNJoYQ
dev-services kafka whitelist topic, partition 0, offset 2: HpMp7uXXZ3AIWnAgQuscJR0WyET
dev-services kafka whitelist topic, partition 0, offset 3: Q8yRHhmJkMzIhhUc1oeAfnkAzQY
dev-services kafka whitelist topic, partition 0, offset 4: LEcKNjSYNTr9X1J83OntZRct0fE
dev-services kafka whitelist topic, partition 0, offset 5: Q8yRHhmJkMzIhhUc1oeAfnkAzQY
dev-services kafka whitelist topic, partition 0, offset 6: aD4LNmnobrJhyMyopPYZsNNJoYQ
dev-services kafka whitelist topic, partition 0, offset 7: aD4LNmnobrJhyMyopPYZsNNJoYQ
dev-services kafka whitelist topic, partition 0, offset 8: HpMp7uXXZ3AIWnAgQuscJR0WyET
dev-services kafka whitelist topic, partition 0, offset 9: Q8yRHhmJkMzIhhUc1oeAfnkAzQY
kaffe
start_with_earliest_message: true # default falsedev-services kaffe whitelist topic, partition 0, offset: 0: HpMp7uXXZ3AIWnAgQuscJR0WyET
dev-services kaffe whitelist topic, partition 0, offset: 1: aD4LNmnobrJhyMyopPYZsNNJoYQ
dev-services kaffe whitelist topic, partition 0, offset: 2: HpMp7uXXZ3AIWnAgQuscJR0WyET
dev-services kaffe whitelist topic, partition 0, offset: 3: Q8yRHhmJkMzIhhUc1oeAfnkAzQY
dev-services kaffe whitelist topic, partition 0, offset: 4: LEcKNjSYNTr9X1J83OntZRct0fE
dev-services kaffe whitelist topic, partition 0, offset: 5: Q8yRHhmJkMzIhhUc1oeAfnkAzQY
dev-services kaffe whitelist topic, partition 0, offset: 6: aD4LNmnobrJhyMyopPYZsNNJoYQ
dev-services kaffe whitelist topic, partition 0, offset: 7: aD4LNmnobrJhyMyopPYZsNNJoYQ
dev-services kaffe whitelist topic, partition 0, offset: 8: HpMp7uXXZ3AIWnAgQuscJR0WyET
dev-services kaffe whitelist topic, partition 0, offset: 9: Q8yRHhmJkMzIhhUc1oeAfnkAzQY
from kaffe.
mrezentes
commented on July 21, 2024
Assigning myself to add recreate documentation.
from kaffe.
mrezentes
commented on July 21, 2024
Once I complete the recreate, I will remove my label and assign back to @sdball.
In the event I cannot recreate, I will close.
from kaffe.
dtykocki
commented on July 21, 2024
I've never been able to reproduce this on a local setup. Will reopen if we encounter again.
from kaffe.
mrezentes
commented on July 21, 2024
I have no idea why I still have access to kaffe. 
I can no longer reproduce this locally.
from kaffe.
dtykocki
commented on July 21, 2024
It's open source now! 🎉
from kaffe.
Related Issues (20)
- Defining multiple handlers HOT 1
- worker_per_topic_partition with multiple topics HOT 1
- Examples not compatible with Elixir 1.10 or 1.11 HOT 2
- extract_der is giving error with SSL HOT 2
- Undefined function exponential_backoff HOT 10
- Offset doesn't get updated between runs and runs crash with OOM errors HOT 4
- async ack - lots of duplicate messages until I restart the application HOT 2
- Kaffe.Producer.produce_sync raises on timeout
- How to set kafka headers when publishing message HOT 1
- Invalid call to raise/reraise on brod/kpro error
- Add support for SCRAM mechanism in SASL authentication. HOT 1
- Module to help write ExUnit tests
- It's impossible to create 2 separate consumers for different topics
- Running mix with kaffe deps fails to download pc package from hex
- Wrong place for configuration
- Config dump on error leaks credentials HOT 1
- Endpoints configuration is weird for using ENV vars
- Missing documentation HOT 8
- Connecting to a TLS-based Kafka instance under AWS MSK? HOT 18
- Receives notification when rebalance in progress/assignments revoked HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kaffe.