Alternate Kafka Broker implementation.
License: Apache License 2.0
If you’re using Knative or if you like the project, please ★ this repository to show your support! 🤩
Eventing Kafka Broker Docs • Eventing Kafka Sink Docs • Eventing Kafka Channel Docs • Contributing • Help Wanted Issues • Open Knative Issues to Work On
| Sponsoring WG |
|---|
| Eventing |
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Problem
Jacoco code coverage report produces multiple reports, one per module (core, receiver, etc)
Persona:
Which persona is this feature for?
Contributor
Exit Criteria
Jacoco produces an aggregated code coverage report of all modules and the report is integrated with the coverage infrastructure.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
The generated reconciler adds <obj-namespace>/<obj-name>, so we don't need to do it. Instead, we can add the object ID that can be useful for debugging.
Problem
broker.spec.delivery has parameters to specify retries, backoffDelay, and backoffPolicy, we need to support them.
Persona:
Which persona is this feature for?
Event Consumer
Exit Criteria
Pass TestBrokerRedelivery test (https://github.com/knative/eventing/blob/master/test/e2e/broker_redelivery_test.go)
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
deadLetterSink/kind feature-request
/area data-plane
/area control-plane
Problem
The Broker spec allows specifying configurations specific to a given Broker object.
We need to clearly define what configurations expose for Kafka specific Brokers and their defaults.
Persona:
Which persona is this feature for?
Exit Criteria
Define Broker configurations
Additional context (optional)
Add any other context about the feature request here.
Problem
A short explanation of the problem, including relevant restrictions.
Persona:
Which persona is this feature for?
Exit Criteria
Reduce container image size ( < 300MB )
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1-4
Additional context (optional)
Problem
The Maven Wrapper is an excellent choice for projects that need a specific version of Maven (or for users that don't want to install Maven at all). Instead of installing many versions of it in the operating system, we can just use the project-specific wrapper script.
Persona:
Which persona is this feature for?
Contributor
Exit Criteria
./mvnw for running tests.Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
/good-first-issue
Describe the bug
We use those scripts for the release, so we need to use such flags.
Expected behavior
Use KO_FLAGS when executing ko resolve in test scripts
Knative release version
<= devel
Additional context
Usage in Eventing:
Problem
Now the receiver just tries to push the events to the topic, without checking if that broker exists.
Exit Criteria
An integration test that replies with 404 when you try to push a valid event to a non existing broker URL
Time Estimate (optional):
2
Additional context (optional)
The problem with receiver is that today it doesn't have knowledge of the existing brokers, aka it doesn't watch the famous data plane config map.
While in dispatcher side you solved the file watcher thread -> vertx communication deploying and undeploying verticles, here I have some ideas that might give an implementation hint:
AtomicReference shared between the watcher thread and the http verticle should suffice.updatebrokers.myapp, updates its internal set of brokers. On file watcher side, every time you receive an update from the file, you broadcast the new set of brokers to updatebrokers.myapp.Choose the simplest approach that fits better both for this issue and eventual next changes to the receiver code that requires knowledge of the available brokers
Problem
partitionkey is the cloudevents way of specifying events partitioning, we should set ProducerRecord key to the partitionkey value.
If there is no partitionkey extension the ProducerRecord key defaults to the empty string.
Persona:
Which persona is this feature for?
Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
Problem
We haven't implemented the finalizer in this PR: #134
Persona:
Which persona is this feature for?
*
Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
finalizer + unit tests
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
The underlying topic might be managed by an external entity (eg it might be created before the CO is created), we need to make sure that we don't delete the topic if it isn't managed by us. To do that we can use the annotation eventing.knative.dev/topic.owner in the status of the KafkaSink. Possible values of this annotation might be:
kafkasink-controller if it's managed by usexternal if it's notcc @matzew @slinkydeveloper if you would like to share your opinion on the additional context design/discussion.
Problem
Since a user can use the KafkaSink independently from the Broker, we can separate the control plane in a different YAML. In this way, a user can choose to use the Broker, the Sink, or both, and we use only one process to handle both.
Persona:
Which persona is this feature for?
Event Consumer
Event Producer
Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
- control plane (
eventing-kafka-control-plane.yaml) - required- sink data plane (
eventing-kafka-sink.yaml)- broker data plane (
eventing-kafka-broker.yaml)Users choose 2 or 3 or both.
Images remain 3 (but we package one per YAML).
Add a check to see if the data plane exists or not and reflect that in the status
/area control-plane
In a followup, Could you have a retry inside the script more than asking to the user to do so?
Originally posted by @slinkydeveloper in #11
/good-first-issue
Describe the bug
We need to include this project inside the knative release process: https://knative.slack.com/archives/CR2141CGY/p1595329443245300
Problem
We should create a test with all the objects we manage. This can prevent
regressions or raise bugs in the implementation or with the configurations
(eg RBAC config).
Test Plan
+---------+ +--------+ +---------+ +---------------------------+
| Broker +---->+ Trigger+---->+KafkaSink+-->+kafka consumer (test image)|
+----+----+ +--------+ +----+----+ +---------------------------+
| ^
| +--------+ |
+--------->+ Trigger+----------+
+--------+
(Use this diagram as test documentation)
kafka-consumer test image.Persona:
Which persona is this feature for?
*
Exit Criteria
An E2E test using Brokers, Triggers and KafkaSinks
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
Add any other context about the feature request here.
/test directory has some examples.Persona:
Which persona is this feature for?
Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
Add any other context about the feature request here.
Problem
Now the data-plane contract (the ConfigMap that configures the data-plane dispatcher and receiver) uses names for entities which are strictly related to knative eventing broker terminology (broker, triggers, etc). This is confusing because these building blocks are pretty much abstracted from the knative component they implement, in particular in relation to the work is happening here #134 and the eventual next plans to implement with this data-plane the KafkaSource component too.
Persona:
Which persona is this feature for? Knative Developer
Exit Criteria
Better names to replace Broker/Trigger in the config map data plane contract.
Additional context (optional)
As raised by @pierDipi, this change may break and bring outages while upgrading to the next version of this project #134 (comment). We need to develop a migration tool that gracefully performs the update #134 (comment).
Problem
The current developer guide documentation references the main knative install docs for instructions on how to create a k8s cluster for this project. However, the referenced knative docs does not include the creation of k8s clusters and assumes that user's already have one created.
Persona:
Which persona is this feature for?
Contributor
Problem
Extract an env var to configure client idle timeout to the dispatcher.
See: #137 (comment)
Persona:
Which persona is this feature for?
System Integrator
Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
/area data-plane
/good-first-issue
Problem
In order to get started contributing a developer needs to generate code.
We can keep generated code by protoc in the repository.
Persona:
Which persona is this feature for?
Contributor
Exit Criteria
hack/dev_data_plane_setup.shhack/update-codegen.shhack/verify-codegen.shdata-plane/.gitignore)Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
/assign
Increase code coverage for control-plane/pkg/reconciler/trigger/trigger_lifecycle.go
code coverage for control-plane/pkg/reconciler/trigger/trigger_lifecycle.go > 70%
https://testgrid.knative.dev/eventing-kafka-broker#coverage
/area control-plane
/kind enhancement
/good-first-issue
Problem
Now the parent data plane pom contains a lot of dependencies that should be specified in child poms.
These should be removed but the parent pom should have the <dependencyManagement> definition to import the Vert.x BOM (https://github.com/vert-x3/vertx-stack#bills-of-materials-maven-bom) and the other dependencies versions (slf4j, junit5, ...).
Then the child modules just need to import the dep without explicating the version (this will keep the version aligned across all modules)
Persona:
Which persona is this feature for?
Contributor
Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Problem
The Broker spec indicates that a Broker implementation SHOULD support Cloudevents Distributed Tracing extension: https://github.com/knative/eventing/blob/master/docs/spec/broker.md#observability
Persona:
Which persona is this feature for?
All of them can benefit from this.
Exit Criteria
Pass the Broker tracing conformance test (https://github.com/knative/eventing/blob/master/test/conformance/broker_tracing_test.go), it needs some refactoring (ie removing the channelTestRunner from parameters) before we can import it.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
3
Additional context (optional)
/kind data-plane
/kind feature-request
Problem
The checkstyle plugin for intellij doesn't really work well and it's a mess to configure the code formatting. Document how to do it.
Persona:
Which persona is this feature for? Developers
Exit Criteria
Some doc that explains how to configure the code formatting
Time Estimate (optional):
1
Problem
Enable maven enforcer plugin to check transitive dependencies convergence:
Persona:
Which persona is this feature for?
Contributor
Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Problem
The initial design document can be viewed only by knative-dev members. We would like to have a summary in the repository in INTERNALS.md
Persona:
Which persona is this feature for?
Exit Criteria
A summary of the initial design document.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1-2
Additional context (optional)
Problem
We need to update to CloudEvents SDK milestone 2, major changes that impact us are related to the Vert.x 4 support.
Persona:
Which persona is this feature for?
Exit Criteria
Upgrade to cloud events sdk milestone2 and all tests pass.
Time Estimate (optional):
How many developer-days do you think this may take to resolve?
1
Additional context (optional)
/area data-plane
Unless you want to trigger a lot of people out there 😆
kafka -> Kafka
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.