aedenj / apache-flink-kotlin-starter Goto Github PK

View Code? Open in Web Editor NEW

6.0 6.0 0.0 159 KB

Kotlin 33.31% Shell 50.27% Makefile 15.23% Dockerfile 1.19%

apache-flink-kotlin-starter's People

Contributors

Stargazers

Watchers

apache-flink-kotlin-starter's Issues

Consider sending Kafka metrics to Prometheus

You have Flink, maybe Kafka too?

Setup Ingress for Flink UI on MiniKube

Why are sourcesets needed in the stock build.gradle?

The suggested build.gradle file, https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/datastream/project-configuration/#gradle, sets various properties on the sourceSets.

    sourceSets {
        main.compileClasspath += configurations.flinkShadowJar
        main.runtimeClasspath += configurations.flinkShadowJar

        test.compileClasspath += configurations.flinkShadowJar
        test.runtimeClasspath += configurations.flinkShadowJar

        javadoc.classpath += configurations.flinkShadowJar
    }

why exactly is this needed? Not doing so doesn't appear to have any immediate consequences.

exclude unnecessary resource files from said build

More recent images and/tools for working with Kafka locally other than Confluent.

Being able to work with kafka locally is immensely helpful. Other than confluent what are the best tools out there for this.

Demonstrate Checkpointing

Demonstrate in standalone job, cluster job and consider demonstration on minikbue

state.backend: filesystem
state.checkpoints.dir: file:///tmp/flink-checkpoints
state.savepoints.dir: file:///tmp/flink-savepoints
execution.checkpointing.interval: 120000 ms
execution.checkpointing.mode: AT_LEAST_ONCE

restart-strategy.fixed-delay.attempts: 3
restart-strategy.fixed-delay.delay: 10 s

Any reason to use anything but JUnit 5? E.g. Spek?

Add Readme Section on Working with Kafka

Describe

scripts
make commands
other utils

Provide an example of an operator test

Remove illegal reflective access warning

The following message is output when the Flink starts,

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/Users/aedenjameson/.gradle/caches/modules-2/files-2.1/org.apache.flink/flink-core/1.14.3/8a608d16c47ee4353d7c7d3f3fe1ce14a1489a96/flink-core-1.14.3.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

If possible, remove.

Parameterize Kube yaml, Dockerfile to reduce repeating values like the jar name

Why are two jars being produced in app/build/libs?

Allow Parameters to Create & Delete Topic Make Tasks

Add and/or Improve the Live Reload feature from the the java starter.

Setup Ingress for Kafka UI on MiniKube

Split out less frequently used Kafka tool like schema registry

get remote jmx working for multiple task managers

After setting up prometheus investigate using service discovery

Parameterize Create & Delete Topics Gradle Tasks

Currently the desired topic names are hardcoded in the gradle tasks. Can parameters be passed in? Or is this flexibility best achieved by just running the scripts/create-topics.sh script directly? It is nice having uniform mechanism for running developer tasks? Better yet, use Make as a task library instead.

Setup Consumer & Producer for Kafka on Kube

Setup ability to run a multi-broker Kafka

Ideally it can be started in a similar way to the Flink task managers using the --scale app=N option in docker compose

aedenj / apache-flink-kotlin-starter Goto Github PK

apache-flink-kotlin-starter's People

Contributors

Stargazers

Watchers

apache-flink-kotlin-starter's Issues

Recommend Projects

Recommend Topics

Recommend Org