cloudera / flink-tutorials Goto Github PK

License: Apache License 2.0

Java 100.00%

flink-tutorials's Introduction

Tutorials for Flink on Cloudera

This repo contains reference Flink Streaming applications for a few example use-cases. These examples should serve as solid starting points when building production grade streaming applications as they include detailed development, configuration and deployment guidelines.

We suggest to refer the tutorials in the following order even though they are designed to be standalone references:

Stateless Monitoring Application

The flink-simple-tutorial application demonstrates some basic capabilities of the DataStream API to build a simple monitoring application with alerting capabilities. It displays the essentials of Flink applications alongside best practices for setting up a robust logging configuration using Kafka.

Stateful Transaction and Query Processor Service

The flink-stateful-tutorial application implements a production grade stateful service for handling incoming item transactions, while also exposing query capabilities.

We dive deeper into structuring streaming application code, state handling and resource configuration. We also present a detailed set up of Kafka data sources and sinks for scalability and discuss validation of our pipeline before deployment.

Flink Security Showcase Application

The flink-secure-tutorial application demonstrates Flink's security features for applications intended to run in secured CDP environments. It covers Kerberos authentication and TLS encryption for HDFS and Kafka connectors.

Flink Quickstart Archetype

The flink-quickstart-archetype is a maven archetype for generating application skeletons specificly for Flink on Cloudera.

Flink Quickstart Skeleton

The flink-quickstart-skeleton is a maven project with the same content the archetype generates. This is for users who do not want or are unable to use the archetype.

flink-tutorials's People

Contributors

Stargazers

Watchers

flink-tutorials's Issues

Failed to build flink-secure-tutorial

Hi, I face this errors while build flink-secure-tutorial

$ cd flink-secure-tutorial
$ mvn clean package
Downloading from cloudera: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/flink/flink-connector-files/1.16.2-csa1.11.0.0/flink-connector-files-1.16.2-csa1.11.0.0.pom
Downloaded from cloudera: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/flink/flink-connector-files/1.16.2-csa1.11.0.0/flink-connector-files-1.16.2-csa1.11.0.0.pom (16 kB at 13 kB/s)
Downloading from cloudera: https://repository.cloudera.com/artifactory/cloudera-repos/junit/junit/4.13.1/junit-4.13.1.pom
Downloaded from cloudera: https://repository.cloudera.com/artifactory/cloudera-repos/junit/junit/4.13.1/junit-4.13.1.pom (25 kB at 24 kB/s)
Downloading from cloudera: https://repository.cloudera.com/artifactory/cloudera-repos/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom
Downloaded from cloudera: https://repository.cloudera.com/artifactory/cloudera-repos/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom (766 B at 762 B/s)
Downloading from cloudera: https://repository.cloudera.com/artifactory/cloudera-repos/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom
Downloaded from cloudera: https://repository.cloudera.com/artifactory/cloudera-repos/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom (2.0 kB at 1.9 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:39 min
[INFO] Finished at: 2024-02-20T12:19:15+07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-secure-tutorial: Could not resolve dependencies for project com.cloudera.flink:flink-secure-tutorial:jar:1.16.2-csa1.11.0.0: Failed to collect dependencies at org.apache.flink:flink-cloudera-registry:jar:1.16.2-csa1.11.0.0 -> org.everit.json:org.everit.json.schema:jar:1.14.0: Failed to read artifact descriptor for org.everit.json:org.everit.json.schema:jar:1.14.0: The following artifacts could not be resolved: org.everit.json:org.everit.json.schema:pom:1.14.0 (absent): Could not transfer artifact org.everit.json:org.everit.json.schema:pom:1.14.0 from/to cdh-build (https://cloudera-build-us-west-1.vpc.cloudera.com/s3/build/43829886/cdh/7.x/maven-repository): Read timed out -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

Please kindly help with this issue. Thank you.

Failed to generate project skeleton

Failed to generate project skeleton with these error message:

$ mvn archetype:generate \
  -DarchetypeGroupId=com.cloudera.flink \
  -DarchetypeArtifactId=flink-quickstart-archetype \
  -DarchetypeVersion=1.16.2-csa1.11.0.0
  
  ...
  
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: flink-quickstart-archetype:1.16.2-csa1.11.0.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.cloudera.flink
[INFO] Parameter: artifactId, Value: sample-project2
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: com.cloudera.flink
[INFO] Parameter: packageInPathFormat, Value: com/cloudera/flink
[INFO] Parameter: package, Value: com.cloudera.flink
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: groupId, Value: com.cloudera.flink
[INFO] Parameter: artifactId, Value: sample-project2
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  31.221 s
[INFO] Finished at: 2023-11-07T17:31:59+07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.2.1:generate (default-cli) on project flink-quickstart-archetype: Unable to add module to the current project as it is not of packaging type 'pom' -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Kerberos KafkaClient issue

Hi,

we have a kerberized, SSL configured Cluster.
"java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /data3/yarn/nm/usercache/xuser/appcache/application_1623160563820_0414/jaas-1578655284572927100.conf
How could I provide this param?
yD security.auth.login.config=="""KafkaClient {...
and
kafka.sasl.jaas.config== "com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;" \

I already tried. KafkaClient in jaas.config too.

BR,
Gabor

cloudera / flink-tutorials Goto Github PK

flink-tutorials's Introduction

Tutorials for Flink on Cloudera

Stateless Monitoring Application

Stateful Transaction and Query Processor Service

Flink Security Showcase Application

Flink Quickstart Archetype

Flink Quickstart Skeleton

flink-tutorials's People

Contributors

Stargazers

Watchers

Forkers

flink-tutorials's Issues

Failed to build flink-secure-tutorial

Failed to generate project skeleton

Kerberos KafkaClient issue

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent