hairless / plink Goto Github PK
View Code? Open in Web Editor NEWPlatform for Flink
License: Mozilla Public License 2.0
Platform for Flink
License: Mozilla Public License 2.0
1、自定义reporter,上报checkpoint信息
2、添加数据库表,用于落地数据
3、提供添加和查询接口
4、在reporter里通过平台添加接口进行上报
描述一下 bug
创建一个flink sql类型任务,定义sql时有用到反引号,任务启动失败
** 使用的Plink版本**
0.2.0
重现过程
具体的重现步骤:
您的功能请求是否与问题有关?请描述。
否
描述您想要的解决方案
其他内容
无
您的功能请求是否与问题有关?请描述。
当机器上的HADOOP发行版是CDH时,机器可能没有HADOOP_HOME这个变量,且HADOOP_CONF_DIR是和HADOOP分开存放的
例如:
$> ls -l /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3
drwxr-xr-x 2 root root 4096 Mar 26 2019 bin
drwxr-xr-x 26 root root 4096 Aug 25 2017 etc
drwxr-xr-x 4 root root 4096 Aug 25 2017 include
drwxr-xr-x 2 root root 61440 Nov 2 12:21 jars
drwxr-xr-x 37 root root 4096 Aug 25 2017 lib
drwxr-xr-x 2 root root 4096 Aug 25 2017 lib64
drwxr-xr-x 3 root root 4096 Aug 25 2017 libexec
drwxr-xr-x 2 root root 4096 Aug 25 2017 meta
drwxr-xr-x 3 root root 4096 Aug 25 2017 share
$> readlink -f /etc/hadoop/conf
/etc/hadoop/conf.cloudera.yarn
$> ls -l /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/etc/hadoop
lrwxrwxrwx 1 root root 10 Aug 25 2017 conf.dist -> conf.empty
drwxr-xr-x 2 root root 4096 Dec 11 16:20 conf.empty
drwxr-xr-x 2 root root 4096 Aug 25 2017 conf.impala
drwxr-xr-x 2 root root 4096 Aug 25 2017 conf.pseudo
drwxr-xr-x 2 root root 4096 Aug 25 2017 conf.pseudo.mr1
描述您想要的解决方案
希望加载HADOOP配置时,能够直接定义HADOOP_CONF_DIR。
现在的情况
final String CONF_SUFFIX = "/etc/hadoop";
Collection<File> files = FileUtils.listFiles(new File(hadoopHome, CONF_SUFFIX), new String[]{"xml"}, false);
CDH的配置 HADOOP_CONF_DIR= /etc/hadoop/conf
无法指定CDH的配置,因为后缀定死了
CDH的HADOOP_HOME=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3
无法指定CDH的配置,因为CDH的HADOOP_HOME/etc/hadoop不是实际的配置文件
其他内容
无
您的功能请求是否与问题有关?请描述。
在查看日志是由于DTO没有重写toString,导致无法查看详细消息
2020-09-04 14:52:16.861 WARN [common-Thread-2] c.g.h.p.schedule.task.InstanceStatusSyncTask : asyncSubmitJob error jobInstance=com.github.hairless.plink.model.pojo.JobInstance@1a422382
描述您想要的解决方案
// BaseModel 重写toString :
public String toString() {
return ToStringBuilder.reflectionToString(this);
}
注 : 要参与一起贡献的,评论后请加下面的 QQ 群.
如 :
//需要后端配合
➜ ~ docker pull registry.cn-hangzhou.aliyuncs.com/hairless/plink:master
Error response from daemon: manifest for registry.cn-hangzhou.aliyuncs.com/hairless/plink:master not found
以下是暂定的Plink 0.2待支持的功能点,有任何新想法请在下面追加评论,
功能点收集完成后会在qq群里进行投票来确定最终的功能点,欢迎大家积极参与
支持flink on yarn提交任务
期望支持Flink YARN Session提交任务
Cluster.mode:local
java.lang.NullPointerException: null
at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:61)
at org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:116)
at org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:61)
at org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:585)
at com.github.hairless.plink.service.impl.LocalFlinkClusterServiceImpl.submitJob(LocalFlinkClusterServiceImpl.java:61)
at com.github.hairless.plink.schedule.task.SubmitJobTask.asyncSubmitJobTask(SubmitJobTask.java:55)
at com.github.hairless.plink.schedule.task.SubmitJobTask$$FastClassBySpringCGLIB$$86b74b1f.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
描述一下 bug
DTO没有重写toString方法,导致日志看不见详细信息
2020-09-04 14:52:16.861 WARN [common-Thread-2] c.g.h.p.schedule.task.InstanceStatusSyncTask : asyncSubmitJob error jobInstance=com.github.hairless.plink.model.pojo.JobInstance@1a422382
** 使用的Plink版本**
0.1.0
重现过程
具体的重现步骤:
其他内容
可添加任何关于bug的其他内容到此处
项目打包
maven release
git release
git new batch 0.1
docker 发版
完善文档
flink新版本已经不再支持window的提交命令
window部署文档之前已经移除
此处需要移除/docs/README.md 中window部署相关的链接
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.