Flume的学习及使用,源码阅读
这部分内容主要在Wiki中:https://github.com/taowenjun/Flume/wiki
Flume的学习及使用,源码阅读
这部分内容主要在Wiki中:https://github.com/taowenjun/Flume/wiki
采用file channel,本地磁盘没有空间
原因:flume启动时的默认最大的堆内存大小是20M,实际环境中数据量较大时,很容易出现OOM问题,在flume的基础配置文件conf下的flume-env.sh中添加
export JAVA_OPTS="-Xms2048m -Xmx2048m -Xss256k -Xmn1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"
并且在flume启动脚本flume-ng中,修改JAVA_OPTS="-Xmx20m"为JAVA_OPTS="-Xmx2048m"
此处我们将堆内存的阈值跳转到了2G,实际生产环境中可以根据具体的硬件情况作出调整
主要原因在于Flume运行之后修改File Channel 的capacity大小
解决方法:capacity参数修改回去,后者将checkpointDir中的数据删除,不过这可能会导致数据丢失。
将exec source的batchTimeout参数调大
Event event = channel.take();
取出来的event可能为空
Configuration config = new Configuration();
String nameservices = context.getString("hdfs.nameservice","ns1");
String namenodes = context.getString("hdfs.namenodes");
String namenode1 = context.getString("hdfs.namenode1");
String namenode2 = context.getString("hdfs.namenode2");
config.set("fs.defaultFS", "hdfs://"+nameservices);
config.set("dfs.nameservices", nameservices);
config.set("dfs.ha.namenodes." + nameservices, namenodes);
String[] nns=namenodes.split(",");
if(nns.length>=2){
config.set("dfs.namenode.rpc-address."+nameservices+"."+nns[0], namenode1);
config.set("dfs.namenode.rpc-address."+nameservices+"."+nns[1], namenode2);
}
//conf.setBoolean(name, value);
config.set("dfs.client.failover.proxy.provider."+nameservices, "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
hdfs.calltimeout可能设置得太低或HDFS需要花费太长的时间。
设置hdfs.calltimeout为更大的值
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.