flume's Issues
HDFS sink连接Hadoop HA
Configuration config = new Configuration();
String nameservices = context.getString("hdfs.nameservice","ns1");
String namenodes = context.getString("hdfs.namenodes");
String namenode1 = context.getString("hdfs.namenode1");
String namenode2 = context.getString("hdfs.namenode2");
config.set("fs.defaultFS", "hdfs://"+nameservices);
config.set("dfs.nameservices", nameservices);
config.set("dfs.ha.namenodes." + nameservices, namenodes);
String[] nns=namenodes.split(",");
if(nns.length>=2){
config.set("dfs.namenode.rpc-address."+nameservices+"."+nns[0], namenode1);
config.set("dfs.namenode.rpc-address."+nameservices+"."+nns[1], namenode2);
}
//conf.setBoolean(name, value);
config.set("dfs.client.failover.proxy.provider."+nameservices, "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
启动Flume出现OOM:java.lang.OutOfMemoryError: Java heap space
原因:flume启动时的默认最大的堆内存大小是20M,实际环境中数据量较大时,很容易出现OOM问题,在flume的基础配置文件conf下的flume-env.sh中添加
export JAVA_OPTS="-Xms2048m -Xmx2048m -Xss256k -Xmn1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"
并且在flume启动脚本flume-ng中,修改JAVA_OPTS="-Xmx20m"为JAVA_OPTS="-Xmx2048m"
此处我们将堆内存的阈值跳转到了2G,实际生产环境中可以根据具体的硬件情况作出调整
flume.source.ExecSource.run(ExecSource.java 327) [timedFlushExecService24-0]
将exec source的batchTimeout参数调大
InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too long.
hdfs.calltimeout可能设置得太低或HDFS需要花费太长的时间。
设置hdfs.calltimeout为更大的值
Unable to add FlumeEventPointer
主要原因在于Flume运行之后修改File Channel 的capacity大小
解决方法:capacity参数修改回去,后者将checkpointDir中的数据删除,不过这可能会导致数据丢失。
org.apache.flume.EventDeliveryException: java.lang.NullPointerException
Event event = channel.take();
取出来的event可能为空
org.apache.flume.ChannelException: Commit failed due to IO error [channel=c1]
采用file channel,本地磁盘没有空间
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.