Giter VIP home page Giter VIP logo

flume's Introduction

flume's People

Contributors

taowenjun avatar

Watchers

James Cloos avatar

flume's Issues

启动Flume出现OOM:java.lang.OutOfMemoryError: Java heap space

原因:flume启动时的默认最大的堆内存大小是20M,实际环境中数据量较大时,很容易出现OOM问题,在flume的基础配置文件conf下的flume-env.sh中添加

export JAVA_OPTS="-Xms2048m -Xmx2048m -Xss256k -Xmn1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"

并且在flume启动脚本flume-ng中,修改JAVA_OPTS="-Xmx20m"为JAVA_OPTS="-Xmx2048m"

此处我们将堆内存的阈值跳转到了2G,实际生产环境中可以根据具体的硬件情况作出调整

Unable to add FlumeEventPointer

主要原因在于Flume运行之后修改File Channel 的capacity大小

解决方法:capacity参数修改回去,后者将checkpointDir中的数据删除,不过这可能会导致数据丢失。

HDFS sink连接Hadoop HA

       Configuration config = new Configuration();
        String nameservices = context.getString("hdfs.nameservice","ns1");
        String namenodes = context.getString("hdfs.namenodes");
        String namenode1 = context.getString("hdfs.namenode1");
        String namenode2 = context.getString("hdfs.namenode2");

        config.set("fs.defaultFS", "hdfs://"+nameservices);
        config.set("dfs.nameservices", nameservices);
        config.set("dfs.ha.namenodes." + nameservices, namenodes);
        String[] nns=namenodes.split(",");
        if(nns.length>=2){
            config.set("dfs.namenode.rpc-address."+nameservices+"."+nns[0], namenode1);
            config.set("dfs.namenode.rpc-address."+nameservices+"."+nns[1], namenode2);
        }

        //conf.setBoolean(name, value);
        config.set("dfs.client.failover.proxy.provider."+nameservices, "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.