Giter VIP home page Giter VIP logo

hadoop-native-macos's Introduction

Hadoop Native Library for macOS

这个仓库提供了一些 macOS 系统下的 Hadoop 本地库,可以消除在 macOS 下运行 Hadoop 程序时的警告:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
using builtin-java classes where applicable

当前目录下的 hadoop-x.x.x 均为 Intel 芯片的本地库文件,Apple M1、Apple M2 等 ARM 芯片请使用 ARM 目录下的文件。

使用方法

hadoop-x.x.x/lib/native 下的文件,替换到本地${HADOOP_HOME}/lib/native 中(删除原来的所有文件)。

不用重启 Hadoop 集群,即可验证警告消失(Hadoop-3.2.1 为例):

hdfs-ls

查看 Hadoop 支持的本地库信息:

cd ${HADOOP_HOME}
bin/hadoop checknative -a

# 主要结果
Native library checking:
hadoop:  true /Users/healchow/bigdata/hadoop-3.2.1/lib/native/libhadoop.dylib
zlib:    true /usr/lib/libz.1.dylib
zstd  :  true /usr/local/Cellar/zstd/1.5.0/lib/libzstd.1.5.0.dylib
snappy:  true /usr/local/Cellar/snappy/1.1.4/lib/libsnappy.1.dylib
lz4:     true revision:10301
bzip2:   false
openssl: false EVP_CIPHER_CTX_reset
ISA-L:   false Loading ISA-L failed: Failed to load libisal.2.dylib (dlopen(libisal.2.dylib, 9): image not found)

hadoop-checknative

其中报错是因为本地没有 snappy 压缩相关的库,暂时忽略。

可用版本

经自我测试,可用的版本如下(欢迎大家补充):

版本号 是否可用
hadoop-2.7.3 未知
hadoop-2.7.7 未知
hadoop-2.8.5 未知
hadoop-2.9.2 未知
hadoop-3.1.3 未知
hadoop-3.2.1 可用
... ...

参考仓库

https://github.com/mingsquall/native-hadoop-library

https://github.com/janlle/mac-hadoop-native-lib-mojove

hadoop-native-macos's People

Contributors

healchow avatar hhoao avatar zhangnew avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

hadoop-native-macos's Issues

Hadoop build with native snappy support for MacOS with M1 chip for hadoop version >= 3.x and <= 3.3

Thanks for creating ARM folder for builds for M1/M2 chip. Can you add support for any hadoop version between 3.0 to 3.3?
For example, I'm using 3.1.1 version and my code fails at

https://github.com/apache/hadoop/blob/branch-3.1.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java#L80

with following error
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks

I tried to build hadoop locally with require.snappy flag but it fails because of different issues.
It would be much helpful and appreciated if you can add native snappy support.

Thank you

Can hadoop2.7.7 run on the mac m1 chip

Hello, I am using the Apple chip m1. When I use hadoop, I encounter the problem of native lib reporting an error. I have deployed hadoop2.7.7 on my virtual machine (aarch64 centos7) and jdk1.8 is also aarch64. Is there any way to make my hadoop run normally?

Adding M1 ARM support for Hadoop 3.2.1

Hi. Is there any chance you can build or help build Hadoop 3.2.1 for Mac M1 ARM please? From the docs it seems that it wasn't possible to build 3.2.0 https://github.com/apache/hadoop/blob/trunk/BUILDING.txt#L416 but I don't see any mention of 3.2.1.

I've spent the day building it using your instructions. I managed to install OpenSSL and also use Protobuf 2.5.0 (even though your comment says to use 3.x https://github.com/healchow/hadoop-native-macos/blob/main/ARM/build.md?plain=1#L3. It looks like 3.2.1 relies on Protobuf 2.5.0.

I get compilation errors and I'm rather lost. Our infrastructure is rather coupled to Hadoop 3.2.1 so it isn't easy for us to upgrade. Any help would be much appreciated. Thank you.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.