healchow / hadoop-native-macos Goto Github PK

Hello, I am using the Apple chip m1. When I use hadoop, I encounter the problem of native lib reporting an error. I have deployed hadoop2.7.7 on my virtual machine (aarch64 centos7) and jdk1.8 is also aarch64. Is there any way to make my hadoop run normally?

Thanks for creating ARM folder for builds for M1/M2 chip. Can you add support for any hadoop version between 3.0 to 3.3?
For example, I'm using 3.1.1 version and my code fails at

https://github.com/apache/hadoop/blob/branch-3.1.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java#L80

with following error
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks

I tried to build hadoop locally with require.snappy flag but it fails because of different issues.
It would be much helpful and appreciated if you can add native snappy support.

Thank you

Hi. Is there any chance you can build or help build Hadoop 3.2.1 for Mac M1 ARM please? From the docs it seems that it wasn't possible to build 3.2.0 https://github.com/apache/hadoop/blob/trunk/BUILDING.txt#L416 but I don't see any mention of 3.2.1.

I've spent the day building it using your instructions. I managed to install OpenSSL and also use Protobuf 2.5.0 (even though your comment says to use 3.x https://github.com/healchow/hadoop-native-macos/blob/main/ARM/build.md?plain=1#L3. It looks like 3.2.1 relies on Protobuf 2.5.0.

I get compilation errors and I'm rather lost. Our infrastructure is rather coupled to Hadoop 3.2.1 so it isn't easy for us to upgrade. Any help would be much appreciated. Thank you.

不知道您知道不？