aloisius / hadoop-s3a Goto Github PK

Hi,

Could you please add an option to specify a proxy ?

You should add something like that in S3AFileSystem.java:
awsConf..setProxyHost("192.168.2.12");
awsConf.clientCfg.setProxyPort(80);

Thanks

What's the license of this package? 🙏

Recently has some struggles with AWS S3 ListObjects operations. Since it is only in the maintain mode while V2 is better documented and better supported.

Can we move towards V2 instead of V1?
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#listObjectsV2-com.amazonaws.services.s3.model.ListObjectsV2Request-

If the first call on an open s3a stream is to seek, the log statement chokes. This is one solution, but seems expensive vs just fixing the log statement?

--- a/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
+++ b/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
@@ -95,7 +95,7 @@ public class S3AInputStream extends FSInputStream {
     if (this.pos == pos) {
       return;
     }
-
+    openIfNeeded();
     LOG.info("Reopening " +  wrappedObject.getKey() + " to seek to new offset " + (pos - this.pos));
     reopen(pos);
   }

Looks like you don't have IAM support just yet. Is that something you are looking to put in?

Has this been tried before ? It seems a lot of functions that are being used are from Hadoop 2.x release and are not available in 1.x

Should we change this or is it not worth the time ?

Hi,

We are trying to upload data to s3 destination with file that need to be encrypted with SSE-KMS keys. Does the s3a file system implementaion support this?

We were able to copy SSE-KMS encypted files from the s3 bucket to AWS EMR cluster's HDFS using command:-

But we were unable to copy files from EMR cluster's HDFS to s3 bucket with AWS SSE-KMS encryption. Does it also support multipart-file upload?

Hope to hear from you soon,
error804.