aloisius / hadoop-s3a Goto Github PK
View Code? Open in Web Editor NEWAn AWS SDK-backed FileSystem driver for Hadoop
An AWS SDK-backed FileSystem driver for Hadoop
Hi,
Could you please add an option to specify a proxy ?
You should add something like that in S3AFileSystem.java:
awsConf..setProxyHost("192.168.2.12");
awsConf.clientCfg.setProxyPort(80);
Thanks
What's the license of this package? ๐
Recently has some struggles with AWS S3 ListObjects operations. Since it is only in the maintain mode while V2 is better documented and better supported.
Can we move towards V2 instead of V1?
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#listObjectsV2-com.amazonaws.services.s3.model.ListObjectsV2Request-
If the first call on an open s3a stream is to seek, the log statement chokes. This is one solution, but seems expensive vs just fixing the log statement?
--- a/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
+++ b/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
@@ -95,7 +95,7 @@ public class S3AInputStream extends FSInputStream {
if (this.pos == pos) {
return;
}
-
+ openIfNeeded();
LOG.info("Reopening " + wrappedObject.getKey() + " to seek to new offset " + (pos - this.pos));
reopen(pos);
}
Looks like you don't have IAM support just yet. Is that something you are looking to put in?
Has this been tried before ? It seems a lot of functions that are being used are from Hadoop 2.x release and are not available in 1.x
Should we change this or is it not worth the time ?
Hi,
We are trying to upload data to s3 destination with file that need to be encrypted with SSE-KMS keys. Does the s3a file system implementaion support this?
We were able to copy SSE-KMS encypted files from the s3 bucket to AWS EMR cluster's HDFS using command:-
hadoop fs -cp s3a://bucketName/data/ hdfs://testip/user/hdfs/
But we were unable to copy files from EMR cluster's HDFS to s3 bucket with AWS SSE-KMS encryption. Does it also support multipart-file upload?
Hope to hear from you soon,
error804.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.