fclairamb / afero-s3 Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
As with my other issue, it would be good to be able to upload with a Cache-Control header in metadata for usage with S3 buckets used to serve web resources.
There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.
Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
.github/workflows/build.yml
actions/checkout v3
golangci/golangci-lint-action v3
actions/setup-go v4
fclairamb/minio-github-actions v0.2
ubuntu 20.04
go.mod
go 1.19
github.com/aws/aws-sdk-go v1.55.5
github.com/spf13/afero v1.11.0
github.com/stretchr/testify v1.9.0
Would it be possible to add a way to set the ACL on newly created objects? For example, I am using Digital Ocean Spaces and I am uploading files that should be accessible by the general public. By default, those files have the ACL set to Private
, while I would benefit from having this set to Public
(by setting the appropriate AWS S3 setting for the ACL that DO understands since DO Spaces follows the AWS S3 api).
I propose that during the Fs initialization we could set the default ACL that would get applied when writing to a file. I am open to suggestions.
If that sounds OK, I'd be more than happy to implement this if you're busy.
All files are uploaded with a Content-Type of binary/octetstream. This is a problem for our use case where we're uploading to a bucket which is used for hosting files for a website. I believe there is functionality in the SDK to auto-detect the Content-Type based on the file extension which might be a better default.
Hi @fclairamb! Thanks for this package :)
I am running into an issue when combining s3.Fs
with afero.BasePathFs
on Windows systems.
My issue is that S3 doesn't like backslash delimited paths (such as Windows file paths).
When nesting an S3 filesystem inside afero.BasePathFs
, the paths are first processed using filepath
package which uses os.PathSeparator
, which is backslash (\
) on Windows.
Therefore when you call afero.BasePathFs.MkdirAll
the path it provides to s3.Fs.MkdirAll
is a Windows path. It uploads "successfully" to S3, however S3 treats the path differently in such a way that makes uploading files from Windows use a different S3 path than from MacOS or Linux.
This is undesirable, at least in my case. The paths, no matter which OS they are created from should be consistent and valid S3 paths using forward slash delimiter.
So I can see two options:
s3.Fs
process the input path and replace all backslashes with forward slashes, at least when runtime.GOOS == "windows"
afero.BasePathFs
to handle overriding the path separator manuallyFor the moment I figure option 1 is more correct since S3 dictates that all paths should be forward slash paths.
Do you have thoughts on this? I can create a PR for this if you agree on the desired behavior.
Issue : Cant list the root ("/") directory of the s3 bucket folder as it is recognized as file rather than a directory
Can someone please take a look at #393 as we can see the issue and the fix for it has been open for a while .
Thanks in advance
Is upgrading to the AWS Go SDK v2 on the roadmap?
Is there any way to create a FS that accesses only a subdir in S3? (basically a prefix/
to everything)
Hi
I'd like to have the library to support SSE: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-server-side-encryption-with-kms.html
As far as I can tell, the way to go would be to add new public fields to the UploadedFileProperties
type, like this:
// UploadedFileProperties defines all the set properties applied to future files
type UploadedFileProperties struct {
ACL *string // ACL defines the right to apply
CacheControl *string // CacheControl defines the Cache-Control header
ContentType *string // ContentType define the Content-Type header
+ // Parameters below mirror the ones from s3.PutObjectInput type from AWS SDK
+ SSEKMSEncryptionContext *string
+ SSEKMSKeyId *string
+ ServerSideEncryption *string
}
And then copy them to the struct passed to AWS SDK in applyFileCreateProps
and in applyFileWriteProps
:
func applyFileCreateProps(req *s3.PutObjectInput, p *UploadedFileProperties) {
if p.ACL != nil {
req.ACL = p.ACL
}
if p.CacheControl != nil {
req.CacheControl = p.CacheControl
}
if p.ContentType != nil {
req.ContentType = p.ContentType
}
+ if p.SSEKMSEncryptionContext != nil {
+ req.SSEKMSEncryptionContext = p.SSEKMSEncryptionContext
+ }
+
+ if p.SSEKMSKeyId != nil {
+ req.SSEKMSKeyId = p.SSEKMSKeyId
+ }
+
+ if p.ServerSideEncryption != nil {
+ req.ServerSideEncryption = p.ServerSideEncryption
+ }
}
(same code in the other function)
Couple of questions:
Thanks,
Alexei
statDirectory fails because filepath.Clean is platform dependent. On windows \
is used as directory separator instead of '/'.
Instead use path.Clean, which always uses unix style path on all OSes.
Lines 254 to 255 in 5eb0a32
Hello,
The following code does not works (nothing written on the bucket), but doesn't give any errors :
package main
import (
"fmt"
"os"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
s3 "github.com/fclairamb/afero-s3"
)
func main() {
// Configure to use MinIO Server
s3Config := &aws.Config{
Credentials: credentials.NewStaticCredentials("***", "****", ""),
Endpoint: aws.String("https://s3.fr-par.scw.cloud"),
Region: aws.String(""),
S3ForcePathStyle: aws.Bool(true),
}
fmt.Printf("Starting....\n")
// You create a session
sess, err := session.NewSession(s3Config)
if err != nil {
fmt.Printf("error: %v\n", err)
}
// Initialize the file system
s3Fs := s3.NewFs("my-bucket", sess)
// And do your thing
file, err := s3Fs.OpenFile("my_new_file.txt", os.O_WRONLY, 0777)
if err != nil {
fmt.Printf("error: %v\n", err)
}
file.WriteString("Hello world !")
file.Close()
}
It is actually the Region that is not correct (works with fr-par), but an error should be thrown ?!
When you open a file that doesn't exist, you get a generic error with. a "state : file does not exist" message rather than an afero.ErrFileNotFound. This makes it difficult to check for not found errors.
Hi, we have been using this library at my job for managing s3 buckets as an FTP client.
We are having some issues concerning the cd
command as it throws a 550 error whenever our application calls the fs.
550 CD issue: stat /<directory_name>: InternalServerError: Internal Server Error\n status code: 500
After investigation, we concluded this error was originating from this library.
We do know that s3 does not support directories but we expected the naming standard /directory/file.ex
to handle the navigation feature.
Is this a known issue and can you provide any support about this?
Thank you
Line 74 in a24edce
If I call f.Readdir
where f.name
is /Path/
then it appends another slash to make it /Path//
which fails
in s3/minio/etc.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.