I noticed that the published rubygems version is pretty far behind, which is breaking support for presigning urls. is there any chance you can publish an update?

Brief Description

When url is called for an attachment stored in Google Cloud Storage for a filename that includes a hash symbol, the URL encoding does not match what Google Cloud is expecting. We realize that this is somewhat of an edge case. The # character should not be used in filenames, but with modern file systems allowing it, please consider adding support in Shrine.

Expected behavior

When the filename includes a # symbol, it should be encoded as part of the filename and not an anchor.

e.g. "my#file.pdf" -> "my%23file.pdf"

https://storage.googleapis.com/`my_bucket`/attachments/file/my%23file.pdf

Actual behavior

Shrine does not URL encode the file name portion of the download URL...

https://storage.googleapis.com/`my_bucket`/attachments/file/my#file.pdf

Simplest self-contained example code to demonstrate issue

1. Upload a file with a hash symbol in the name.
2. Attempt to download it.

System configuration

Ruby version: 2.5.0

Shrine version: 3.4.0
shrine-google_cloud_storage: 1.34.1

Thanks for your consideration - I will attempt a PR if I can figure it out.

If the file to be copied is in a GCS shrine storage and we can access it, then we can call copy and not download / upload it again (see S3 storage for reference)

Hi, this is a great plugin and has worked well for us until recently when a scenario came up in which we need to delete files in a bucket which has object versioning enabled. Would you be open to supporting an option to delete a file as well as all of its versions?

GCS has an option on buckets to make them a "Requester Pays" type of bucket: https://cloud.google.com/storage/docs/requester-pays. In short, this allows tracing of who/what is actually requesting an upload/download.

To query a bucket where "requester pays" is enabled, the userProject needs to be added in the URL. The GCS Ruby library provides this as an optional user_project keyword argument to Project#bucket: https://github.com/googleapis/google-cloud-ruby/blob/7523214b3c64f88db5e96269b397b066abf4b92e/google-cloud-storage/lib/google/cloud/storage/project.rb#L208-L216

When this is not passed, an error is thrown when listing/uploading files in a requester-pays bucket: Bucket is a requester pays bucket but no user project provided. (Google::Cloud::InvalidArgumentError)

PR: #56

Currently google_cloud_storage does not support batched requests (but google-api-client does). There is a feature request to add it:
googleapis/google-cloud-ruby#2914

Once it is implemented, we should use it for clear! and multi-delete. Sample implem: #31 (review)

Originally posted by @janko in #37 (comment)

Needs to be done before 3.0.0 release

We should allow the user to use options to provide custom parameters to the GCS API calls, and a way to provide them once for all when configuring the storage

I'm loosing my mind.

I'm using Shrine (https://github.com/janko-m/shrine) with Google Cloud Storage (https://github.com/renchap/shrine-google_cloud_storage), but when I start the PUT call I get this:

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
</Message>
<StringToSign>
PUT
image/jpeg
1518399402
/mybucket.appspot.com/7d5e4aad1e3a737fb8d2c59571fdb980.jpg
</StringToSign>
</Error>

I followed this info (http://shrinerb.com/rdoc/classes/Shrine/Plugins/PresignEndpoint.html) for presign_endpoint, but still nothing:

class FileUploader < Shrine
  plugin :presign_endpoint, presign_options: -> (request) do
    filename     = request.params["filename"]
    extension    = File.extname(filename)
    content_type = Rack::Mime.mime_type(extension)

    {
      content_type: content_type
    }
  end
end

I tried with and without this (restarting the Rails server everytime).

Where am I wrong?

I also tried with Postman with a PUT to that URL and withtout any content-type. But still nothing.

I read here: googleapis/google-cloud-node#1976 and here: googleapis/google-cloud-node#1695

How can I try without Rails?

Is there a REPL (or similar) to try with my credentials and with a file?

When trying to remove a model that is missing a file, the library crashes with

Caught error notFound: Not Found
Error - #<Google::Apis::ClientError: notFound: Not Found>

Similar to the S3 plugin

We cant rename files in GCS, but we can copy it and delete the previous one.

From Janko :

Could #open be implemented in terms of Down.open? E.g. Down.open(url(id))? That would be useful for things like restore_cached_data plugin, which extracts metadata from cached files (which in this case might remote, directly uploaded to Google Cloud Storage), so that it doesn't need to download the whole file just to extract metadata.
For the restore_cached_plugin, when a cached file is sent in a hidden field (e.g. with direct uploads or on validation errors), an attacker could technically modify the "metadata" attributes, thus bypassing file validations. The best way that I found to solve this in a generic way was to give the ability to re-extract metadata on that assignment. This is also useful when doing direct uploads to services that don't extract metadata, to be able to extract metadata manually.
Extracting metadata uses #open, which I previously implemented in storages to download the whole file, which wasn't ideal since that part is synchronous. However, I managed to come up with Down.open which creates an IO representing the remote file, which will download only how much is read. Yeah, that would require a signed URL. That's the way I implemented it with S3, relying on the fact that signed URLs will always work. Internally Down.open uses a generic Down::ChunkedIO, which might be usable if you don't have a URL (I used it in GridFS).

This will require #7 first

Getting RuntimeError (Could not load the default credentials. Browse to
https://developers.google.com/accounts/docs/application-default-credentials
for more information on digital ocean, Puma, Nginx, Capistrano, Rails.
i've also added env varaiables in .env file.
GCS_BUCKET='photos.foobrstock.com'
GOOGLE_CLOUD_PROJECT="foobrsnaps"
GOOGLE_CLOUD_KEYFILE="Rails.root.join('config/foobrstock-service.json')"

But still getting the same error for few days.

It seems there is a rate-limit on the number of write-operations on one file in GCS (Google::Cloud::ResourceExhaustedError: rateLimitExceeded: The total number of changes to the object <bucket>/foo exceeds the rate limit. Please reduce the rate of create, update, and delete requests.)

As most tests are done using the /foo object, the test suite sometimes fails.

Each test should use its own object to avoid this.

We need a way to set the ACL used on the file, either globally for the backend or per-upload

The original image is successfully being stored in the 'cache' storage, and subsequently when it comes time to upload, it makes it successfully to the copy method

However, the original file that gets copied over to the main 'store' storage will lose it's Content Type.

I tested using the copy functionality directly using the google-cloud-storage gem. The content type remains.

I have been unable to track down the issue.

Update 1
I was able to narrow it down by deleting this do loop - It appears those options are overwriting the content type.

Update 2
Setting any custom headers as described on the readme will cause the content-type to be emptied

Shrine::Storage::GoogleCloudStorage.new(
  bucket: "store",
  default_acl: 'publicRead',
  object_options: {
    cache_control: 'public, max-age: 7200'
  },
)

This will allow direct upload, generation of links to private files

We need this to clean out the bucket when running tests. This should be doable in an efficient way using GCS batch commands.

Is there an example of the copy process for using with a rails model?

Does it still need to use attacher.set to avoid the problem of deleting the original?

Or is it as simple as...

duplicate = original.dup
duplicate.image = original.image
duplicate.save

The version used currently by the latest release is 1.7.1, which depends on google-api-client 0.14.2, which is pretty far behind (causing issues integrating with bigquery specifically).

Is this a conscious decision or an oversight? :)

Getting ArgumentError: project_id is missing in syslog sidekiq on submitting. Also the images are not bieng processed in the Background.
my .env:-
GCS_BUCKET='Bucket Name'
GOOGLE_CLOUD_PROJECT="Project Name"
GOOGLE_APPLICATION_CREDENTIALS=./some-service.json

I see in #7 that signed urls are implemented with presign.

I'm thinking about access to an expiring url from an existing activerecord model with a file, and the implementation here seems like a different pattern than that of the S3#url.

I'm new to shrine, so please point out what I'm missing.

Assuming a model Agreement and a shrine field document_data, with S3 docs it looks as though you should be able to agreement.document.url(expires_in: 300).

Shouldn't the expected usage here be agreement.document.url(expires: 300) based on the above pattern, the UploadedFile api, and the GCS Using signed urls?

• Is that correct?
• Should this gem match the usage (except making options align with GCS, in this case expires)?
• Can I already exec presign from my model and I'm missing it?

This allows to concatenate multiple files in 1 API call

Hi there,

I noticed an issue when uploading a file using a service account with the role storage object administrator (!= storage administrator).
The upload fails when accessing the bucket in #get_bucket.
The google-cloud-storage gem provides a solution in form of the skip_lookup option:

storage = Google::Cloud::Storage.new
bucket = storage.bucket "sample-bucket" # raises
Google::Cloud::PermissionDeniedError: forbidden:  ... iam.gserviceaccount.com does not have storage.buckets.get access to sample-bucket

bucket = storage.bucket "sample-bucket", skip_lookup: true
=> #<Google::Cloud::Storage::Bucket:0x00...>

Do you see any issues with using the skip_lookup option in shrine-google_cloud_storage ?

More info can be found here: googleapis/google-cloud-ruby#1588