alexandrubagu / gcs_signed_url Goto Github PK

View Code? Open in Web Editor NEW

23.0 3.0 9.0 294 KB

Create Signed URLs for Google Cloud Storage

Home Page: https://hex.pm/packages/gcs_signed_url

License: MIT License

Elixir 99.87% Euphoria 0.13%

gcs google cloud storage google-cloud-storage gcs-signed-url signed urls

gcs_signed_url's Introduction

gcs_signed_url - Create Signed URLs for Google Cloud Storage

Important

This package works with Elixir >= 1.8 and Erlang/OTP >= 22.3

Hex Installation

Add gcs_signed_url to your list of dependencies in mix.exs:

def deps do
  [{:gcs_signed_url, "~> 0.4"}]
end

Usage

This library creates signed URLs for the Google Cloud Storage in three steps:

Create a string to sign (use V4 signing process - V2 signing process is stil supported but deprecated)
Sign the string to sign with the private key of a Google service account (GSA)
Form the URL including the signature

The actual signing can be done on-premise on the machine your application is executed or it can be delegated to the Google IAM SignBlob API. The method of signing depends on your setup. Both methods work for creating V4 or V2 (deprecated) signatures.

Google IAM SingBlob API - Preferred on the GKE

If your application runs on the Google Kubernetes Engine, the preferred way of accessing Google Cloud services is through a Workload Identity. In this scenario, you run your application under a Kubernetes service account (KSA) which is related to a GSA. Using Goth, you would get an OAuth2 access token and use it to create a signed URL through the SignBlob API.

In this scenario, the GSA you get the access token for (GSA_AUTH) acts as a Google Service Account GSA_SIGNER and signs the URL on his behalf. This requires the GSA_AUTH go have the Google IAM permission iam.serviceAccounts.signBlob on the GSA_SIGNER, e.g. by giving it the built in role roles/iam.serviceAccountTokenCreator on GSA_SIGNER.

GSA_AUTH and GSA_SIGNER can also be the same service account in which case he needs to have the permission iam.serviceAccounts.signBlob on itself.

Example

iex> {:ok, %{token: access_token}} = Goth.Token.for_scope("https://www.googleapis.com/auth/devstorage.read_write")
iex> oauth_config = %GcsSignedUrl.SignBlob.OAuthConfig{service_account: "project@gcs_signed_url.iam.gserviceaccount.com", access_token: access_token}
iex> GcsSignedUrl.generate_v4(oauth_config, "my-bucket", "my-object.jpg", verb: "PUT", expires: 1800, headers: ["Content-Type": "application/jpeg"])
{:ok, "https://storage.googleapis.com/my-bucket/my-object.jpg?X-Goog-Expires=1800..."}

On-Premise Signing

In this scenario you have a service account key in form of a JSON file on your machine. The library will use the private key to create the signature, no network calls are needed.

Load the client

iex> GcsSignedUrl.Client.load_from_file("/home/alexandrubagu/config/google.json")

iex> service_account = service_account_json_string |> Jason.decode!
iex> GcsSignedUrl.Client.load(service_account)

Generate signed url

GcsSignedUrl.generate_v4(client, "my-bucket", "my-object.mp4")
GcsSignedUrl.generate_v4(client, "my-bucket", "my-object.mp4", expires: 3 * 3600) # 3 hours

gcs_signed_url's People

Contributors

Stargazers

Watchers

Forkers

cchko boudra kianmeng syamilmj dvic code-supply jechol okarifrankline robinboers

gcs_signed_url's Issues

MalformedSecurityHeader

I got this error, V2 API

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>MalformedSecurityHeader</Code>
<Message>Your request has a malformed header.</Message>
<ParameterName>signature</ParameterName>
<Details>Signature was not base64 encoded</Details>
</Error>

GKE Workload Identity Support

On the GKE, the preferred way of providing service account authentication are workload identities. In this scenario, there would not be a service account key in form of a json so we would not be able to actually sign the string to sign on the pod. Instead creating a signed URL would work as follows:

Create the string to sign
Get an token for the Google service account provided by the workload identity (provided by the caller using the Goth library)
Send a request to https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/signBlob using the token in order to sign the string.

I haven't worked it out completely but I think that should be the correct way on the GKE.

Tag released package versions

Thank you for contributing this open source package to the Elixir community. It's very useful and your work on is greatly appreciated.

It would be useful for consumers of this package (like myself) to be able to match versions of this package in Hex back to the code which was used to publish them. This is additionally valuable when using products like Dependabot to automate the process of update dependancies in Elixir projects. Achieving this should be relatively simple, tagging the version control commit which is used to publish the package should be enough.

Adding and using a changelog on top of tagging package versions would be even better, but I understand that this would add extra work for you.

Is the SignBlob API used now deprecated?

Hi there!

I think that the API endpoint used for signing is this one. If so, this gets turned off on July 1. Any plans to update to this other one instead?

If not, I might give it a go.

Remove fixtures in favor of defining local setup blocks

@mruoss I think we can remove fixtures and define local setup blocks. I think is easier to follow what test does in the end.

defmodule GcsSignedUrl.CanonicalRequestTest do
  use ExUnit.Case
  alias GcsSignedUrl.CanonicalRequest

  describe "create/4" do
    @signed "content-type;x-foo;x-zed"
    @canonical "content-type:application/json\nx-foo:foo,BA  R,gamma\nx-zed:zed\n"

    setup do
      {:ok, headers: %GcsSignedUrl.Headers{signed: @signed, canonical: @canonical}}
    end

    test "creates canonical request according to specs", %{headers: headers} do
      canonical_request =
        CanonicalRequest.create("GET", "bucket/object.jpg", "foo=bar&alpha=beta", headers)

      assert "GET\nbucket/object.jpg\nfoo=bar&alpha=beta\n#{headers.canonical}\n#{headers.signed}\nUNSIGNED-PAYLOAD" ==
               canonical_request
    end
  end
end

WDYT ?

GcsSignedUrl.generate_v4 doesn't work

Issue description

We have provided correct roles and permissions which are storage bucket creator and ServiceAccountTokenCreator to the service account, still getting 403 permission denied error

Ran the following commands:

{:ok, token} = Goth.Token.fetch(%{})

account="[email protected]"

oauth_config = %GcsSignedUrl.SignBlob.OAuthConfig{service_account: account, access_token: token.token}

GcsSignedUrl.generate_v4(oauth_config, "sample-bucket", "fresh.pdf")
{:error,
"403 PERMISSION_DENIED: The caller does not have permission Make sure the authorized SA has role roles/iam.serviceAccountTokenCreator on the SA passed in the URL."}

Link to documentation

MalformedSecurityHeader : Facing issue while passing content-type

This is code:

singed_url_tup =
            fetch_access_token()
            |> get_gcs_outh_config()
            |> GcsSignedUrl.generate_v4(
              storage_path_map["bucket"],
              storage_path_map["object_id"],
              verb: verb,
              expires: expiry_second,
              headers: ["Content-Type": "application/jpeg"]
            )

The generated url:

https://storage.googleapis.com/random/s6d432fd1-b5cc-4a05-a453-267fff98c261.jpg?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=abc%40random-dev.iam.gserviceaccount.com%2F20211022%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20211022T151548Z&X-Goog-Expires=600800&**X-Goog-SignedHeaders=content-type%3Bhost&**X-Goog-Signature=794adb8c8367fe16aea8086308f385b9b3af100f74ec6e8f10341e53719910bd1e83adfb1c1919a2e41a0f93f1f298ae57d8de43678468db247d995bb2523eece774c8501d83befd9fa8bf3216041245531c117b11ea1b63e5c7fb4644cf505c58726c4714f38eeee6cb77b57768b3b745367f88149aba9275df1868f2f79511ac1e539a85a8899decda1b40257f0465580b77c6c7147d19935a8a957651ceae549916f109c183f2658ddb1520eb25c11c7d2658716edd7c6685b5417c82b399ac00daa07a086ba74e12a6ef8e098176ef5c9506216c7d84a9719d0893964468c0d96f63a345eae357dc200ccb2755c7d1a256dc16c01108ae52f8c2d221e33a

When I am not passing header as an argument, then PUT/GET works fine. But as soon as I am passing headers: ["Content-Type": "application/jpeg"] as header getting 400 while uploading image to the PUT signed url.

However, the content-type X-Goog-SignedHeaders=content-type%3Bhost& (cropped from above mention Signed URL) is not getting effected if I pass images/png or application/jpeg

The module is really helpful and quite smooth. This is the first time I am facing issue with this module. Any help would be appreciated.

GcsSignedUrl.hours_after returns invalid number for v4

hours_after/1 returns some long number even if you try to get for any number of hours, as per Google docs maximum value is 604800 seconds (7 days ) as you can see example below the integers returns by hours_after for 3 hours is some big number.

hours_after should return a valid number of seconds.

Erlang/OTP 24 [erts-12.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [dtrace]
Interactive Elixir (1.13.0) - press Ctrl+C to exit (type h() ENTER for help)

{:gcs_signed_url, "~> 0.4"},

>>  GcsSignedUrl.hours_after(3)
1649075864