fullstorydev / emulators Goto Github PK

Hi Full Story Dev,

From scouring the interwebs, it's my opinion that this is the highest quality open source GCS emulator available today. Thank you!!

My Goal

For one of my projects, I've leveraging the https://github.com/ncruces/go-gcp/gmutex package to do some distributed locking. I'd love to be able to run integration tests against the gcsemu cli to avoid needing credentials and improve test execution time.

My Problem

However, that package relies on the following Google Cloud Storage XML API endpoints which are not available in this project yet:

• HEAD Object
• Simple PUT Object
• Compose PUT Object

Proposed Solution

Would you be willing to accept a contribution that added support for these endpoints? I've been hacking on a fork and making some progress.

Since the XML API has some differences in how conditions are passed to the server via query params and a different path structure, I'd probably:

• rename the existing g.Handler() -> g.JsonHandler() for these routes: https://github.com/fullstorydev/emulators/blob/master/storage/gcsemu/parse.go#L9-L14
• add a new g.XmlHandler() for these routes: https://github.com/fullstorydev/emulators/blob/master/storage/gcsemu/parse.go#L16

I'd probably use the test suite for this project and from gmutex and examining the docs to validate I've implemented the behavior correctly.

I understand this is a big ask, and I completely understand if you'd rather decline a contribution of this nature. Let me know if you're willing to consider and discuss. Many thanks for all the excellent work on this project to date!

I'd like to run the Object Storage emaulator.
Is there any pullable public Docker image available or do I have to build them locally?

If there isn't any available, would be such feature considered for inclusion?
It is feasible with some Github Actions magic and a Docker Hub (free plan) account, which could build & push them automatically on each commit.

Description

this issue is for the GCS emulator

When doing a resumable upload a post request is sent to the emulator which returns the "real" address to a file in GCS in the Location header.

Example POST to the emulator from the official go client:
POST http://localhost:8080/upload/storage/v1/b/my-bucket/o
The response contains has the Location header pointing to https://www.googleapis.com/storage/v1/b/my-bucket/o/my-object.ext?upload_id=4

This location field is then used by the official storage library for further requests, essentially skipping the emulator for all subsequent upload chucks (hanging until context is cancelled in my case).

Fix 1

I've verified a fix by manually changing the location header value to http://localhost:8080/storage/v1/b/my-bucket/o/my-object.ext?upload_id=4 before it is used by the official client. I did so by overriding the value using the debugger in this file: google.golang.org/[email protected]/storage/v1/storage-gen.go:10295

I suspect these fields come from storage/gcsemu/meta.go?

Fix 2

Setting the chunk size of the writer to >= file.size() also solves this issue, but then it isn't a resumable upload...

It would be nice if you could update so that the location header points to the local emulator rather than the real official GCS url.

I know you say that I shouldn't rely on STORAGE_EMULATOR_HOST, but with this (hopefully) minor fix the library works as expected for me at least...

EDIT:

I've verified that this quickfix solves the issue for me:

diff --git a/storage/gcsemu/meta.go b/storage/gcsemu/meta.go
index aee05f1..df8740e 100644
--- a/storage/gcsemu/meta.go
+++ b/storage/gcsemu/meta.go
@@ -47,10 +47,10 @@ func scrubMeta(meta *storage.Object) {
 // Return the URL for a file.  Note that this will generally not be useful as an input an http request due to the way
 // that net/url does character substitution (see comments above).
 func bucketUrl(bucket string) string {
-	return fmt.Sprintf("https://www.googleapis.com/storage/v1/b/%s", bucket)
+	return fmt.Sprintf("http://localhost:8080/storage/v1/b/%s", bucket)
 }
 
 // Return the URL for a file.
 func objectUrl(bucket string, filepath string) string {
-	return fmt.Sprintf("https://www.googleapis.com/storage/v1/b/%s/o/%s", bucket, filepath)
+	return fmt.Sprintf("http://localhost:8080/storage/v1/b/%s/o/%s", bucket, filepath)
 }

The storage layer is pluggable, but the interface contains private types, so it can't be implemented by any type outside of the gcsemu package. Neither of the two included storage implementations meet my needs, so I would like to be able to provide my own.

What do you think about exporting gcsemu.httpBaseUrl?

Probably similar to #7

When calling batch endpoint in real GCS, files that doesn't exist return NotFound. When using the emulator it never returns an error.

Real GCS returns an *googleapi.Error with Code 404

Hi, First of all, congratulations on great work on these Google Emulators.

I am trying to use a storage emulator to test one golang application that uses GCS. But when I try test error on delete bucket that doesn't exist bucket. The emulator doesn't return any error to inform that Bucket doesn't exist.

The real GCS API returns one error when I try to delete on the bucket that doesn't exist.

Is this behavior expected by this emulator?

The GCS emulator is a super useful utility, but it is frustrating to have to install golang, which we do not otherwise use, in our devcontainer to use it. Would love to see x86-64 and arm64 binaries, perhaps using GitHub packages.