ota-insight / bqwriter Goto Github PK

Contact Details

[email protected]

What happened?

8cb7bcd

Weird thing is that this only appears in a new service written by a new joiner while existing services using bqwriter have never seen such a crash (because the library would crash if you give a nil result to that ch, given it does not expect that).

We need to investigate if this is because of a mistake on our end or just something that needs to be documented or something that might need to be fixed or is already fixed on bigquery's side.

Version

0.7.0 (Latest)

What platform are you seeing the problem on?

MacOS, Linux

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Contact Details

No response

Summary of your proposal

We currently support the insertAll API and soon also the storage API. What we do not yet support is batch loading as documented in https://cloud.google.com/bigquery/docs/batch-loading-data. They give an example of a single file, but we could do it for any number of files as well as any reader in general.

Need to investigate the specifics, but it does look like it is still in scope.

Motivation for your proposal

It's a different kind of use case of BQWriter, still writing data into BQ but for an entirely different purpose. For those purposes batch loading is better suited as the timing isn't as critical, and with that a reduction of cost is a given.

Alternatives for your proposal

Do not support it and explicitly document so instead.

Version

0.3.1 (Latest)

What platform are you mostly using or planning to use our software on?

MacOS

Code of Conduct

• I agree to follow this project's Code of Conduct

Contact Details

No response

Summary of your proposal

Use the upstream bigquery/storage/managedwriter package instead of our forked version.
One of the issues that ideally is resolved prior to doing so is googleapis/google-cloud-go#5094.

Motivation for your proposal

The main motivation is that we have less code to maintain ourselves.

Alternatives for your proposal

The alternative is to either not use a managedWriter in which case we're going to reinvent the wheel most likely. The only other obvious option is to continue using our fork but that means we'll also need to maintain it ourselves, and do so mostly alone.

Version

0.3.1 (Latest)

What platform are you mostly using or planning to use our software on?

MacOS

Code of Conduct

• I agree to follow this project's Code of Conduct

Contact Details

[email protected]

Summary of your proposal

To change the default connection url you can use the optionsFn parameters to the bigquery.NewClient.

Making the impact as minimum as possible and not changing the behavior used today. The option.ClientOption can be part of the method calls where the bigquery.NewClient is called.
"google.golang.org/api/option"

e.g:
func NewStreamer(ctx context.Context, projectID, dataSetID, tableID string, cfg *StreamerConfig, opts ...option.ClientOption) (*Streamer, error)

where then they can be passed on to (storage, batch and insertall) e.g:

client, err := storage.NewClient(
					projectID, dataSetID, tableID,
					encoder, protobufDescriptor,
					logger,
					opts..., 

and the pass it on to the google bigquery client:
writer, err := managedwriter.NewClient(ctx, projectID, opts...)

To use in test code:

bqWriter, err := bqwriter.NewStreamer(
		ctx, projectId, datasetId, tableId,
		&bqwriter.StreamerConfig{
			...
		},
		option.WithEndpoint("localhost:9050"),
		option.WithoutAuthentication(),
	)
	if err != nil {
		panic(err)
	}

Motivation for your proposal

The motivation for this change:
To use the 'https://github.com/goccy/bigquery-emulator' as part of the integration tests during development and deploy pipeline.

bigquery-emulator is a locally running biguery emulator, that easily can run during testing. Fort this to work though, local connection needs to be supported.

Alternatives for your proposal

The work around use today is a fork of the bqwriter master. There the OptionFn parameter is past to the used methods.

Alternatively think about if the bqwriter should have it's own OptionFn pattern to handle optional parameters sent in on create.

Version

0.4.1 (Latest)

What platform are you mostly using or planning to use our software on?

Linux

Code of Conduct

• I agree to follow this project's Code of Conduct