ota-insight / bqwriter Goto Github PK
View Code? Open in Web Editor NEWStream data into Google BigQuery concurrently using InsertAll() or BQ Storage.
License: Apache License 2.0
Stream data into Google BigQuery concurrently using InsertAll() or BQ Storage.
License: Apache License 2.0
bqwriter/internal/bigquery/storage/client.go
Line 144 in 8cb7bcd
(nil, nil)
which smells like an anti-pattern to me? For now we capture this case by returning a new error in an unreleased patch ourselves:
Weird thing is that this only appears in a new service written by a new joiner while existing services using bqwriter have never seen such a crash (because the library would crash if you give a nil result to that ch, given it does not expect that).
We need to investigate if this is because of a mistake on our end or just something that needs to be documented or something that might need to be fixed or is already fixed on bigquery's side.
0.7.0 (Latest)
MacOS, Linux
No response
No response
We currently support the insertAll API and soon also the storage API. What we do not yet support is batch loading as documented in https://cloud.google.com/bigquery/docs/batch-loading-data. They give an example of a single file, but we could do it for any number of files as well as any reader in general.
Need to investigate the specifics, but it does look like it is still in scope.
It's a different kind of use case of BQWriter, still writing data into BQ but for an entirely different purpose. For those purposes batch loading is better suited as the timing isn't as critical, and with that a reduction of cost is a given.
Do not support it and explicitly document so instead.
0.3.1 (Latest)
MacOS
No response
Use the upstream bigquery/storage/managedwriter
package instead of our forked version.
One of the issues that ideally is resolved prior to doing so is googleapis/google-cloud-go#5094.
The main motivation is that we have less code to maintain ourselves.
The alternative is to either not use a managedWriter in which case we're going to reinvent the wheel most likely. The only other obvious option is to continue using our fork but that means we'll also need to maintain it ourselves, and do so mostly alone.
0.3.1 (Latest)
MacOS
To change the default connection url you can use the optionsFn parameters to the bigquery.NewClient.
Making the impact as minimum as possible and not changing the behavior used today. The option.ClientOption can be part of the method calls where the bigquery.NewClient is called.
"google.golang.org/api/option"
e.g:
func NewStreamer(ctx context.Context, projectID, dataSetID, tableID string, cfg *StreamerConfig, opts ...option.ClientOption) (*Streamer, error)
where then they can be passed on to (storage, batch and insertall) e.g:
client, err := storage.NewClient(
projectID, dataSetID, tableID,
encoder, protobufDescriptor,
logger,
opts...,
and the pass it on to the google bigquery client:
writer, err := managedwriter.NewClient(ctx, projectID, opts...)
To use in test code:
bqWriter, err := bqwriter.NewStreamer(
ctx, projectId, datasetId, tableId,
&bqwriter.StreamerConfig{
...
},
option.WithEndpoint("localhost:9050"),
option.WithoutAuthentication(),
)
if err != nil {
panic(err)
}
The motivation for this change:
To use the 'https://github.com/goccy/bigquery-emulator' as part of the integration tests during development and deploy pipeline.
bigquery-emulator is a locally running biguery emulator, that easily can run during testing. Fort this to work though, local connection needs to be supported.
The work around use today is a fork of the bqwriter master. There the OptionFn parameter is past to the used methods.
Alternatively think about if the bqwriter should have it's own OptionFn pattern to handle optional parameters sent in on create.
0.4.1 (Latest)
Linux
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.