Comments (8)
This is another backwards incompatible change, we should be prepared to drop data at the end of it. Here is where we currently modify properties to satisfy BigQuery column naming:
jsonschema-transpiler/src/ast.rs
Lines 450 to 464 in a532e90
from jsonschema-transpiler.
This is another backwards incompatible change, we should be prepared to drop data at the end of it.
Yes, I want to roll together any remaining disruptive changes in the next few weeks so that we can deploy reliable prod stacks. In particular, I'd love to roll this together with deploying a final main ping schema representation.
from jsonschema-transpiler.
in direct2parquet we did snake case partially because of the issues with camel case described here: https://docs.aws.amazon.com/athena/latest/ug/tables-databases-columns-names.html
from jsonschema-transpiler.
We discussed this in the data platform team meeting today, and there was a question as to where this should happen in the pipeline. The choices are:
- Transform in the Decoder immediately after validating the payload using the JSON schema, meaning that the decoded streams in PubSub also have snake_case normalized payloads
- Transform in the BigQuery sink
- Transform in BigQuery via a view generated from the schema
I worry that trying to handle this in a BigQuery view won't be tenable. It's potentially a very large amount of query text for the view, which could introduce performance issues.
from jsonschema-transpiler.
I worry that trying to handle this in a BigQuery view won't be tenable. It's potentially a very large amount of query text for the view, which could introduce performance issues.
that could also be an issue with bigquery query size quota:
Maximum unresolved standard SQL query length — 1 MB
Maximum resolved legacy and standard SQL query length — 12 MB
The limit on resolved query length includes the length of all views and wildcard tables referenced by the query.
from jsonschema-transpiler.
See proposal document: https://docs.google.com/document/d/1lY0yGiC8Okx0eJAzI_Pik3-kW4CgNBN2OhMLHlM_uLQ/edit
from jsonschema-transpiler.
The snake casing document is closing today. After some refactoring about where we apply snake casing, we are still planning to use snake case in BQ, so this change is needed.
from jsonschema-transpiler.
@acmiyaguchi is going to work on this early next week.
from jsonschema-transpiler.
Related Issues (20)
- `build.rs` causes `cargo publish` to fail HOT 3
- Support `date-time` format in JSON schemas HOT 1
- Add integration tests against reference data
- Validate BigQuery schemas by round-trip via Avro HOT 1
- Validate input jsonschema before transpiling HOT 1
- Use `TIMESTAMP` instead of `DATETIME` in BigQuery HOT 1
- Consider supporting legacy BigQuery types HOT 4
- Provide an error handling mechanism to drop, cast, or panic on under-specified fields HOT 4
- `--resolve drop` fails on simple schema with incompatible multi-typed items
- Support PubSub schemas
- Support bytes as a data type HOT 1
- Union of string and bytes should be bytes
- Add option to make all fields nullable for BQ schemas
- Decide how to handle union types (like [string, int]) in BQ HOT 10
- Gather Data Science feedback on union and tuple types HOT 6
- Fail mozilla-pipeline-schemas CI if schemas contain nullable array elements HOT 2
- doc.rs fails to build as of v1.6.0
- `<unknown>` property field is dropped from an object HOT 1
- [Proposal] Add python bindings
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jsonschema-transpiler.