Convert camel case bigquery column names to snake case about jsonschema-transpiler HOT 8 CLOSED

This is another backwards incompatible change, we should be prepared to drop data at the end of it. Here is where we currently modify properties to satisfy BigQuery column naming:

from jsonschema-transpiler.

This is another backwards incompatible change, we should be prepared to drop data at the end of it.

Yes, I want to roll together any remaining disruptive changes in the next few weeks so that we can deploy reliable prod stacks. In particular, I'd love to roll this together with deploying a final main ping schema representation.

from jsonschema-transpiler.

in direct2parquet we did snake case partially because of the issues with camel case described here: https://docs.aws.amazon.com/athena/latest/ug/tables-databases-columns-names.html

from jsonschema-transpiler.

We discussed this in the data platform team meeting today, and there was a question as to where this should happen in the pipeline. The choices are:

1. Transform in the Decoder immediately after validating the payload using the JSON schema, meaning that the decoded streams in PubSub also have snake_case normalized payloads
2. Transform in the BigQuery sink
3. Transform in BigQuery via a view generated from the schema

I worry that trying to handle this in a BigQuery view won't be tenable. It's potentially a very large amount of query text for the view, which could introduce performance issues.

from jsonschema-transpiler.

I worry that trying to handle this in a BigQuery view won't be tenable. It's potentially a very large amount of query text for the view, which could introduce performance issues.

that could also be an issue with bigquery query size quota:

• Maximum unresolved standard SQL query length — 1 MB

• Maximum resolved legacy and standard SQL query length — 12 MB

  The limit on resolved query length includes the length of all views and wildcard tables referenced by the query.

from jsonschema-transpiler.

See proposal document: https://docs.google.com/document/d/1lY0yGiC8Okx0eJAzI_Pik3-kW4CgNBN2OhMLHlM_uLQ/edit

from jsonschema-transpiler.

The snake casing document is closing today. After some refactoring about where we apply snake casing, we are still planning to use snake case in BQ, so this change is needed.

from jsonschema-transpiler.

@acmiyaguchi is going to work on this early next week.

from jsonschema-transpiler.