thinkingmachines / bqup Goto Github PK

View Code? Open in Web Editor NEW

30.0 30.0 11.0 222 KB

(pronounced BACK-wup) BigQuery backup scripts

Home Page: https://pypi.org/project/bqup/

License: MIT License

Python 98.62% Makefile 1.38%

bqup's People

Contributors

Stargazers

Watchers

Forkers

bhuesemann rbw-lpp niarepo gopinath678 bzieba-appfire spartez urtho ajkruse livtanong mkrso cgobb

bqup's Issues

Add option/change view files to .SQL

Change view files to .SQL format for syntax highlighting in github. Mainly for code review purposes

Add support for service account keys

Order of keys in json schemata is not consistent

If i'm not mistaken, it's just a matter of adding sort_keys=True to the json.dumps call.

Behavior confirmed in the ff. environment as per @profwacko :

Python 3.5.3
bqup==0.0.6

script errors on "MODEL" type

ValueError is raised there is a BQ MODEL type in the BQ project. This occurs in projects with BQ ML models e.g. psc-demand.

sample stacktrace:

Traceback (most recent call last):
  File "/home/psc-bqup/.local/bin/bqup", line 11, in <module>
    load_entry_point('bqup', 'console_scripts', 'bqup')()
  File "/home/psc-bqup/bqup/bqup/main.py", line 33, in main
    p = Project(project_id or None, args['--schema'])
  File "/home/psc-bqup/bqup/bqup/project.py", line 16, in __init__
    map(partial(Dataset, self, export_schema), self.client.list_datasets()))
  File "/home/psc-bqup/bqup/bqup/dataset.py", line 22, in __init__
    map(partial(Table, self, export_schema), project.client.list_tables(bq_dataset.reference)))
  File "/home/psc-bqup/bqup/bqup/table.py", line 30, in __init__
    raise ValueError('Unrecognized table type: {}'.format(self.table_type))
ValueError: Unrecognized table type: MODEL

Update README.md

Minimize noise in README
https://www.makeareadme.com/

Optimize exporting without -x

Running without the -x flag should still create empty files for each table.

Ongoing work in the optimization branch

Add support for exporting to filesystem

Import view / table usage stats

First, figure out if this is even possible

View definition doesn't include 'CREATE OR REPLACE VIEW' syntax

Add 'CREATE OR REPLACE VIEW ' in view definition.

Put deps in setup.py

setup(
  ...
  install_requires=[deps here],
)

Use BigQuery Python Client Library

ValueError for materialized view table types

ValueError: Unrecognized table type: MATERIALIZED_VIEW

One click deploy

What's the best way to set up a one-click deploy thing?

Use python3.7 and use f-strings instead of format

Add CODE_OF_CONDUCT and CONTRIBUTING

You can check geomancer's CODE_OF_CONDUCT and CONTRIBUTING

Verbose flag is broken

Support for case insensitive filesystems

Work in https://github.com/thinkingmachines/bqup/tree/case-insensteve

Add class and method documentation in docstring

Add numpy style documentation in the docstrings.

More references:

Add tests

Error when running `bqup`

Traceback (most recent call last):
  File "/Users/jessicatiu/miniconda3/bin/bqup", line 11, in <module>
    load_entry_point('bqup', 'console_scripts', 'bqup')()
  File "/Users/jessicatiu/bqup/bqup/main.py", line 33, in main
    p = Project(project_id or None, args['--schema'])
  File "/Users/jessicatiu/bqup/bqup/project.py", line 16, in __init__
    map(partial(Dataset, self, export_schema), self.client.list_datasets()))
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 199, in _items_iter
    for page in self._page_iter(increment=False):
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 230, in _page_iter
    page = self._next_page()
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 349, in _next_page
    response = self._get_next_page_response()
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 401, in _get_next_page_response
    query_params=params)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/cloud/bigquery/client.py", line 311, in _call_api
    return call()
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request
    self._auth_request, method, url, request_headers)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/auth/credentials.py", line 122, in before_request
    self.refresh(request)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/oauth2/credentials.py", line 136, in refresh
    self._client_secret))
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/oauth2/_client.py", line 237, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/oauth2/_client.py", line 111, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/oauth2/_client.py", line 61, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n  "error" : "invalid_grant",\n  "error_description" : "Bad Request"\n}')