brazil-data-cube / bdc-db Goto Github PK

The packages LCCS-DB and BDC-Catalog store tables in different schemas. Packages could register the name of the schemas used inside. Then the CLI could use this register to create the namespaces.

We face the following error while executing bdc-db alembic upgrade (on cube-builder/collection-builder)

alembic.util.exc.CommandError: Can't locate revision identified by 'f639d83487b6'

We have noticed that a few files on repository has a mixed CRLF and LF line ending.

The db command group should have the following commands:

• init - Initialize a database instance.

• create-schema - Create database schema (tables).

• drop-schema - Drop database schema (tables).

• destroy - Destroy database instance.

We should rename the bdc_db.sqltypes.JSONSchemaType to JSONB to keep data semantic.

We should add test case for RefResolver $ref.

Migrate the build pipeline to drone ci.

We should review the package base libraries and their dependencies to help pip to solve dependency graph instead making alot downloads.

The table PostGIS spatial_ref_sys table can be used in different packages. The BDC-Catalog package has a model for this table and could move its definition to the BDC-DB package.

File "/usr/local/lib/python3.7/site-packages/bdc_db/cli.py", line 15, in <module>
--
  File "/usr/local/lib/python3.7/site-packages/bdc_db/cli.py", line 15, in <module>
    from sqlalchemy_utils.functions import (create_database, database_exists,
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
    from .aggregates import aggregated  # noqa
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
    from .functions.orm import get_column_key
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
    from .database import (  # noqa
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
    from .orm import quote
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
    from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' 

Related with sqlalchemy/sqlalchemy#6000

• drop-triggers

• drop-scripts

• setup.py
• setup.cfg
• run-tests.sh
• pytest.ini
• MANIFEST.in
• LICENSE
• .gitignore
• bdc_db/version.py
• bdc_db/__init__.py

We should consider to remove the Flask-SQLAlchemy and use SQLAlchemy directly. It allow us to extend our library on others frameworks like FastAPI.

Keep in mind to configure database model to support mult-threaded sessions.

kvesteri/sqlalchemy-utils#432

• command added to db command group

• alembic command group for managing upgrade recipes

The BDC-DB module should support 1.4X version since it was solved #48

The command line load-scripts should have an option to load specific module name scripts.

For example:
If we have bdc_collectors, myapp and myotherapp registered, and i would like to load only data from bdc_collectors and myapp:

bdc-db load-scripts --modules bdc_collectors,myapp

When executing the CLI load-file command the following message appears:

click.echo(f'Loading file {file}...', bold=True, fg='yellow')
TypeError: echo() got an unexpected keyword argument 'bold'

Suggestion:

Change click.echo to click.secho.

Tasks:

• Inform the right version in: bdc_db/version.py
• Run all the tests.
• Review the CHANGES.rst.
• Review links in the documentation for the stable version (README.rst).
• Create a tag named: v0.4.0.
• In the master branch increase version in bdc_db/version.py to 0.6.0a.
• Notify to the community the release of a new version.

The custom data type bdc_db.sqltypes.JSONB is giving a warning on SQLAlchemy 1.4+:

SAWarning: TypeDecorator JSONB(astext_type=Text()) will not produce a cache key because the cache_ok attribute is not set to True. This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions. Set this attribute to True if this type object's state is safe to use in a cache key, or False to disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)

It should set the property cache_ok = True on class level to avoid performance degradation while comparing/set value to model.