simonsobs / acondbs Goto Github PK

• Implement the database migration with Flask-Migrate

• Implement authentication for GraphQL mutation

Implement feature to push CSV files to GitHub

Implement feature to export the DB contents to CSV files

Make the setup similar to that of nextline-graphql.

add pagination to GraphQL queries

This is to make the code similar to nextline and, eventually, turn ProductDB into plugins of nextline-graphql.

• Develop a Python client library (Python API)
  • Status
    • A repo was created for the development: https://github.com/simonsobs/productdb
  • Plans
    • Proceed for the development

• Implement GraphQL create mutations for all product objects

Although ProductDB and Nextline appear quite different, their backends are actually very similar. Both are GraphQL servers with SQLAlchemy. Nextline has an additional feature of executing Python code. ProductDB and Nextline can be
implemented as the same pluggable framework with different plugins.

The Nextline backend is already implemented as a pluggable framework with plugins. Nextline-graphql is the pluggable framework (It needs to be renamed for clarity). ProductDB can be turned into plugins of nextline-graphql.

The advantage of this approach is large. Generic plugins can be used by both projects. For example, we can implement a plugin for full-text DB search and use it for both ProductDB and Nextline. In addition, site-specific features can be implemented as plugins. For example, we can develop a plugin for ProductDB for SO without affecting ProductDB for ACT.

It is not clear what will happen to the backup if DB changes while a backup is being taken.

In the current implementation, the backup to CSV files is taken table by table. It might be possible that a change in the DB across multiple tables is reflected in tables backed up later in the backup but not in tables backed up earlier, which will end up with an inconsistent set of backup CSV files.

It is not clear if it is sufficient to use the same SQLAlchemy connection during the backup.

backend for simonsobs/acondbw#9

Currently, the combination of a product type and product name needs to be unique. This will be an inconvenient constraint in the future, for example, if acondbs is ported to manage other types of data. In fact, it caused a problem when importing Yelp data.

• Implement notifications of new products and updates
  • e.g., to email, slack
• suggested in simonsobs/productdb#5

Implement feature to trigger the backup

• Upgrade to Graphene version 3
  • Status
    • Acondbs uses Graphene version 2 as of the commit d5e315e
    • Graphene version 3 is in beta as of September 2020
  • Plans
    • Investigate the impact of upgrading after Graphene version 3 is released
    • Upgrade when ready

Implement feature to manage CSV files with git

Implement features to back up the database

The initial backup plan: export the DB contents to CSV files and push them to GitHub.

Features to implement

• export the DB contents to CSV files #15
• manage CSV files with git #16
• push CSV files to GitHub #17
• trigger the backup #18
  • the backup needs to be taken whenever the DB contents change
  • the backup should not happen too frequently so as not to stretch the server too much