Giter VIP home page Giter VIP logo

timmikeladze / arango-migrate Goto Github PK

View Code? Open in Web Editor NEW
12.0 4.0 3.0 486 KB

🥑 Database migration tools and CLI for ArangoDB. Apply migrations in a transaction-safe manner with optional before/after hooks and dry-run support.

License: MIT License

JavaScript 23.73% TypeScript 76.27%
arango arangodb migrations database database-management migration arango-migrate javascript typescript arangodb-migrate arangodb-migration

arango-migrate's Introduction

arango-migrate

Apply migrations to an ArangoDB in a transaction-safe manner with optional before/after hooks and dry-run support.

Getting Started

yarn add arango-migrate -D

Note: Check out a functioning sample in the arango-migrate-example repository.

Usage

Usage: cli [options]

Options:
  -c, --config <config>  path to a js config file. Defaults to ./config.migrate.js
  -u, --up               run up migrations. Defaults to running all unapplied migrations if no --to parameter is provided
  -d, --down             run down migrations. --to parameter is required
  -t, --to <version>     run migrations to and including a specific version
  -i --init <name>       initialize a new migration file
  -l --list              list all applied migrations
  -dr --dry-run          dry run. Executes migration lifecycle functions but never commits the transaction to the database or writes to the migration history log
  -nh --no-history       Skips writing to the migration history log. Use this with caution since the applied migrations will not be saved in the migration
                         history log, opening the possibility of applying the same migration multiple times and potentially dirtying your data
  -h, --help             display help for command

Configuration

Create a config.migrate.js file in the root of your project. This file contains the database connection information and options for running migrations.

Example:

import 'dotenv/config'

export default {
    dbConfig: {
        databaseName: process.env.ARANGO_NAME,
        url: process.env.ARANGO_URL,
            auth: {
              username: process.env.ARANGO_USERNAME,
              password: process.env.ARANGO_PASSWORD || ''
        }
    },
    autoCreateNewCollections: true, // defaults to true if not specified
    migrationHistoryCollection: 'migration_history', // defaults to 'migration_history' if not specified
    migrationsPath: './migrations'
}

Initialize a new migration

yarn arango-migrate -i new-migration-name

This will create an empty migration file in the migrations directory.

Simple migration example

This migration will create new collections todo, user, an edge collection user_todo and then insert documents in them. Additional lifecycle functions can be added to the migration file, see the full list of options below. If a collection does not exist it will be created by default (set autoCreateNewCollections option to false to disable this behavior).

import { CollectionType } from 'arangojs'

const migration = {
  description: 'Simple migration',
  async collections () {
    // All collections used in this migration must be defined here. A string or an options object can be used.
    return [
      'todo',
      'user',
      {
        collectionName: 'user_todo_edge',
        options: {
          type: CollectionType.EDGE_COLLECTION
        }
      }]
  },
  async up (db, step) {
    // Using the `step` function, add a new document to the collection as part of this migration's transaction.
    await step(async () => await db.collection('todo').save({
      _key: '1',
      name: 'Buy milk'
    }))

    await step(async () => await db.collection('user').save({
      _key: '1',
      name: 'John Doe'
    }))

    await step(async () => await db.collection('user_todo_edge').save({
      _from: 'user/1',
      _to: 'todo/1'
    }))
  }
}

export default migration

Running up migrations

yarn arango-migrate -u

Runs all un-applied migrations.

yarn arango-migrate -u -t 2

Runs all un-applied migrations up to and including migration with version number 2.

Understanding ArangoDB transactions and the step function

Individual migrations are ran within a transaction in order to keep the database in a valid state if a migration fails. The migration's transaction is committed to ArangoDB after the up or down functions are executed.

Observe how the second argument of the up and down functions is a function called step. This is a special function which allows you to add valid ArangoDB operations to the transaction.

For example in order to add a new document to the todo collection.

const up = async (db, step) => {
    const todoItem = await step(async () => await db.collection('todo').save({
        _key: '1',
        name: 'Buy milk'
    }))
    return todoItem;
}

Read more about transactions in ArangoDB

Anatomy of a migration

export interface Migration {
  /**
   * Defines all the collections that will be used as part of this migration.
   * @returns {Promise<Collections>} An array of collection names or an array of collection options.
   */
  collections(): Promise<Collections>;
  /**
   * Optional function that configures how the transaction will be executed. See ArangoDB documentation for more information.
   * @returns {Promise<TransactionOptions>} - The transaction options.
   */
  transactionOptions?: () => Promise<TransactionOptions>;
  /**
   * Optional description of what the migration does. This value will be stored in the migration log.
   */
  description?: string,
  /**
   * Optional function that will be called before the migration's `up` function is executed.
   * @param {Database} db -  Database instance.
   * @returns {Promise<*>} - Value returned will be passed to the `up` function.
   */
  beforeUp?: (db: Database) => Promise<any>
  /**
   * Function that will be called to perform the `up` migration.
   * @param {Database} - db Database instance.
   * @param {StepFunction} - step The `step` function is used to add valid ArangoDB operations to the transaction.
   * @param {*} data Optional value received from the `beforeUp` function.
   */
  up: (db: Database, step: StepFunction, data?: any) => Promise<any>;
  /**
   * Optional function that will be called after the migration's `up` function is executed.
   * @param {Database} db - Database instance.
   * @param {*} data - Value returned from the `up` function.
   * @returns {Promise<*>} - Value returned will be passed to the `afterUp` function.
   */
  afterUp?: (db: Database, data?: any) => Promise<void>
  /**
   * Optional function that will be called before the migration's `down` function is executed.
   * @param {Database} db - Database instance.
   * @returns {Promise<*>} - Value returned will be passed to the `down` function.
   */
  beforeDown?: (db: Database) => Promise<any>
  /**
   * Function that will be called to perform the `down` migration.
   * @param {Database} db - Database instance.
   * @param {StepFunction} - step The `step` function is used to add valid ArangoDB operations to the transaction.
   * @param {*} data - Optional value received from the `beforeDown` function.
   */
  down?: (db: Database, step: StepFunction, data?: any) => Promise<any>;
  /**
   * Optional function that will be called after the migration's `down` function is executed.
   * @param {Database} db - Database instance.
   * @param {*} data - Optional value received from the `beforeDown` function.
   * @returns {Promise<*>} - Value returned will be passed to the `afterDown` function.
   */
  afterDown?: (db: Database, data?: any) => Promise<any>
}

Contributing

To get started with development, clone the repository and install dependencies with your package manager of choice. This project uses yarn by default.

Then copy the .env.example file to .env. The default values should work for most cases.

Now start the ArangoDB docker container by running docker-compose up -d.

Once the container is running make sure everything is working by running yarn test and yarn build. If both commands succeed you are ready to start contributing to this project.

arango-migrate's People

Contributors

dependabot[bot] avatar eonae avatar mrkuzzmin avatar renovate-bot avatar renovate[bot] avatar timmikeladze avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

arango-migrate's Issues

Unable to use in project/package where "type": "module"

Great project but I'm having issues with using it in a project/package when "type": "module" is set in the package.json.
I get the following errors:

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/user42/Documents/ITDev/projects/proj123/config.migrate.js from /Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js not supported.
config.migrate.js is treated as an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which declares all .js files in that package scope as ES modules.
Instead rename config.migrate.js to end in .cjs, change the requiring code to use dynamic import() which is available in all CommonJS modules, or change "type": "module" to "type": "commonjs" in /Users/user42/Documents/ITDev/projects/proj123/package.json to treat all .js files as CommonJS (using .mjs for all ES modules instead).

    at Function.n.validateConfigPath (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:3560)
    at Object.<anonymous> (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:14046) {
  code: 'ERR_REQUIRE_ESM'

// Using config: config.migrate.js
const { database } = require('@abc-lib/load-config').loadConfig();

module.exports = {
  dbConfig: {
    databaseName: database.database,
    url: database.host,
    auth: {
      username: database.user,
      password: database.password,
    },
  },
};

Also tried a config with -C after renaming the config to config.migrate.cjs but that produced other errors

/Users/user42/Documents/ITDev/projects/proj123/config.migrate.cjs:1
const { database } = require('@abc-lib/load-config').loadConfig();
                     ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/user42/Documents/ITDev/projects/proj123/node_modules/@abc-lib/load-config/lib/index.js from /Users/user42/Documents/ITDev/projects/proj123/config.migrate.cjs not supported.
Instead change the require of index.js in /Users/user42/Documents/ITDev/projects/proj123/config.migrate.cjs to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/Users/user42/Documents/ITDev/projects/proj123/config.migrate.cjs:1:22)
    at Function.n.validateConfigPath (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:3560)
    at Object.<anonymous> (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:14046) {
  code: 'ERR_REQUIRE_ESM'
}

// Using config: config.migrate.cjs
const { database } = require('@abc-lib/load-config').loadConfig();

module.exports = {
  dbConfig: {
    databaseName: database.database,
    url: database.host,
    auth: {
      username: database.user,
      password: database.password,
    },
  },
};

And also tried using ESM in the config but got these errors:

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/user42/Documents/ITDev/projects/proj123/config.migrate.js from /Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js not supported.
Instead change the require of config.migrate.js in /Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js to a dynamic import() which is available in all CommonJS modules.
    at Function.n.validateConfigPath (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:3560)
    at Object.<anonymous> (/Users/user42/Documents/ITDev/projects/proj123/node_modules/arango-migrate/dist/cli.js:2:14046) {
  code: 'ERR_REQUIRE_ESM'
}

// Using // config.migrate.js
import { loadConfig } from '@awt-lib/load-config';

const { database } = loadConfig();

export default {
  dbConfig: {
    databaseName: database.database,
    url: database.host,
    auth: {
      username: database.user,
      password: database.password,
    },
  },
};

Would be great to use this but I cannot see how to work around this.

Creating Edges

Is it possible to create edges? Can't find it in source code

change incrementing to timestamp

We use your package in a large development team
we have problems with increment names (in parallel, several developers can create different migrations with the same numbers)
It makes sense to change the name to timestamps

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • chore(deps): update dependency eslint to v9
  • chore(deps): update dependency eslint-plugin-n to v17
  • 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

docker-compose
docker-compose.yml
github-actions
.github/workflows/main.yml
  • xinova/arangodb-action v1
  • actions/checkout v3
  • actions/setup-node v3
  • c-hive/gha-yarn-cache v2
npm
package.json
  • commander ^9.4.1
  • glob ^8.0.3
  • slugify ^1.6.5
  • @ryansonshine/commitizen 4.2.8
  • @ryansonshine/cz-conventional-changelog 3.3.4
  • @types/glob 8.0.0
  • @types/jest 29.2.0
  • @typescript-eslint/eslint-plugin 5.41.0
  • @typescript-eslint/parser 5.41.0
  • arangojs ^8.6.0
  • dotenv 16.0.3
  • eslint 8.26.0
  • eslint-config-standard 17.0.0
  • eslint-plugin-import 2.26.0
  • eslint-plugin-n 16.4.0
  • eslint-plugin-node 11.1.0
  • eslint-plugin-promise 6.1.1
  • eslint-plugin-typescript-sort-keys 2.1.0
  • husky 8.0.1
  • jest 29.2.2
  • lint-staged 13.0.3
  • microbundle 0.15.1
  • release-it 17.0.1
  • ts-jest 29.0.3
  • typescript 4.8.4
  • arangojs >=7.8.0
  • semver >=7.5.2

  • Check this box to trigger a request for Renovate to run again on this repository

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.