Giter VIP home page Giter VIP logo

applyalter's Introduction

ApplyAlter - tool for applying alterscripts

This tool allows easy and robust deployment process - both manual (releases handed to OPS) and automated (even continuous deployment, to limited extent). Unlike some more radical approaches, applyalter is designed for database-first design and fully supports data changes and migrations (ie not just DDL, but also DML).

Main advantages over ".sql" scripts are:

  • execute single file, zip archive or whole directory
  • commands are executed in transaction (by default)
  • in-build checks to skip creating already existing entities (tables, columns etc.)
  • custom checks to skip changing already changed data
  • each executed alterscript is logged to log table (with execution time, duration and checksum)
  • fine-grained error control - by default every error is fatal, but some of them can be ignored for specific commands
  • support for characters in any encoding (as long as it's well-formed XML)
  • dry-run mode
  • database identification in script metadata
  • attached files (BLOB, CLOB or CSV parsed into columns)
  • environment-only scripts
  • special support for large-scale data changes that cannot fit into single transaction
  • not tied to one specific database (used on db2, postgresql and oracle)

How to build and run

  1. Download sources / clone repository.
  2. Build jar package by maven:
  • maven package
  • If you want to have some JDBC driver embedded, simply activate one of the embed_XXXX profile (for example -Pembed_postgresql_8). See pom.xml content for details. If you need different version, just add a new profile (and please contribute via pull request).
  • Note that there are several environment variables used to store build data (BUILD_ID, BUILD_URL, GIT_BRANCH, GIT_COMMIT). These variables are automatically provided by Hudson or Jenkins build server, but they are completely optional.
  1. Check that the jar file is successfuly built and it is runnable:
    java -jar target/applyalter-*.jar
    
    Basic usage and version info should be displayed.
  2. When running for real, do not forget that non-embedded jdbc drivers must be present beside the executable jar and their names must be in the manifest classpath. Manifest classpath is, by default: db2jcc.jar db2jcc4.jar postgresql.jar ojdbc14.jar ojdbc5.jar

Database configuration

First parameter is always confgiuration file that describes how to connect to database(s). The support of multiple database might be a little confusing here: all database in single configuration file are supposed to form "cluster" and have the same schema except minor difference. This feature is not intended to be used as a way to store all databases in single global configuration file!

Basic format is quite simple: the root element db contains list of database instances:

<db>
  <pginstance>
    <id>brand0</id>
    <type>aden</type>
    <host>localhost</host>
    <!-- <port>5432</port> -->
    <db>brand0</db>
    <user>podlesh</user>
    <pass>some_password</pass>
    <properties>
      <property name="loggerLevel" value="DEBUG"/>
    </properties>
  </pginstance>
  <placeholders>
    <param name="roleRW" value="pgs_brand0_rw"/>
    <param name="roleRO" value="pgs_brand0_ro"/>
  </placeholders>
</db>

Name of the database instance element defines database type, used driver and supported options:

element driver
pginstance org.postgresql.Driver when <pass> is missing or empty, $HOME/.pgpass content is used
dbinstance com.ibm.db2.jcc.DB2Driver remote connection, requires all options including port
db2native com.ibm.db2.jcc.DB2Driver local native connection, uses only db element
oracle-instance oracle.jdbc.driver.OracleDriver Oracle support is only rudimentary.

Each database instance must contain element id with unique identifier of that instance; this identifier must be unique inside single configuration file, but not globally. This value is also used instead of type when that element is missing; this is convenient for the most common case of single database per config file.

element description
type custom identifier of database type or application ; only used to filter alterscripts by matching apropriate element in them
host required by all except db2native
port required by dbinstance, optional for pginstance and oracle-instance
db database name used by DBMS; always required
user username; required by dbinstance, optional for pginstance when $HOME/.pgpass is present and contains match
pass password; required by dbinstance, optional for pginstance when $HOME/.pgpass is present and contains match
properties additional JDBC properties (optional, usually not needed)

Element placeholders contains list of parameters which can be used in apply alter script. Parameters are surrounded with double curly brackets. e.g. if alter script contains {{roleRW}} it will be replaced by pgs_brand0_rw.

Alterscript

Each alterscript is single XML file containing two main sections: metadata and commands. Metadata describe where could the alterscript be executed (it should match the configuration file content, otherwise error is reported) and provide some advanced features. Commands are then executed.

Whenever possible, each alterscript is executed in single transaction, with rollback on error. On success, record about execution is stored to special table APPLYALTER_LOG .

This default behaviour can be altered by setting alterscript autocommit to true. Every sql command in the alterscript is then committed right away after execution. Rollback is NOT available, that's why altertscript is skipped in DRY mode. This should be used with caution.

Package log table and queries

All alterscripts executed in single invocation (ie all commandline arguments) are considered single "package" and their complete checksum is recorded to special table applyalter_pkg. Unlike applyalter_log (which records single alterscripts), this one is not checked in any way and used only to query information about past invocations. Main use case is the "check_mode" of ansible module.

To query this table and list packages (invocations), specify option --query-pkg with path to output file. After standard execution, all records with the same SHA1 checksums and the same database ID are found and written as XML file.

  • Query result also includes this very invocation, as long as it was really executed (ie SHARP mode, which is default).
  • If there are no alterscripts to load (ie the only parameter is configuration file), complete history is dumped for this database.
  • Explicit checksum to query can be specified by option --query-pkg-hash

applyalter's People

Contributors

fafejtao avatar jirinovak avatar lancaricp avatar matvijcuka avatar mikrobestie avatar mira-minarik avatar podlesh avatar radimk avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

applyalter's Issues

Allow non-transactional alters

Sometimes there is a need to run commands outside of the transaction, like creating index in Postgres concurrently. There should be a per-alter option tu run commands without transaction.

efficient remote detection of applied scripts

When integrating with automated deployment tools, we need a way to efficiently report whether some alterscript archive have been already successfully applied or not.
It is, of course, possible to execute applyalter each time, just for case - it's just very time consuming and in case manually invoked deployment, human operators run out of patience quite soon (especially for databases that have no new alterscript in years).

The main challenge here is to decide what level should it operate on: applyalter works on single alterscript level, while release management / continuous deployment tools works with whole artifacts = archives (zip).

Documentation on github

The Applyalter documentation should be on github (either in readme.md or on wiki) and in English. And this should be the reference and most up-to-date documentation.

Default schema for PG is "wasg2"

For some historical reason, the default schema is "wasg2" even for PostgreSQL databases. This causes checks to scan this schema instead of expected "public" when searching for metadata. When the <schema>public</schema> is explicitly set, the checks work as inteneded.

For example my failing check was for the column
<check type="column" name="content" table="internal_note" inverted="true" />
which caused the alter to be skipped even if the column did exist.

simplified configuration

In most (99%) cases, <id> and <type> are the same; <type> should be optional.

Also, the strict check of known types is sometimes overkill; in some cases, it forces creation of fake database.

Incorrect postgresql syntax <migration-id-list>

Migration id list tag has incorrect syntax when applied against posgresql.
Seems to be db2 syntax.

Fails with:
[2016-02-29 08:41:31.472] creating temporary table session.MGR_IDS [2016-02-29 08:41:31.475] creating temporary table by query: DECLARE GLOBAL TEMPORARY TABLE session.MGR_IDS AS (select id from jd.document) DEFINITION ONLY on commit preserve rows NOT LOGGED on rollback delete rows WITH REPLACE ch.ips.g2.applyalter.ApplyAlterException: failed to create temporary table session.MGR_IDS: ERROR: syntax error at or near "TEMPORARY" at ch.ips.g2.applyalter.MigrationIdList.createTempTable(MigrationIdList.java:142) at ch.ips.g2.applyalter.MigrationIdList.execute(MigrationIdList.java:158) at ch.ips.g2.applyalter.ApplyAlter.executeStatement(ApplyAlter.java:509) at ch.ips.g2.applyalter.ApplyAlter.applySingleAlter(ApplyAlter.java:471) at ch.ips.g2.applyalter.ApplyAlter.applyWithoutClosing(ApplyAlter.java:417) at ch.ips.g2.applyalter.ApplyAlter.apply(ApplyAlter.java:394) at ch.ips.g2.applyalter.ApplyAlter.apply(ApplyAlter.java:305) at ch.ips.g2.applyalter.ApplyAlter.main(ApplyAlter.java:773) Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "TEMPORARY" at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1592) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1327) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:451) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:350) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:304) at ch.ips.g2.applyalter.DbUtils.executeUpdate(DbUtils.java:69) at ch.ips.g2.applyalter.MigrationIdList.createTempTable(MigrationIdList.java:134) ... 7 more

Support Java 17

When the applyalter is run on Java 17, the following exception is thrown:

java.lang.ExceptionInInitializerError
	at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:648)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:436)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:376)
	at com.thoughtworks.xstream.XStream.<init>(XStream.java:314)
	at ch.ips.g2.applyalter.ApplyAlter.<init>(ApplyAlter.java:156)
	at ch.ips.g2.applyalter.ApplyAlter.main(ApplyAlter.java:936)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field protected volatile java.util.Properties java.util.Properties.defaults accessible: module java.base does not "opens java.util" to unnamed module @405e4200
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at com.thoughtworks.xstream.core.util.Fields.find(Fields.java:26)
	at com.thoughtworks.xstream.converters.collections.PropertiesConverter.<clinit>(PropertiesConverter.java:41)
	... 6 more

XStream should be updated to more recent version

Safer default behaviour - incremental mode

ApplyAlter should default to incremental mode unless told otherwise.

  1. do incremental mode by default, keep -s for backwards compatibility
  2. add -S parameter to explicitly disable this behaviour
  3. auto-create applyalter log structures if not present, unless -S is specified

Thought: Default behaviour could be enhanced to default to "oneshot" (-S) mode when only one script is present (the script is single xml), and to incremental mode when .zip with multiple alters is present.

structured output

Current plaintext output is suitable for humans, but not for automatic tools like puppet or ansible.
We need option to create structured (json, xml) report that could be safely transported and parsed.

remove "inlined" jdbc drivers

DB2 and postgresql JDBC drivers are currently inlined. This is convenient when used internally, but unsuitable for public version (especially the db2 which is not legally available).

  1. remove from build
  2. add jars to manifest class path
  3. test with automatic or explicit class path

maybe add some documentation

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.