rolang / dumbo Goto Github PK
View Code? Open in Web Editor NEWSimple database migration tool for Postgres with skunk on JVM and Native
License: MIT License
Simple database migration tool for Postgres with skunk on JVM and Native
License: MIT License
E.g. for operations like CREATE INDEX CONCURRENTLY
...
Skunk changed the tracing library in v 1.0.0-M1, making dumbo unusable with it
When testing this library we noticed that it was not possible to run it through sbt with scala 2.13.
It results in this exception:
java.nio.file.FileSystemNotFoundException: null
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156)
at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:142)
at java.base/java.nio.file.Path.of(Path.java:208)
at java.base/java.nio.file.Paths.get(Paths.java:98)
at dumbo.ResourceFilePath$.$anonfun$fromResourcesDir$3(ResourceFilePath.scala:25)
at delay @ dumbo.ResourceFilePath$.$anonfun$fromResourcesDir$2(ResourceFilePath.scala:24)
at delay @ dumbo.ResourceFilePath$.fromResourcesDir(ResourceFilePath.scala:22)
at flatMap @ dumbo.ResourceFilePath$.fromResourcesDir(ResourceFilePath.scala:22)
For this code:
Dumbo
.withResourcesIn[F]("db/migration")
.apply(
sessionResource = Session.single[F](
host = dbConf.host.value,
port = dbConf.port.value,
user = dbConf.user.value,
database = dbConf.db,
password = dbConf.password.value.some,
),
defaultSchema = schema.getOrElse("public"),
)
.runMigration
.flatMap { result =>
Async[F].delay(println(s"Migration completed with ${result.migrationsExecuted} migrations"))
}
After digging a bit it seems like Paths.get
does not handle jar files jar:file:/Users/.../target/bg-jobs/sbt_fac89327/job-21/target/92a8dfce/9d592533/test_2.13-27e70d62.jar!/db/migration
Any advice on how to proceed?
Repeatable migrations have a description and a checksum, but no version. Instead of being run just once, they are (re-)applied every time their checksum changes.
This is very useful for managing database objects whose definition can then simply be maintained in a single file in version control. They are typically used for
- (Re-)creating views/procedures/functions/packages/...
- Bulk reference data reinserts
Within a single migration run, repeatable migrations are always applied last, after all pending versioned migrations have been executed. Repeatable migrations are applied in the order of their description.
It is your responsibility to ensure the same repeatable migration can be applied multiple times. This usually involves making use of CREATE OR REPLACE clauses in your DDL statements.
Naming:
Hi! I just noticed that loading the migration files from the resources directory doesn't work on windows.
In ResourcesReader getClass().getResourceAsStream(path.toString)
it uses the platform path separators which turns the resource path into \\db\migration\V001__init.sql
but java only understands /db/migration/V001__init.sql
so I get a NPE on getResourceAsStream.
fs2.CompositeFailure: Multiple exceptions were thrown (2), first java.lang.NullPointerException: Cannot invoke "java.io.InputStream.read(byte[], int, int)" because "is$1" is null
at fs2.CompositeFailure$.apply(CompositeFailure.scala:58)
at fs2.CompositeFailure$.apply(CompositeFailure.scala:45)
at fs2.Pull$.addError$1(Pull.scala:1173)
at fs2.Pull$.viewCont$1(Pull.scala:1180)
at fs2.Pull$.$anonfun$compile$22(Pull.scala:1217)
at get @ skunk.util.Pool$.free$1(Pool.scala:171)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
at flatMap @ fs2.Pull$.$anonfun$compile$21(Pull.scala:1215)
at update @ skunk.util.StatementCache$$anon$2.put(StatementCache.scala:48)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
at update @ skunk.util.StatementCache$$anon$2.put(StatementCache.scala:48)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
at fromSync @ de.bitmarck.bms.zpc.datahub.Main$.applicationResource(Main.scala:75)
at uncancelable @ fs2.Compiler$Target.uncancelable(Compiler.scala:165)
at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
at handleErrorWith @ fs2.Compiler$Target.handleErrorWith(Compiler.scala:161)
at modify @ skunk.util.Pool$.$anonfun$ofF$2(Pool.scala:100)
Caused by: java.lang.NullPointerException: Cannot invoke "java.io.InputStream.read(byte[], int, int)" because "is$1" is null
at fs2.io.package$.$anonfun$readInputStream$3(io.scala:48)
Hi @rolang! Big fan of Flyway and the Typelevel ecosystem! Thanks for putting this open source project together.
I'm using dumbo
for a pretty simple CLI application. It'd be great to be able to fine-tune the text output from dumbo
that's currently produced via Console[F].println
using a logging framework; as a dependency it'd be even better if it leveraged SLF4J via something like log4cats-slf4j
.
I'd be happy to take a crack at this myself if it's something you'd be interested in!
Scan given location for migration scripts recursively (as Flyway does).
E.g. include scripts from sub directories given structure like
migrations
V1__a.sql
subdir1
V2__b.sql
subdir2
V3__c.sql
When I tried out Dumbo for the first time (on a cockroachDB database that has been working with Skunk+Flyway for a while), I hit an error. Excess stack traces are boring so I trimmed for relevance.
skunk.exception.DecodeException:
π₯
π₯ DecodeException
π₯
π₯ Problem: Decoding error.
π₯ Detail: This query's decoder was unable to decode a row of data.
π₯
π₯ The statement under consideration was defined
π₯ at Β«skunk internalΒ»:0
π₯
π₯ SELECT oid typid, typname, typarray, typrelid
π₯ FROM pg_type
π₯ WHERE typnamespace IN (
π₯ SELECT oid
π₯ FROM pg_namespace
π₯ WHERE nspname = ANY(current_schemas(true))
π₯ )
π₯
π₯ The row in question returned the following values (truncated to 15 chars).
π₯
π₯ typid oid -> 4294967108 βββ java.lang.NumberFormatException (see below)
π₯ typname name -> pg_aggregate
π₯ typarray oid -> 0
π₯ typrelid oid -> 4294967108
π₯
π₯ The decoder threw the following exception:
π₯
π₯ java.lang.NumberFormatException: For input string: "4294967108"
π₯ java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
π₯ java.base/java.lang.Integer.parseInt(Integer.java:668)
π₯ java.base/java.lang.Integer.parseInt(Integer.java:786)
π₯ scala.collection.StringOps$.toInt$extension(StringOps.scala:910)
π₯ skunk.util.Typer$ProtocolOps.$init$$$anonfun$4(Typer.scala:180)
π₯ skunk.Codec$.simple$$anonfun$2(Codec.scala:113)
...
Root cause is a bug in Skunk's oid
Codec, which tries to fit an unsigned Int into an Int π±
This potentially has a lot of surface area in Skunk's Typer and may not be a straightforward fix.
As a possibly easier workaround, would you consider some config option to instead use the simpler Typer.Strategy.BuiltinsOnly
in Dumbo, which IIUC won't need to hit the problematic code?
Add test of the command line docker build before publishing the docker image.
Add support for reading migration files from embedded resources
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.