jfischoff / tmp-postgres Goto Github PK

Create temporary postgres instances

License: BSD 3-Clause "New" or "Revised" License

Haskell 98.92% Shell 1.08%

tmp-postgres's Introduction

tmp-postgres

tmp-postgres provides functions for creating a temporary postgres instance. By default it will create a temporary data directory and a temporary directory for a UNIX domain socket for postgres to listen on.

Here is an example using the exception-safe 'with' function:

with $ \db -> bracket
  (connectPostgreSQL (toConnectionString db))
  close $
  \conn -> execute_ conn "CREATE TABLE foo (id int)"

To extend or override the defaults use withConfig (or startConfig).

tmp-postgres ultimately calls initdb (optionally) , postgres and createdb (optionally).

All of the command line, environment variables and configuration files that are generated by default for the respective executables can be extended.

In general tmp-postgres is useful if you want a clean temporary postgres and do not want to worry about clashing with an existing postgres instance (or needing to ensure postgres is already running).

Here are some different use cases for tmp-postgres and their respective configurations:

The default 'with' and 'start' functions can be used to make a sandboxed temporary database for testing.
By disabling initdb one could run a temporary isolated postgres on a base backup to test a migration.
By using the 'stopPostgres' and 'withRestart' functions one can test backup strategies.

WARNING!! Ubuntu's PostgreSQL installation does not put initdb on the PATH. We need to add it manually. The necessary binaries are in the /usr/lib/postgresql/VERSION/bin/ directory, and should be added to the PATH

echo "export PATH=$PATH:/usr/lib/postgresql/VERSION/bin/" >> /home/ubuntu/.bashrc

Installation

macOS

$ brew install postgres
$ stack install tmp-postgres

Ubuntu

Ubuntu's PostgreSQL installation does not put initdb on the PATH. We need to add it manually.

$ sudo apt-get install postgresql-VERSION
$ echo "export PATH=$PATH:/usr/lib/postgresql/VERSION/bin/" >> /home/ubuntu/.bashrc
$ stack install tmp-postgres

GitHub Actions

Assuming you are using one of the standard workflow containers, simply add workflow steps which install postgresql as above. For example,

jobs:
  build:
    runs-on: ubuntu-18.04
    steps:
      - name: "Install postgresql"
        run: "sudo apt-get update && sudo apt-get install postgresql-10"

      # ...

      - name: "Run test"
        run: |
          export PATH=$PATH:/usr/lib/postgresql/10/bin
          stack test

tmp-postgres's People

Contributors

Stargazers

Watchers

Forkers

tebello-thejane blaisorblade ajmcmiddlin creichert mwotton paweln1986 circuithub edutainmentlive catdarick shinzui tonyalaribe georgefst theoddler noteed tommyengstrom lambdamechanic 9999years

tmp-postgres's Issues

Provide helpers for accessing nested data in DB

Change defaultConfig doc to mention IpAddress and UnixSocket

Right now it refers to CIpAddress and CUnixSocket

Add ability to specify the temporary directory root directory

README example wrong

The README gives the following as an example

result <- start []
case result of
  Left err -> print err
  Right tempDB -> do
     -- Do stuff
     stop tempDB

but it seems it should be more like

result <- start defaultOptions
case result of
  Left err -> print err >> pure Nothing
  ...

Include stderr when in `InitDbFailed` and `CreateDbFailed`

This will require teeing the output

Test failures

On Stackage nightly. Full log

Database.Postgres.Temp.Internal
  Database.Postgres.Temp.Internal
    startWithLogger/stop
      deletes the temp dir and postgres on exception in InitDB
      deletes the temp dir and postgres on exception in WriteConfig FAILED [1]
      deletes the temp dir and postgres on exception in FreePort FAILED [2]
      deletes the temp dir and postgres on exception in StartPostgres FAILED [3]
      deletes the temp dir and postgres on exception in WaitForDB FAILED [4]
      deletes the temp dir and postgres on exception in CreateDB FAILED [5]
      deletes the temp dir and postgres on exception in Finished FAILED [6]
      creates a useful connection string and stop still cleans up FAILED [7]
      can override settings FAILED [8]

Failures:

  test/Database/Postgres/Temp/InternalSpec.hs:37: 
  1) Database.Postgres.Temp.Internal.Database.Postgres.Temp.Internal.startWithLogger/stop deletes the temp dir and postgres on exception in WriteConfi
g
       did not get expected exception: Except

  test/Database/Postgres/Temp/InternalSpec.hs:37: 
  2) Database.Postgres.Temp.Internal.Database.Postgres.Temp.Internal.startWithLogger/stop deletes the temp dir and postgres on exception in FreePort
       did not get expected exception: Except

  test/Database/Postgres/Temp/InternalSpec.hs:37: 
  3) Database.Postgres.Temp.Internal.Database.Postgres.Temp.Internal.startWithLogger/stop deletes the temp dir and postgres on exception in StartPostg
res
       did not get expected exception: Except

  test/Database/Postgres/Temp/InternalSpec.hs:37: 
  4) Database.Postgres.Temp.Internal.Database.Postgres.Temp.Internal.startWithLogger/stop deletes the temp dir and postgres on exception in WaitForDB
       did not get expected exception: Except

  test/Database/Postgres/Temp/InternalSpec.hs:37: 
  5) Database.Postgres.Temp.Internal.Database.Postgres.Temp.Internal.startWithLogger/stop deletes the temp dir and postgres on exception in CreateDB
       did not get expected exception: Except

  test/Database/Postgres/Temp/InternalSpec.hs:37: 
  6) Database.Postgres.Temp.Internal.Database.Postgres.Temp.Internal.startWithLogger/stop deletes the temp dir and postgres on exception in Finished
       did not get expected exception: Except

  test/Database/Postgres/Temp/InternalSpec.hs:43: 
  7) Database.Postgres.Temp.Internal.Database.Postgres.Temp.Internal.startWithLogger/stop creates a useful connection string and stop still cleans up
       uncaught exception: ErrorCall (InitDBFailed (ExitFailure 127)
       CallStack (from HasCallStack):
         error, called at test/Database/Postgres/Temp/InternalSpec.hs:50:27 in main:Database.Postgres.Temp.InternalSpec)

  test/Database/Postgres/Temp/InternalSpec.hs:58: 
  8) Database.Postgres.Temp.Internal.Database.Postgres.Temp.Internal.startWithLogger/stop can override settings
       uncaught exception: ErrorCall (InitDBFailed (ExitFailure 127)
       CallStack (from HasCallStack):
         error, called at test/Database/Postgres/Temp/InternalSpec.hs:64:29 in main:Database.Postgres.Temp.InternalSpec)

Randomized with seed 1122578657

Reuse module description in README.md and cabal description

I need to have only one top level description

Support faster db creation setup

Starting a whole process for a test is pretty time consuming. However creating a database, migrating it and using it as a template to start other tests is probably faster.

After we have benchmarks (#52) we can test this theory and if it is faster we should add some support for this workflow.

Use a cabal formatter

Add `silent` defaults

Add silentDefaultConfig and silentProcessConfig which log to /dev/null

Reverse topologically sort the config types in the doc

Create soak test to make sure resources are properly cleaned up

Add benchmarks

Doc mentions`defaultConfig` sets the client connection params but it does not

The overview documentations recommends starting with the defaultConfig because it sets the postgres db connection info. It doesn't but everything still works. Need to document what is actually happening.

Add hint

This should run in CI and locally

Remove Partial suffix and add a Complete suffix

Include PartialPlan in complete plan error

Create vagrant files with different linux distributions

For development it would be nice to be able to test against multiple platforms. Having different vagrant files for different distributions would help.

Provide exe?

Hi,

If you are interested, can you provide this as an exe as well?

Thanks

Add additional test to cover missing cases of `optionsToDefaultConfig`

There is some coverage of optionsToDefaultConfig but not enough

Place `Resources` after `Config`

Lenses for configuration

Having stock lenses for configuration might make mappending to defaultConfig easier.

defaultPostgresConf extra = defaultConfig <> mempty
  { configPlan = mempty
    { partialPlanConfig = extra
    }
  }

vs.

defaultPostgresConf extra = defaultConfig & configPlanL . partialPlanConfigL <>~ extra

Run code coverage on CI

Remove `Lastoid` only accumulate configuration

Being able to replace configuration is too confusing. It leads to situations with createdb where mappend on the default to add args to the createdb plan makes a unexpectedly incomplete plan.

Instead we should remove Lastoid and only accumulate plans.

To get around the issues with environment variables we should make a type that means "inherit the process environment variables" and "add these specific ones". During the creation of the complete plan the inherit constructor signals that getEnvironment should be called.

Log out `PartialPlan`s and `Plans` etc

We should make noisyDefaultConfig that logs out everything and it should logout the combined PartialPlan and complete Plan.

Investigate not finding a free port when listening on a UNIX domain socket

I am doing this because pg_tmp did but I don't think I have to. Should verify.

"createDB" appears to also be incorrect.

As with the initDB issue, createDB also appears to be misspelt. At least, it fails for me on Ubuntu, and I can't find any evidence that the capitalisation would work on any other platform.

Assuming that I'm not mistaken about createDB never being the correct name, I shall create a PR to change it to createdb now.

createdb often starts before the database is truly ready

See this output:

The files belonging to this database system will be owned by user "ollie".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  en_US.UTF-8
  CTYPE:    en_GB.UTF-8
  MESSAGES: en_US.UTF-8
  MONETARY: en_US.UTF-8
  NUMERIC:  en_US.UTF-8
  TIME:     en_US.UTF-8
The default text search configuration will be set to "english".

Data page checksums are disabled.

creating directory /tmp/tmp-postgres11933/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    /home/ollie/.nix-profile/bin/pg_ctl -D /tmp/tmp-postgres11933/data -l logfile start

LOG:  connection received: host=[local]
LOG:  incomplete startup packet
LOG:  database system was shut down at 2017-10-19 10:17:28 GMT
LOG:  connection received: host=[local]
FATAL:  the database system is starting up
LOG:  MultiXact member wraparound protections are now enabled
LOG:  connection received: host=[local]
FATAL:  the database system is starting up
createdb: could not connect to database template1: FATAL:  the database system is starting up
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
LOG:  received fast shutdown request
LOG:  aborting any active transactions
LOG:  autovacuum launcher shutting down
LOG:  shutting down
LOG:  database system is shut down
query-tests: CreateDBFailed (ExitFailure 1)

Repeatedly running this command does eventually work. Notice that createdb has failed before the output database system is ready.

Use `Control.Applicative.Lift.Errors` instead of `Validation` from the `either` package

No reason to add the dependency when the Control.Applicative.Lift.Errors is fine.

Clean up tests

The test file is fugly

`startConfig` documentation is out of date

The documentation describes the process for creating a final Plan incorrectly

Cannot start the server if PGUSER environment variable is set

I have PGUSER=u23sq215lddvbv set when I run my tests, as it's used to connect to a shared staging database. My tests fail with the log:

LOG:  connection received: host=[local]
LOG:  connection authorized: user=u23sq215lddvbv database=template1
FATAL:  role "u23sq215lddvbv" does not exist
LOG:  connection received: host=[local]
LOG:  connection authorized: user=u23sq215lddvbv database=template1
FATAL:  role "u23sq215lddvbv" does not exist
LOG:  connection received: host=[local]
LOG:  connection authorized: user=u23sq215lddvbv database=template1
FATAL:  role "u23sq215lddvbv" does not exist
LOG:  connection received: host=[local]
LOG:  connection authorized: user=u23sq215lddvbv database=template1
FATAL:  role "u23sq215lddvbv" does not exist
LOG:  connection received: host=[local]
LOG:  connection authorized: user=u23sq215lddvbv database=template1
FATAL:  role "u23sq215lddvbv" does not exist
LOG:  connection received: host=[local]
LOG:  connection authorized: user=u23sq215lddvbv database=template1
FATAL:  role "u23sq215lddvbv" does not exist
LOG:  connection received: host=[local]
LOG:  connection authorized: user=u23sq215lddvbv database=template1
FATAL:  role "u23sq215lddvbv" does not exist

ad infinitum.

I believe the problem is the call waitDB doesn't specify a user, so it's inheriting whatever is in the environment.

Perhaps it would be good to run the PostgreSQL executables in an empty/controlled environment?