elixir-ecto / ecto_sql Goto Github PK
View Code? Open in Web Editor NEWSQL-based adapters for Ecto and database migrations
Home Page: https://hexdocs.pm/ecto_sql
License: Apache License 2.0
SQL-based adapters for Ecto and database migrations
Home Page: https://hexdocs.pm/ecto_sql
License: Apache License 2.0
ecto_sql/lib/ecto/adapters/postgres.ex
Line 40 in 3bad7d2
ecto_sql/lib/ecto/adapters/myxql.ex
Line 24 in 3bad7d2
However neither adapters actually supports this option.
Postgrex.start_link
takes this option but it does something different:
* `:timeout` - Socket receive timeout when idle in milliseconds (default:
`#{@timeout}`);
https://github.com/elixir-ecto/postgrex/blob/v0.14.1/lib/postgrex.ex#L46:L47
We should either change the code or the docs, thoughts? Note, for Ecto.Adapters.MySQL/Mariaex the docs are the same and the code works as advertised.
Erlang/OTP 21 [erts-10.1.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [hipe] [dtrace]
Elixir 1.7.4 (compiled with Erlang/OTP 21)
I'm migrating a library from ecto 2.2 to ecto 3.0.
The library contains a test Repo under test/support/repo.ex
and associated migrations under test/support/migrations/*.exs
.
I've configured the priv:
option for the repo accordingly for the test
Mix config:
config :my_lib, MyLib.Test.Repo,
database: "my_lib_test",
username: "postgres",
password: "postgres",
priv: "test/support"
Running MIX_ENV=test mix ecto.migrations
produces:
Repo: MyLib.Test.Repo
Status Migration ID Migration Name
--------------------------------------------------
Similarly, MIX_ENV=test mix ecto.migrate
does not apply any migrations:
mix ecto.migrate
21:30:12.472 [debug] QUERY OK source="schema_migrations" db=1.0ms
SELECT s0."version"::bigint FROM "schema_migrations" AS s0 FOR UPDATE []
21:30:12.474 [info] Already up
The migration scripts in the test/support/migrations
directory should be applied.
I believe this behaviour is due to the migrations path being initially validated relative to the mix project: https://github.com/elixir-ecto/ecto_sql/blob/master/lib/mix/ecto_sql.ex#L92
Then later re-evaluated relative to the Application.app_dir
: https://github.com/elixir-ecto/ecto_sql/blob/master/lib/ecto/migrator.ex#L36 which evaluates to _build/test/lib/my_lib/test/support/migrations
. As this directory doesn't exist, no migrations are found.
If the new behaviour is intended, then any advice on managing a test-only repository and migrations for a library, such that the migrations are not included in the final hex package would be appreciated.
Thanks
Ecto 3.0 was released which references this project as the upgrade path for database users. I see that a release candidate was published a few days ago. Are there any issues preventing the release candidate getting promoted to stable?
Erlang/OTP 21 [erts-10.1.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]
Elixir 1.7.4 (compiled with Erlang/OTP 21)
PG_VERSION=9.6.6
* ecto 2.2.11 (Hex package) (mix)
locked at 2.2.11 (ecto) 4bb8f117
* postgrex 0.13.5 (Hex package) (mix)
locked at 0.13.5 (postgrex) 3d931aba
OSX 10.14.1
This code:
queryable
|> order_by([desc: :begin_at, desc: :id])
|> distinct([:begin_at, :id])
generates the following request:
SELECT DISTINCT ON (m0."begin_at",
m0."id") m0."id",
...)
...
GROUP BY m0."id"
ORDER BY m0."begin_at",
m0."id",
m0."begin_at" DESC,
m0."id" DESC
SELECT DISTINCT ON (m0."begin_at",
m0."id") m0."id",
...)
...
GROUP BY m0."id"
ORDER BY m0."begin_at" DESC,
m0."id" DESC
It appears there is a difference between these two way of writing the order_by()s
.
See elixir-ecto/ecto#504 for initial discussion on distinct_on+order_by.
See elixir-ecto/ecto#1669 for what seems to be a very similar bug report.
Note that I explicitly select distinct
order_by fields to avoid
SELECT DISTINCT ON expressions must match initial ORDER BY expressions
Current workaround:
EDIT: I haven't tried ecto_sql v3 but code in connection.sql looks similiar enough.
If you create a union
query and call Repo.to_sql,
then change the query to use intersect
and call Repo.to_sql
again, you get the exact same generated SQL (the UNION
isn't changing to INTERSECT
)
I created a failing test case in my fork of ecto_sql
which demonstrates the issue (branch is issue/union-then-intersect
):
I ran this with ECTO_ADAPTER=pg
- I haven't tried it with MySQL.
UNION
should change to INTERSECT
in the second to_sql
call.
1.7.4
postgres 11.1
3.0
postgrex 0.14.1
macos 10.13.6
config :avocado, Avocado.Repo,
migration_default_prefix: "avocadoz"
The database for Avocado.Repo has been created
[info] == Running 20190115121352 Avocado.Repo.Migrations.CreateUsers.change/0 forward
[info] create table avocadoz.users
** (Postgrex.Error) ERROR 3F000 (invalid_schema_name) schema "avocadoz" does not exist
(ecto_sql) lib/ecto/adapters/sql.ex:620: Ecto.Adapters.SQL.raise_sql_call_error/1
(elixir) lib/enum.ex:1314: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql) lib/ecto/adapters/sql.ex:707: Ecto.Adapters.SQL.execute_ddl/4
(ecto_sql) lib/ecto/migration/runner.ex:301: Ecto.Migration.Runner.log_and_execute_ddl/3
(ecto_sql) lib/ecto/migration/runner.ex:110: anonymous fn/2 in Ecto.Migration.Runner.flush/0
(elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto_sql) lib/ecto/migration/runner.ex:108: Ecto.Migration.Runner.flush/0
(stdlib) timer.erl:166: :timer.tc/1
(ecto_sql) lib/ecto/migration/runner.ex:26: Ecto.Migration.Runner.run/7
(ecto_sql) lib/ecto/migrator.ex:211: Ecto.Migrator.attempt/7
(ecto_sql) lib/ecto/migrator.ex:113: anonymous fn/4 in Ecto.Migrator.do_up/4
(ecto_sql) lib/ecto/migrator.ex:193: anonymous fn/3 in Ecto.Migrator.run_maybe_in_transaction/5
(ecto_sql) lib/ecto/adapters/sql.ex:814: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection) lib/db_connection.ex:1349: DBConnection.run_transaction/4
(ecto_sql) lib/ecto/migrator.ex:192: Ecto.Migrator.run_maybe_in_transaction/5
(elixir) lib/task/supervised.ex:89: Task.Supervised.do_apply/2
(elixir) lib/task/supervised.ex:38: Task.Supervised.reply/5
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Adding the prefix to the migration itself yields the same result.
I tried execute "create schema avocadoz;"
and got:
** (RuntimeError) could not find migration runner process for #PID<0.91.0>
(ecto_sql) lib/ecto/migration/runner.ex:288: Ecto.Migration.Runner.runner/0
(ecto_sql) lib/ecto/migration/runner.ex:122: Ecto.Migration.Runner.execute/1
priv/repo/migrations/20190115121352_create_users.exs:4: (module)
(stdlib) erl_eval.erl:680: :erl_eval.do_apply/6
(elixir) lib/code.ex:715: Code.load_file/2
(ecto_sql) lib/ecto/migrator.ex:476: Ecto.Migrator.extract_module/2
(ecto_sql) lib/ecto/migrator.ex:433: anonymous fn/5 in Ecto.Migrator.do_migrate/4
(elixir) lib/enum.ex:3281: Enumerable.List.reduce/3
(elixir) lib/enum.ex:1968: Enum.reduce_while/3
(ecto_sql) lib/ecto/migrator.ex:427: Ecto.Migrator.migrate/4
(ecto_sql) lib/ecto/adapters/sql.ex:814: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection) lib/db_connection.ex:1349: DBConnection.run_transaction/4
obvious
Most database engines that was first developed for *nix operating system stores UUID type binary in order as human readable representation is rendered. As for many things, Microsoft did not respect this standard, so fist part of 4 bytes, bytes in reversed order and next 2 bytes also, after that, other bytes are not changed.
standard order:
<<b1, b2, b3, b4, ?-, b5, b6, ?-, b7, b8, ?-, b9, b10, ?-, b11, b12, b13, b14, b15, b16>>
tds order:
<<b4, b3, b2, b1, ?-, b6, b5, ?-, b8, b7, ?-, b9, b10, ?-, b11, b12, b13, b14, b15, b16>>
Earlier, before ecto 3.x I used custom type Tds.UUID instead of builtin and had to rewrite all support integration tests. Since most of integration test are using UUID this would mean that if I add MsSql adapter to ecto_sql repository, I have to write my own test cases, schemas and migrations in order to cover all cases properly.
So far I don't have idea how to nicely override Ecto.UUID since this is out of adapter reach, any suggestions that do not involve rewriting all that integration cases are welcome!!!
Upgrading
One of our developers created migrations with different file names but with the same module name inside. These migrations contain different instructions inside.
From logs:
[info] == Running 20180107124803 App.Repo.Migrations.AddCoherenceToUser.change/0 forward
...
[info] == Running 20180123144846 App.Repo.Migrations.AddCoherenceToUser.change/0 forward
It worked well
The first migration runs twice with different timestamps, and we have conflicts.
Allow conflicting module names or show warning.
P. S: Is it safe to just rename a module?
Edited: okay, a warning is shown.
I get an errror running my migrations. I'm running postgrex 14.0 with ecto 3.0
Here is the error I get
00:41:38.291 [error] an exception was raised logging %DBConnection.LogEntry{call: :prepare_execute, connection_time: 344317, decode_time: 8247259, params: [], pool_time: 71427753, query: %Postgrex.Query{cache: :reference, columns: nil, name: "", param_formats: nil, param_oids: nil, param_types: nil, ref: nil, result_formats: nil, result_oids: nil, result_types: nil, statement: ["CREATE TABLE ", "IF NOT EXISTS ", [34, "schema_migrations", 34], 32, 40, [[[], [[34, "version", 34], 32, "bigint", [[], []]], ", "], [34, "inserted_at", 34], 32, ["timestamp", "(0)"], [[], []]], [", ", "PRIMARY KEY (", [[], 34, "version", 34], ")"], 41, []], types: nil}, result: {:ok, %Postgrex.Query{cache: :reference, columns: nil, name: "", param_formats: [], param_oids: [], param_types: [], ref: #Reference<0.2077835036.3294363649.48140>, result_formats: [], result_oids: nil, result_types: nil, statement: ["CREATE TABLE ", "IF NOT EXISTS ", [34, "schema_migrations", 34], 32, 40, [[[], [[34, "version", 34], 32, "bigint", [[], []]], ", "], [34, "inserted_at", 34], 32, ["timestamp", "(0)"], [[], []]], [", ", "PRIMARY KEY (", [[], 34, "version", 34], ")"], 41, []], types: {Postgrex.DefaultTypes, #Reference<0.2077835036.3294494721.48090>}}, %Postgrex.Result{columns: nil, command: :create_table, connection_id: 7250, messages: [%{code: "42P07", file: "parse_utilcmd.c", line: "206", message: "relation \"schema_migrations\" already exists, skipping", routine: "transformCreateStmt", severity: "NOTICE", unknown: "NOTICE"}], num_rows: 0, rows: nil}}}: ** (ArgumentError) argument error
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (stdlib) :ets.lookup(Telemetry.HandlerTable, [:union, :repo, :query])
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (telemetry) lib/telemetry/handler_table.ex:59: Telemetry.HandlerTable.list_for_event/1
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (telemetry) lib/telemetry.ex:76: Telemetry.execute/3
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:754: Ecto.Adapters.SQL.log/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (db_connection) lib/db_connection.ex:1303: DBConnection.log/5
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (postgrex) lib/postgrex.ex:188: Postgrex.query_prepare_execute/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:288: Ecto.Adapters.SQL.query!/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (elixir) lib/enum.ex:1314: Enum."-map/2-lists^map/1-0-"/2
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: 00:41:38.329 [error] an exception was raised logging %DBConnection.LogEntry{call: :begin, connection_time: 251680, decode_time: nil, params: nil, pool_time: 36295, query: :begin, result: {:ok, %DBConnection{conn_mode: nil, conn_ref: #Reference<0.2077835036.3294363649.48247>, pool_ref: {:pool_ref, #PID<0.133.0>, #Reference<0.2077835036.3294494721.47914>, nil, #Reference<0.2077835036.3294494721.48135>, #Reference<0.2077835036.3294363649.48246>}}, %Postgrex.Result{columns: nil, command: :commit, connection_id: 7250, messages: [], num_rows: nil, rows: nil}}}: ** (ArgumentError) argument error
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (stdlib) :ets.lookup(Telemetry.HandlerTable, [:union, :repo, :query])
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (telemetry) lib/telemetry/handler_table.ex:59: Telemetry.HandlerTable.list_for_event/1
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (telemetry) lib/telemetry.ex:76: Telemetry.execute/3
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:754: Ecto.Adapters.SQL.log/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (db_connection) lib/db_connection.ex:1303: DBConnection.log/5
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (db_connection) lib/db_connection.ex:760: DBConnection.transaction/3
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:692: Ecto.Adapters.SQL.lock_for_migrations/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/migrator.ex:317: Ecto.Migrator.lock_for_migrations/3
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: 00:41:38.367 [error] an exception was raised logging %DBConnection.LogEntry{call: :prepare_execute, connection_time: 1215295, decode_time: 14453, params: [], pool_time: nil, query: %Postgrex.Query{cache: :reference, columns: nil, name: "ecto_50", param_formats: nil, param_oids: nil, param_types: nil, ref: nil, result_formats: nil, result_oids: nil, result_types: nil, statement: "SELECT s0.\"version\"::bigint FROM \"schema_migrations\" AS s0 FOR UPDATE", types: nil}, result: {:ok, %Postgrex.Query{cache: :reference, columns: ["version"], name: "ecto_50", param_formats: [], param_oids: [], param_types: [], ref: #Reference<0.2077835036.3294363649.48276>, result_formats: [:binary], result_oids: [20], result_types: [Postgrex.Extensions.Int8], statement: "SELECT s0.\"version\"::bigint FROM \"schema_migrations\" AS s0 FOR UPDATE", types: {Postgrex.DefaultTypes, #Reference<0.2077835036.3294494721.48090>}}, %Postgrex.Result{columns: ["version"], command: :select, connection_id: 7250, messages: [], num_rows: 94, rows: [[20170817002627], [20170817003845], [20170817045138], [20170901160212], [20170907160212], [20170908082120], [20170908085748], [20170917070541], [20170917082508], [20170922051950], [20170922192527], [20170923023516], [20170923165044], [20171002105454], [20171007031041], [20171008031041], [20171030105824], [20171212143912], [20180101092243], [20180103215359], [20180105134830], [20180106073049], [20180106103608], [20180106134459], [20180106154245], [20180110072641], [20180115162350], [20180118162023], [20180207012117], [20180211015839], [20180211042058], [20180211123318], [20180218022325], [...], ...]}}}: ** (ArgumentError) argument error
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (stdlib) :ets.lookup(Telemetry.HandlerTable, [:union, :repo, :query])
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (telemetry) lib/telemetry/handler_table.ex:59: Telemetry.HandlerTable.list_for_event/1
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (telemetry) lib/telemetry.ex:76: Telemetry.execute/3
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:754: Ecto.Adapters.SQL.log/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (db_connection) lib/db_connection.ex:1303: DBConnection.log/5
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:531: Ecto.Adapters.SQL.execute!/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:523: Ecto.Adapters.SQL.execute/5
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto) lib/ecto/repo/queryable.ex:147: Ecto.Repo.Queryable.execute/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: 00:41:38.383 [info] Already up
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: Success!
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: 00:41:38.398 [error] an exception was raised logging %DBConnection.LogEntry{call: :commit, connection_time: 3696570, decode_time: nil, params: nil, pool_time: nil, query: :commit, result: {:ok, %Postgrex.Result{columns: nil, command: :commit, connection_id: 7250, messages: [], num_rows: nil, rows: nil}}}: ** (ArgumentError) argument error
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (stdlib) :ets.lookup(Telemetry.HandlerTable, [:union, :repo, :query])
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (telemetry) lib/telemetry/handler_table.ex:59: Telemetry.HandlerTable.list_for_event/1
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (telemetry) lib/telemetry.ex:76: Telemetry.execute/3
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:754: Ecto.Adapters.SQL.log/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (db_connection) lib/db_connection.ex:1303: DBConnection.log/5
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (db_connection) lib/db_connection.ex:1363: DBConnection.run_transaction/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/adapters/sql.ex:692: Ecto.Adapters.SQL.lock_for_migrations/4
Nov 20 09:41:38 1362f30f9ec7 staging_web.1.de5bs80ltl8wxnayh9dh1hqq3: (ecto_sql) lib/ecto/migrator.ex:317: Ecto.Migrator.lock_for_migrations/3
Here is my mix.lock
%{
"absinthe": {:hex, :absinthe, "1.4.13", "81eb2ff41f1b62cd6e992955f62c22c042d1079b7936c27f5f7c2c806b8fc436", [:mix], [{:dataloader, "~> 1.0.0", [hex: :dataloader, repo: "hexpm", optional: true]}, {:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm"},
"absinthe_plug": {:hex, :absinthe_plug, "1.4.6", "ac5d2d3d02acf52fda0f151b294017ab06e2ed1c6c15334e06aac82c94e36e08", [:mix], [{:absinthe, "~> 1.4.11", [hex: :absinthe, repo: "hexpm", optional: false]}, {:plug, "~> 1.3.2 or ~> 1.4", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"},
"artificery": {:hex, :artificery, "0.2.6", "f602909757263f7897130cbd006b0e40514a541b148d366ad65b89236b93497a", [:mix], [], "hexpm"},
"base64url": {:hex, :base64url, "0.0.1", "36a90125f5948e3afd7be97662a1504b934dd5dac78451ca6e9abf85a10286be", [:rebar], [], "hexpm"},
"bcrypt_elixir": {:hex, :bcrypt_elixir, "1.1.1", "6b5560e47a02196ce5f0ab3f1d8265db79a23868c137e973b27afef928ed8006", [:make, :mix], [{:elixir_make, "~> 0.4", [hex: :elixir_make, repo: "hexpm", optional: false]}], "hexpm"},
"bunt": {:hex, :bunt, "0.2.0", "951c6e801e8b1d2cbe58ebbd3e616a869061ddadcc4863d0a2182541acae9a38", [:mix], [], "hexpm"},
"certifi": {:hex, :certifi, "2.4.2", "75424ff0f3baaccfd34b1214184b6ef616d89e420b258bb0a5ea7d7bc628f7f0", [:rebar3], [{:parse_trans, "~>3.3", [hex: :parse_trans, repo: "hexpm", optional: false]}], "hexpm"},
"cloudex": {:hex, :cloudex, "1.2.1", "fd2b115c8519399f11c63d11c13ff98a88d381196c01228a206e0af12dee13dd", [:mix], [{:httpoison, "~> 1.3.0", [hex: :httpoison, repo: "hexpm", optional: false]}, {:poison, "~> 3.1.0", [hex: :poison, repo: "hexpm", optional: false]}, {:timex, "~> 3.4.2", [hex: :timex, repo: "hexpm", optional: false]}, {:tzdata, "~> 0.5.11", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm"},
"combine": {:hex, :combine, "0.10.0", "eff8224eeb56498a2af13011d142c5e7997a80c8f5b97c499f84c841032e429f", [:mix], [], "hexpm"},
"comeonin": {:hex, :comeonin, "4.1.1", "c7304fc29b45b897b34142a91122bc72757bc0c295e9e824999d5179ffc08416", [:mix], [{:argon2_elixir, "~> 1.2", [hex: :argon2_elixir, repo: "hexpm", optional: true]}, {:bcrypt_elixir, "~> 0.12.1 or ~> 1.0", [hex: :bcrypt_elixir, repo: "hexpm", optional: true]}, {:pbkdf2_elixir, "~> 0.12", [hex: :pbkdf2_elixir, repo: "hexpm", optional: true]}], "hexpm"},
"connection": {:hex, :connection, "1.0.4", "a1cae72211f0eef17705aaededacac3eb30e6625b04a6117c1b2db6ace7d5976", [:mix], [], "hexpm"},
"cowboy": {:hex, :cowboy, "2.6.0", "dc1ff5354c89e36a3e3ef8d10433396dcff0dcbb1d4223b58c64c2d51a6d88d9", [:rebar3], [{:cowlib, "~> 2.7.0", [hex: :cowlib, repo: "hexpm", optional: false]}, {:ranch, "~> 1.7.0", [hex: :ranch, repo: "hexpm", optional: false]}], "hexpm"},
"cowlib": {:hex, :cowlib, "2.7.0", "3ef16e77562f9855a2605900cedb15c1462d76fb1be6a32fc3ae91973ee543d2", [:rebar3], [], "hexpm"},
"credo": {:hex, :credo, "1.0.0", "aaa40fdd0543a0cf8080e8c5949d8c25f0a24e4fc8c1d83d06c388f5e5e0ea42", [:mix], [{:bunt, "~> 0.2.0", [hex: :bunt, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
"db_connection": {:hex, :db_connection, "2.0.2", "440c05518b0bdca0469dafaf45403597430448c1281def14ef9ccaa41833ea1e", [:mix], [{:connection, "~> 1.0.2", [hex: :connection, repo: "hexpm", optional: false]}], "hexpm"},
"decimal": {:hex, :decimal, "1.5.0", "b0433a36d0e2430e3d50291b1c65f53c37d56f83665b43d79963684865beab68", [:mix], [], "hexpm"},
"dialyxir": {:hex, :dialyxir, "0.5.1", "b331b091720fd93e878137add264bac4f644e1ddae07a70bf7062c7862c4b952", [:mix], [], "hexpm"},
"distillery": {:hex, :distillery, "2.0.12", "6e78fe042df82610ac3fa50bd7d2d8190ad287d120d3cd1682d83a44e8b34dfb", [:mix], [{:artificery, "~> 0.2", [hex: :artificery, repo: "hexpm", optional: false]}], "hexpm"},
"earmark": {:hex, :earmark, "1.2.6", "b6da42b3831458d3ecc57314dff3051b080b9b2be88c2e5aa41cd642a5b044ed", [:mix], [], "hexpm"},
"ecto": {:hex, :ecto, "3.0.2", "9d06ece60e2fc9b7593e74ede23747424ef32390016baca85cea3f4b84081c52", [:mix], [{:decimal, "~> 1.5", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:poison, "~> 2.2 or ~> 3.0", [hex: :poison, repo: "hexpm", optional: true]}], "hexpm"},
"ecto_enum": {:hex, :ecto_enum, "1.1.0", "d44fe2ce6e1c0e907e7c3b6456a69e0f1d662348d8b4e2a662ba312223d8ff62", [:mix], [{:ecto, ">= 2.0.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:mariaex, ">= 0.0.0", [hex: :mariaex, repo: "hexpm", optional: true]}, {:postgrex, ">= 0.0.0", [hex: :postgrex, repo: "hexpm", optional: true]}], "hexpm"},
"ecto_sql": {:hex, :ecto_sql, "3.0.1", "7f0e123a3b4d3b8ac67da6ba3014c53dbed63e614c5086f3e99d01cab5defe27", [:mix], [{:db_connection, "~> 2.0", [hex: :db_connection, repo: "hexpm", optional: false]}, {:ecto, "~> 3.0.2", [hex: :ecto, repo: "hexpm", optional: false]}, {:mariaex, "~> 0.9.1", [hex: :mariaex, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.14.0", [hex: :postgrex, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.2.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm"},
"eflame": {:hex, :eflame, "1.0.1", "0664d287e39eef3c413749254b3af5f4f8b00be71c1af67d325331c4890be0fc", [:mix], [], "hexpm"},
"elixir_make": {:hex, :elixir_make, "0.4.2", "332c649d08c18bc1ecc73b1befc68c647136de4f340b548844efc796405743bf", [:mix], [], "hexpm"},
"ex_doc": {:hex, :ex_doc, "0.19.1", "519bb9c19526ca51d326c060cb1778d4a9056b190086a8c6c115828eaccea6cf", [:mix], [{:earmark, "~> 1.1", [hex: :earmark, repo: "hexpm", optional: false]}, {:makeup_elixir, "~> 0.7", [hex: :makeup_elixir, repo: "hexpm", optional: false]}], "hexpm"},
"excoveralls": {:hex, :excoveralls, "0.10.3", "b090a3fbcb3cfa136f0427d038c92a9051f840953ec11b40ee74d9d4eac04d1e", [:mix], [{:hackney, "~> 1.13", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm"},
"exprintf": {:hex, :exprintf, "0.2.1", "b7e895dfb00520cfb7fc1671303b63b37dc3897c59be7cbf1ae62f766a8a0314", [:mix], [], "hexpm"},
"exprof": {:hex, :exprof, "0.2.3", "8d4d657d73fc0c9ef1e30b2f9207b26ccbd2aec2baf1ca43f0b6d244c841c9f8", [:mix], [{:exprintf, "~> 0.2", [hex: :exprintf, repo: "hexpm", optional: false]}], "hexpm"},
"file_system": {:hex, :file_system, "0.2.6", "fd4dc3af89b9ab1dc8ccbcc214a0e60c41f34be251d9307920748a14bf41f1d3", [:mix], [], "hexpm"},
"gettext": {:hex, :gettext, "0.16.1", "e2130b25eebcbe02bb343b119a07ae2c7e28bd4b146c4a154da2ffb2b3507af2", [:mix], [], "hexpm"},
"guardian": {:hex, :guardian, "1.1.1", "be14c4007eaf05268251ae114030cb7237ed9a9631c260022f020164ff4ed733", [:mix], [{:jose, "~> 1.8", [hex: :jose, repo: "hexpm", optional: false]}, {:phoenix, "~> 1.0 or ~> 1.2 or ~> 1.3", [hex: :phoenix, repo: "hexpm", optional: true]}, {:plug, "~> 1.3.3 or ~> 1.4", [hex: :plug, repo: "hexpm", optional: true]}, {:poison, "~> 2.2 or ~> 3.0", [hex: :poison, repo: "hexpm", optional: false]}], "hexpm"},
"hackney": {:hex, :hackney, "1.14.3", "b5f6f5dcc4f1fba340762738759209e21914516df6be440d85772542d4a5e412", [:rebar3], [{:certifi, "2.4.2", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "6.0.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "1.0.1", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "1.0.2", [hex: :mimerl, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "1.1.4", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}], "hexpm"},
"httpoison": {:hex, :httpoison, "1.3.1", "7ac607311f5f706b44e8b3fab736d0737f2f62a31910ccd9afe7227b43edb7f0", [:mix], [{:hackney, "~> 1.8", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"},
"idna": {:hex, :idna, "6.0.0", "689c46cbcdf3524c44d5f3dde8001f364cd7608a99556d8fbd8239a5798d4c10", [:rebar3], [{:unicode_util_compat, "0.4.1", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm"},
"jason": {:hex, :jason, "1.1.2", "b03dedea67a99223a2eaf9f1264ce37154564de899fd3d8b9a21b1a6fd64afe7", [:mix], [{:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm"},
"jose": {:hex, :jose, "1.8.4", "7946d1e5c03a76ac9ef42a6e6a20001d35987afd68c2107bcd8f01a84e75aa73", [:mix, :rebar3], [{:base64url, "~> 0.0.1", [hex: :base64url, repo: "hexpm", optional: false]}], "hexpm"},
"makeup": {:hex, :makeup, "0.5.5", "9e08dfc45280c5684d771ad58159f718a7b5788596099bdfb0284597d368a882", [:mix], [{:nimble_parsec, "~> 0.4", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm"},
"makeup_elixir": {:hex, :makeup_elixir, "0.10.0", "0f09c2ddf352887a956d84f8f7e702111122ca32fbbc84c2f0569b8b65cbf7fa", [:mix], [{:makeup, "~> 0.5.5", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm"},
"metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [:rebar3], [], "hexpm"},
"mime": {:hex, :mime, "1.3.0", "5e8d45a39e95c650900d03f897fbf99ae04f60ab1daa4a34c7a20a5151b7a5fe", [:mix], [], "hexpm"},
"mimerl": {:hex, :mimerl, "1.0.2", "993f9b0e084083405ed8252b99460c4f0563e41729ab42d9074fd5e52439be88", [:rebar3], [], "hexpm"},
"nimble_parsec": {:hex, :nimble_parsec, "0.4.0", "ee261bb53214943679422be70f1658fff573c5d0b0a1ecd0f18738944f818efe", [:mix], [], "hexpm"},
"number": {:hex, :number, "0.5.7", "da1ee0434f3c7b84360d33b335d0ee2f86f247f45fbe2e7636e6f55680b464dd", [:mix], [{:decimal, "~> 1.2", [hex: :decimal, repo: "hexpm", optional: false]}], "hexpm"},
"observer_cli": {:hex, :observer_cli, "1.4.1", "8c829bb28e3f316a361216314405fe296d8dd2fd7409dfa53fa4ac1cff3797a0", [:rebar3], [{:recon, "2.3.6", [hex: :recon, repo: "hexpm", optional: false]}], "hexpm"},
"parent": {:hex, :parent, "0.5.1", "fed79e79402a1f91bffb117e0ef01fd802345dde79fb70df317f8857f2679237", [:mix], [], "hexpm"},
"parse_trans": {:hex, :parse_trans, "3.3.0", "09765507a3c7590a784615cfd421d101aec25098d50b89d7aa1d66646bc571c1", [:rebar3], [], "hexpm"},
"phoenix": {:hex, :phoenix, "1.4.0", "56fe9a809e0e735f3e3b9b31c1b749d4b436e466d8da627b8d82f90eaae714d2", [:mix], [{:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}, {:phoenix_pubsub, "~> 1.1", [hex: :phoenix_pubsub, repo: "hexpm", optional: false]}, {:plug, "~> 1.7", [hex: :plug, repo: "hexpm", optional: false]}, {:plug_cowboy, "~> 1.0 or ~> 2.0", [hex: :plug_cowboy, repo: "hexpm", optional: true]}], "hexpm"},
"phoenix_ecto": {:hex, :phoenix_ecto, "4.0.0", "c43117a136e7399ea04ecaac73f8f23ee0ffe3e07acfcb8062fe5f4c9f0f6531", [:mix], [{:ecto, "~> 3.0", [hex: :ecto, repo: "hexpm", optional: false]}, {:phoenix_html, "~> 2.9", [hex: :phoenix_html, repo: "hexpm", optional: true]}, {:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"},
"phoenix_html": {:hex, :phoenix_html, "2.12.0", "1fb3c2e48b4b66d75564d8d63df6d53655469216d6b553e7e14ced2b46f97622", [:mix], [{:plug, "~> 1.5", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"},
"phoenix_live_reload": {:hex, :phoenix_live_reload, "1.2.0", "3bb31a9fbd40ffe8652e60c8660dffd72dd231efcdf49b744fb75b9ef7db5dd2", [:mix], [{:file_system, "~> 0.2.1 or ~> 0.3", [hex: :file_system, repo: "hexpm", optional: false]}, {:phoenix, "~> 1.4", [hex: :phoenix, repo: "hexpm", optional: false]}], "hexpm"},
"phoenix_pubsub": {:hex, :phoenix_pubsub, "1.1.1", "6668d787e602981f24f17a5fbb69cc98f8ab085114ebfac6cc36e10a90c8e93c", [:mix], [], "hexpm"},
"plug": {:hex, :plug, "1.7.1", "8516d565fb84a6a8b2ca722e74e2cd25ca0fc9d64f364ec9dbec09d33eb78ccd", [:mix], [{:mime, "~> 1.0", [hex: :mime, repo: "hexpm", optional: false]}, {:plug_crypto, "~> 1.0", [hex: :plug_crypto, repo: "hexpm", optional: false]}], "hexpm"},
"plug_cowboy": {:hex, :plug_cowboy, "2.0.0", "ab0c92728f2ba43c544cce85f0f220d8d30fc0c90eaa1e6203683ab039655062", [:mix], [{:cowboy, "~> 2.5", [hex: :cowboy, repo: "hexpm", optional: false]}, {:plug, "~> 1.7", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"},
"plug_crypto": {:hex, :plug_crypto, "1.0.0", "18e49317d3fa343f24620ed22795ec29d4a5e602d52d1513ccea0b07d8ea7d4d", [:mix], [], "hexpm"},
"poison": {:hex, :poison, "3.1.0", "d9eb636610e096f86f25d9a46f35a9facac35609a7591b3be3326e99a0484665", [:mix], [], "hexpm"},
"postgrex": {:hex, :postgrex, "0.14.0", "f3d6ffea1ca8a156e0633900a5338a3d17b00435227726baed8982718232b694", [:mix], [{:connection, "~> 1.0", [hex: :connection, repo: "hexpm", optional: false]}, {:db_connection, "~> 2.0", [hex: :db_connection, repo: "hexpm", optional: false]}, {:decimal, "~> 1.0", [hex: :decimal, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm"},
"ranch": {:hex, :ranch, "1.7.0", "9583f47160ca62af7f8d5db11454068eaa32b56eeadf984d4f46e61a076df5f2", [:rebar3], [], "hexpm"},
"recon": {:hex, :recon, "2.3.6", "2bcad0cf621fb277cabbb6413159cd3aa30265c2dee42c968697988b30108604", [:rebar3], [], "hexpm"},
"sentry": {:hex, :sentry, "7.0.3", "093fa4b6937760afb9a5fcb0e4a9092a305b6c0ff26a710e977614b201feab75", [:mix], [{:hackney, "~> 1.8 or 1.6.5", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: true]}, {:phoenix, "~> 1.3", [hex: :phoenix, repo: "hexpm", optional: true]}, {:plug, "~> 1.6", [hex: :plug, repo: "hexpm", optional: true]}, {:plug_cowboy, "~> 1.0 or ~> 2.0", [hex: :plug_cowboy, repo: "hexpm", optional: true]}], "hexpm"},
"site_encrypt": {:git, "https://github.com/sasa1977/site_encrypt.git", "c76576b8948b77d74cb1808b5eedc75c983a545c", []},
"ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.4", "f0eafff810d2041e93f915ef59899c923f4568f4585904d010387ed74988e77b", [:make, :mix, :rebar3], [], "hexpm"},
"swoosh": {:hex, :swoosh, "0.20.0", "9a6c13822c9815993c03b6f8fccc370fcffb3c158d9754f67b1fdee6b3a5d928", [:mix], [{:cowboy, "~> 1.0.1 or ~> 1.1 or ~> 2.4", [hex: :cowboy, repo: "hexpm", optional: true]}, {:gen_smtp, "~> 0.12", [hex: :gen_smtp, repo: "hexpm", optional: true]}, {:hackney, "~> 1.9", [hex: :hackney, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mime, "~> 1.1", [hex: :mime, repo: "hexpm", optional: false]}, {:plug, "~> 1.4", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm"},
"telemetry": {:hex, :telemetry, "0.2.0", "5b40caa3efe4deb30fb12d7cd8ed4f556f6d6bd15c374c2366772161311ce377", [:mix], [], "hexpm"},
"timex": {:hex, :timex, "3.4.2", "d74649c93ad0e12ce5b17cf5e11fbd1fb1b24a3d114643e86dba194b64439547", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.10", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 0.1.8 or ~> 0.5", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm"},
"turbolinks": {:hex, :turbolinks, "1.0.1", "ee48ddc70884c738ff5a58a004d2565276cf5841f3ef43c22afd884d0404603e", [:mix], [{:phoenix, "~> 1.0", [hex: :phoenix, repo: "hexpm", optional: false]}, {:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: false]}], "hexpm"},
"tzdata": {:hex, :tzdata, "0.5.19", "7962a3997bf06303b7d1772988ede22260f3dae1bf897408ebdac2b4435f4e6a", [:mix], [{:hackney, "~> 1.0", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm"},
"unicode_util_compat": {:hex, :unicode_util_compat, "0.4.1", "d869e4c68901dd9531385bb0c8c40444ebf624e60b6962d95952775cac5e90cd", [:rebar3], [], "hexpm"},
"x509": {:hex, :x509, "0.3.0", "c6f3db66960c6e4f424d1e6cca5c7d730e0a577af8dc115a613f4560ce1df6d3", [:mix], [], "hexpm"},
}
let me know if you need anything more. I you have ideas on where to search.
I am using postgres 11.1, that might be the cause of the error
We ran into a scenario recently where we had downtime for the following reason:
ALTER TABLE
commands)This caused all queries to be queued behind the migration, which was waiting to get a lock.
We'd like to automatically set a lock timeout on migrations to avoid this problem in future. The command is:
SET lock_timeout TO '5s'
We could add this manually inside every migration we create but that is tedious and prone to people forgetting. Is there a way to use ecto to automatically execute this SQL at the start of every DDL transaction?
ecto_sql 3.1.1
and postgrex 0.14.0
I'm using ecto 3.1.6
, ecto_sql 3.1.1
and postgrex 0.14.0
with PostgreSQL 11.2.
When I add an IO.inspect
here, a particular query shows :query_time
of 2587 μs. But the telemetry measurements reported to my handler show query_time: 2587000
. So it appears to me that the times are being reported in nanoseconds. I don't see this documented anywhere.
I would like to document somewhere that the measurements
are in nanoseconds, as it took me some puzzling to figure this out. Can someone advise me on where that documentation should go? So far I don't see where these numbers are actually generated.
My Repo config uses the :after_connect
option to set search_path
for Postgres:
config :my_app, MyApp.Repo,
after_connect: {Postgrex, :query!, ["SET search_path=\"$user\",public,shared_extensions", []]}
This was working fine previously on Ecto v2.2.10. However, since upgrading to Ecto 3.0 I'm starting to see this error in console:
[error] Postgrex.Protocol (#PID<0.512.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.685.0> timed out because it checked out the connection for longer than 15000ms
When I removed the after_connect
option from config then the error no longer shows up.
No error in console
hexpm[wm-ecto-3]% mix ecto.rollback
[debug] QUERY OK source="schema_migrations" db=1.1ms
SELECT s0."version"::bigint FROM "schema_migrations" AS s0 FOR UPDATE []
[info] == Running Hexpm.Repo.Migrations.AddServiceToUsers.down/0 forward
[info] execute "DELETE FROM users WHERE username = 'hexdocs'"
[info] execute "ALTER TABLE users ALTER password SET NOT NULL"
[info] alter table users
[info] == Migrated in 0.0s
** (ErlangError) Erlang error: :timeout_value
(stdlib) proc_lib.erl:933: :proc_lib.stop/3
(elixir) lib/gen_server.ex:917: GenServer.stop/3
(ecto_sql) lib/mix/tasks/ecto.rollback.ex:93: anonymous fn/4 in Mix.Tasks.Ecto.Rollback.run/2
(elixir) lib/enum.ex:768: Enum."-each/2-lists^foreach/1-0-"/2
(elixir) lib/enum.ex:768: Enum.each/2
(mix) lib/mix/task.ex:319: Mix.Task.run_task/3
(mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2
(elixir) lib/code.ex:767: Code.require_file/2
The rollback actually succeeds but there shouldn't be an error.
No error.
The following test case in a blank Phoenix project will cause the inner task to hang indefinitely (until Ecto timeout):
test "should allow creating an item from a task within a transaction", %{conn: _conn} do
Repo.transaction(fn ->
Task.async(fn ->
%Item{}
|> Repo.insert!()
end)
|> Task.await()
end)
end
The error is:
1) test failing test case (EctoFailCaseWeb.PageControllerTest)
test/ecto_fail_case_web/controllers/page_controller_test.exs:4
** (EXIT from #PID<0.363.0>) an exception was raised:
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 978ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information
(ecto_sql) lib/ecto/adapters/sql.ex:621: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto) lib/ecto/repo/schema.ex:649: Ecto.Repo.Schema.apply/4
(ecto) lib/ecto/repo/schema.ex:262: anonymous fn/15 in Ecto.Repo.Schema.do_insert/4
(ecto) lib/ecto/repo/schema.ex:164: Ecto.Repo.Schema.insert!/4
(elixir) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
(elixir) lib/task/supervised.ex:35: Task.Supervised.reply/5
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Switching the Ecto sandbox to shared mode and toggling async to false
doesn't seem to fix the issue. However, the same code in a non-test environment (e.g. IEx running in dev mode) succeeds without issue.
Full example project (essentially just a mix phx.new
with the above test case and a single schema) at: https://github.com/dkulchenko/ecto-fail-case
Present in master.
This means that we need to ship the psql tool to where we build our database, which is less than ideal.
Ecto.Adapters.Postgres.structure_load delegates to psql
Ecto.Adapters.Postgres.structure_load performs the work without reliance on the psql
tool.
Doing avg(d.production)
by default returns a %Decimal{}
, to have it return a float I usually do type(avg(d.production), :float)
or use fragment
, both work fine.
However, I've tried multiple ways of having the same behavior while using window functions:
over(type(avg(d.production), :float), :interval)
throws unknown window function type/2. See Ecto.Query.WindowAPI for all available functions
type(over(avg(d.production), :interval), :float)
throws type(over(avg(d.production()), :interval), :float) is not a valid query expression.
The select returns that value as float
I am working on migrating a Postgres back-end from Ecto 2.2 to Ecto 3.0.5 (To coincide with the 1.3 Triplex release). This is for these indices:
From my migration file:
create(unique_index(:account, ["(lower(email))"]))
create(unique_index(:account, ["(lower(username))"]))
In Ecto 2.2, the index names in postgres are created as:
account_lower_username__index
account_lower_email__index
But in Ecto 3.0.5, the index names in postgres are created as:
account__lower_username_index
account__lower_email_index
Please note that the double underscore in the index names moved between the two versions. Because this is a custom index, I must specify the constraint name in my changeset to handle any violations correctly:
|> unique_constraint(:username, name: :account_lower_username__index)
|> unique_constraint(:email, name: :account_lower_email__index)
Because this is a multi-tenant application, I will have Ecto 2.2 created indexes and Ecto 3.0.5 created indexes being used by the same changeset.
My workaround is to edit my existing migration documents to include the legacy name when creating those indices. I use:
create(unique_index(:account, ["(lower(email))"], name: :account_lower_email__index))
create(unique_index(:account, ["(lower(username))"], name: :account_lower_username__index))
And the changesets for current tenant schemas continue to work for new tenants.
Other options would have been to either drop and recreate indices in a migration to get the new naming convention (specifying the old name due to the naming convention change), or to add additional unique constraints to the changesets for all possible index names moving forward.
Index names for a custom index field changed between versions. Due to multi-tenancy, the unique_constraint in the changeset no longer matches the index names between existing tenants and new tenants.
The behavior should be consistent, or noted in the changelog.
For monitoring solutions, it'd be nice to have the Repo
module name in the Telemetry metadata map.
This would enable us to report metrics with the proper namespace in the condition that there are multiple Repo
s in an application. Without this we couldn't tell the difference between tables with the same name in different databases...
Thanks!
Running the following as part of a phoenix conn test using MyappWeb.ConnCase causes the test to pass, but the exception is printed to the terminal after the test has completed. I believe it could be something to do with the COMMIT
inside the stored procedure.
stored procedure:
create table db_test
(
id int auto_increment
primary key,
`key` varchar(32) default 'NULL' null,
val text default 'NULL' null
);
create procedure db_test(IN inKey varchar(32), IN inVal text)
BEGIN
START TRANSACTION;
INSERT INTO db_test (`key`, val) VALUES (inKey, inVal);
COMMIT;
END;
controller:
def test_db(conn, _params) do
{:ok, v} = Repo.query("CALL test.db_test('3', 'maybe')")
json(conn, v.rows)
end
exception:
10:02:07.200 [error] GenServer #PID<0.456.0> terminating
** (CaseClauseError) no case clause matching: {:idle, %Mariaex.Protocol{binary_as: :field_type_var_string, buffer: "", cache: #Reference<0.2931625945.744620033.246778>, catch_eof: false, connection_id: #PID<0.418.0>, cursors: %{}, datetime: :structs, deprecated_eof: true, json_library: Poison, lru_cache:{100, #Reference<0.2931625945.744620033.246779>}, opts: [socket_options: [], sock_type: :tcp, cache_size: 100, port: 3306, datetime: :structs, repo: Myapp.Repo, telemetry_prefix: [:myapp, :repo], otp_app: :myapp, timeout: 15000, pool_timeout: 5000, pool_size: 10, username: "xxxx", password: "xxxx", database: "myapp_test", hostname: "localhost"], seqnum: 0, sock: {Mariaex.Connection.Tcp, #Port<0.82>}, ssl_conn_state: :not_used, state: :running, state_data: nil, timeout: 15000, transaction_status: :idle}}
(ecto_sql) lib/ecto/adapters/sql/sandbox.ex:514: Ecto.Adapters.SQL.Sandbox.pre_checkin/4
(db_connection) lib/db_connection/ownership/proxy.ex:210: DBConnection.Ownership.Proxy.pool_done/5
(stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
(stdlib) gen_server.erl:711: :gen_server.handle_msg/6
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:DOWN, #Reference<0.2931625945.744488961.246901>, :process, #PID<0.455.0>, :shutdown}
10:02:07.200 [error] Mariaex.Protocol (#PID<0.418.0>) disconnected: ** (DBConnection.ConnectionError)client #PID<0.456.0> exited
The test will pass and no error will be printed to the console. As happened with Ecto 2 / Phoenix 1.3.4.
Rather than warn or hang, it would be ideal if migrations using the :concurrent
option would automatically turn off the migration table lock for that migration only.
I agree that this new lock introduced in 3.1 is a great default to have for most codebases and migrations. But when you need to run something concurrently, it seems unwise to turn this lock off for all migrations, when you only really need it off for a single migration.
More context:
Master branches of ecto and ecto_sql.
The link to the ecto_sql documentation (https://hexdocs.pm/ecto_sql) is broken.
Documentation for ecto_sql should be displayed. It currently does not because ecto_sql is not a published package on hex. This also breaks the Ecto.Adapters.SQL link in the ecto README.
Elixir 1.8.1
Ecto SQL 3.0.5
Ecto 3.0.7
Dialyzer complains that type Ecto.Adapter.on_conflict
does not exist.
See the type being used here.
The type should be fixed.
We started the discussion about making the ecto.load
smarter and skip loaded databases. We'll need a function that can check if a table exists.
In Postgres we can check like this:
SELECT to_regclass('schema_name.table_name')
In MySQL we can check like this:
SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'yourdb'
AND table_name = 'testtable'
LIMIT 1;
We thought that we can add this function to Ecto.Adapters.SQL
, however, I wonder how we can do it?
Today the Adapters.SQL
has these behaviours:
Ecto.Adapter
Ecto.Adapter.Migration
Ecto.Adapter.Queryable
Ecto.Adapter.Schema
Ecto.Adapter.Transaction
Wich one should we add the function table_existence_check_query
?
We have also the
Ecto.Adapter.Migration
Ecto.Adapter.Structure
I'm a little lost where I should put the callback.
The discussion started here: https://groups.google.com/forum/#!topic/elixir-ecto/8aw2DL_c9Yc
The idea of the feature came from here: #108
I'm getting this error:
15:41:37.483 [debug] QUERY ERROR db=1.1ms queue=1.9ms
INSERT INTO "packages" ("meta","name") VALUES ($1,$2) RETURNING "id" [%{}, "foo"]
15:41:37.484 [error] Postgrex.Protocol (#PID<0.294.0>) disconnected: ** (Postgrex.Error) ERROR 3B001 (invalid_savepoint_specification) no such savepoint
1) test greets the world (ExampleMasterTest)
test/example_master_test.exs:10
** (Postgrex.Error) ERROR 3B001 (invalid_savepoint_specification) no such savepoint
code: Repo.insert!(Package.changeset(%Package{}, %{name: "foo", meta: %{}}))
stacktrace:
(ecto_sql) lib/ecto/adapters/sql.ex:584: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto) lib/ecto/repo/schema.ex:600: Ecto.Repo.Schema.apply/4
(ecto) lib/ecto/repo/schema.ex:226: anonymous fn/15 in Ecto.Repo.Schema.do_insert/3
(ecto) lib/ecto/repo/schema.ex:128: Ecto.Repo.Schema.insert!/3
test/example_master_test.exs:11: (test)
Turns out I forgot to set config :postgrex, json_library: ...
which is needed for map field. When I set it, it's fine. It's possible I've somehow misconfigured the repo too.
Repo to reproduce: https://github.com/wojtekmach/example_master
Perhaps a compile-time error from Ecto.Adapters.Postgres that json library is missing.
Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe] [dtrace]
Elixir 1.7.4 (compiled with Erlang/OTP 21)
With multiple pending migrations, run mix ecto.migrate -n 1
. Rather than only running a single migration, it runs all pending migrations.
The -n 1
flag should work and cause Ecto to only migrate up by a single version, rather than running all pending migrations.
In an older application that has a test_helper.exs
file like this one (Phoenix 1.1.x generator):
ExUnit.start
Mix.Task.run "ecto.create", ["--quiet"]
Mix.Task.run "ecto.migrate", ["--quiet"]
Ecto.Adapters.SQL.Sandbox.mode(MyApp.Repo, :manual)
The first time tests are run after a new migration is added, the run fails with this stacktrace:
$ mix test
Generated myapp app
** (ArgumentError) argument error
(stdlib) :ets.lookup_element(Ecto.Repo.Registry, #PID<0.1011.0>, 3)
(ecto) lib/ecto/repo/registry.ex:18: Ecto.Repo.Registry.lookup/1
(ecto) lib/ecto/adapter.ex:127: Ecto.Adapter.lookup_meta/1
(ecto_sql) lib/ecto/adapters/sql/sandbox.ex:486: Ecto.Adapters.SQL.Sandbox.lookup_meta!/1
(ecto_sql) lib/ecto/adapters/sql/sandbox.ex:387: Ecto.Adapters.SQL.Sandbox.mode/2
Ideally the repo should still be up and usable after the Ecto migration runs, or there could be a more helpful error message. Since the migration is silent, this showed up as flaky tests that were quite hard to track down.
Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Elixir 1.8.0 (compiled with Erlang/OTP 21)
* ecto 3.0.7 (Hex package) (mix)
* ecto_sql 3.0.5 (Hex package) (mix)
postgrex 0.14.1 (Hex package) (mix)
but doesn't really matterDarwin 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64
but doesn't matter as wellIn umbrella (haven't tested that in single project repository) when I define my application in mix.exs
as:
def application do
[
mod: {MyApp.Application, []},
env: [
ecto_repos: [MyApp.Repo]
]
]
end
It states that it cannot find :ecto_repos
configuration option in defined applications:
warning: could not find Ecto repos in any of the apps: [:my_app, :my_app_web].
You can avoid this warning by passing the -r flag or by setting the
repositories managed by those applications in your config/config.exs:
config :my_app, ecto_repos: [...]
It should load application environment as well as configuration files ("normal" precedence rules should apply).
Create a new project and add two new migrations:
$ mix phx.new hello
$ cd hello
$ mix phx.gen.schema User users
$ mix phx.gen.schema Account accounts
Edit the second migration, priv/repo/migrations/*_create_accounts.exs
, and change "create table" to "create_table":
defmodule Hello.Repo.Migrations.CreateAccounts do
use Ecto.Migration
def change do
create_table(:accounts) do
timestamps()
end
end
end
Then create the database and run the migrations:
$ mix ecto.create
$ mix ecto.migrate
You'll get a CompileError, similar to this:
[info] == Running 20190111231353 Hello.Repo.Migrations.CreateUsers.change/0 forward
[info] create table users
[info] create index users_email_index
[info] == Migrated 20190111231353 in 0.0s
** (CompileError) priv/repo/migrations/20190111231403_create_accounts.exs:12: undefined function create_table/2
(stdlib) lists.erl:1338: :lists.foreach/2
(stdlib) erl_eval.erl:680: :erl_eval.do_apply/6
(elixir) lib/code.ex:715: Code.load_file/2
(ecto_sql) lib/ecto/migrator.ex:476: Ecto.Migrator.extract_module/2
(ecto_sql) lib/ecto/migrator.ex:433: anonymous fn/5 in Ecto.Migrator.do_migrate/4
(elixir) lib/enum.ex:3281: Enumerable.List.reduce/3
(elixir) lib/enum.ex:1968: Enum.reduce_while/3
(ecto_sql) lib/ecto/migrator.ex:427: Ecto.Migrator.migrate/4
(ecto_sql) lib/ecto/adapters/sql.ex:814: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection) lib/db_connection.ex:1349: DBConnection.run_transaction/4
(ecto_sql) lib/ecto/adapters/sql.ex:723: Ecto.Adapters.SQL.lock_for_migrations/5
(ecto_sql) lib/ecto/migrator.ex:318: Ecto.Migrator.lock_for_migrations/3
(ecto_sql) lib/mix/tasks/ecto.migrate.ex:110: anonymous fn/4 in Mix.Tasks.Ecto.Migrate.run/2
If you check the database, you'll see the users
table was created, but schema_migrations
has no rows in it. If you fix the error and run the migration again, it'll fail because users
already exists.
$ mix ecto.migrate
[info] == Running 20190111231353 Hello.Repo.Migrations.CreateUsers.change/0 forward
[info] create table users
** (Postgrex.Error) ERROR 42P07 (duplicate_table) relation "users" already exists
(ecto_sql) lib/ecto/adapters/sql.ex:620: Ecto.Adapters.SQL.raise_sql_call_error/1
(elixir) lib/enum.ex:1314: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql) lib/ecto/adapters/sql.ex:707: Ecto.Adapters.SQL.execute_ddl/4
(ecto_sql) lib/ecto/migration/runner.ex:301: Ecto.Migration.Runner.log_and_execute_ddl/3
(ecto_sql) lib/ecto/migration/runner.ex:110: anonymous fn/2 in Ecto.Migration.Runner.flush/0
(elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto_sql) lib/ecto/migration/runner.ex:108: Ecto.Migration.Runner.flush/0
(stdlib) timer.erl:166: :timer.tc/1
(ecto_sql) lib/ecto/migration/runner.ex:26: Ecto.Migration.Runner.run/7
(ecto_sql) lib/ecto/migrator.ex:211: Ecto.Migrator.attempt/7
(ecto_sql) lib/ecto/migrator.ex:113: anonymous fn/4 in Ecto.Migrator.do_up/4
(ecto_sql) lib/ecto/migrator.ex:193: anonymous fn/3 in Ecto.Migrator.run_maybe_in_transaction/5
(ecto_sql) lib/ecto/adapters/sql.ex:814: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection) lib/db_connection.ex:1349: DBConnection.run_transaction/4
(ecto_sql) lib/ecto/migrator.ex:192: Ecto.Migrator.run_maybe_in_transaction/5
(elixir) lib/task/supervised.ex:89: Task.Supervised.do_apply/2
(elixir) lib/task/supervised.ex:38: Task.Supervised.reply/5
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
I would expect schema_migrations
to be updated after the first migration but prior to the second migration being loaded and failing to compile.
(Per IRC discussion monologue)
Given modules:
defmodule Post do
@schema_prefix "public"
schema "posts" do
# ...
has_many :comments, Comment
end
end
defmodule Comment do
@schema_prefix "child"
schema "comments" do
# ...
belongs_to :post, Post
end
end
defmodule Repo do
# ...
end
config :app, Repo,
after_connect: {Postgrex, :query!, ["SET search_path TO child", []]}
Given call:
Comment |> preload(:posts) |> Repo.all()
I would expect that the prefix of the particular schema would override the prefix set on the repo. Actually, the prefixes should be in the following order:
repo.prefix || schema.prefix || query.prefix || nil
Running ecto.migrate
runs all migrations and finishes...with a crash.
I have a alias setup to run migrations for a few repos. But mix crashes after the first repo is migrated.
ie. mix ecto.migrate --repo A --repo B
will migrate repo A
but exit before migrating B
.
It seems that the offender is this. I can comment that out and it works as expected.
My team and I would like to be able to migrate all repos without extra typing. :)
Why was this function removed in a minor version?
https://github.com/elixir-ecto/ecto_sql/blob/v3.1.1/lib/mix/ecto_sql.ex#L8
I'm working on MSSQL adapter and having trouble to understand how to disable default "FOR UPDATE" :migration_lock value without implementing my custom Ecto.Adapter.SQL.lock_for_migrations/4
function in adapter.
The meta I'm getting looks like this
%{
cache: #Reference<0.224688069.2605580292.202322>,
opts: [
timeout: 15000,
pool_timeout: 5000,
pool_size: 5,
pool: DBConnection.Ownership
],
pid: #PID<0.290.0>,
sql: Ecto.Adapters.MsSql.Connection,
telemetry: {:debug, [], [:ecto, :integration, :test_repo, :query]}
}
Where TestRepo is configured like this
Application.put_env(
:ecto_sql,
TestRepo,
migration_lock: nil,
hostname: System.get_env("SQL_HOSTNAME"),
username: System.get_env("SQL_USERNAME"),
password: System.get_env("SQL_PASSWORD"),
database: "ecto_test",
pool: Ecto.Adapters.SQL.Sandbox,
pool_size: 5,
set_allow_snapshot_isolation: :on,
filter_null_on_unique_indexes: true
)
ecto_sql/lib/ecto/adapters/sql/sandbox.ex
Lines 212 to 232 in b9dfe39
we drop :ownership_timeout
option (ecto_sql/lib/ecto/adapters/sql.ex:450)
ecto_sql/lib/ecto/adapters/sql.ex
Lines 427 to 453 in b9dfe39
then when Ecto.Adapters.SQL.checkout(DB.Repo)
:ownership_timeout
options for my repo config in config/test.exs
is not available.
ecto_sql/lib/ecto/adapters/sql/sandbox.ex
Lines 485 to 486 in b9dfe39
don't clean up option in ecto_sql/lib/ecto/adapters/sql.ex
The function Ecto.Adapters.Postgres.Connection.execute/4
is not handling the case where Postgrex.execute/4
returns {: ok, Postgrex.Result.t}
execute/4 specification: execute(conn, Postgrex.Query.t, list, Keyword.t) :: {:ok, Postgrex.Result.t} | {:error, Postgrex.Error.t}
Add condition for {:ok, Postgrex.Result.t}
Example:
{:ok _} = ok ->
ok
Can such cases occur?
{:ok, %{ref: ^ref}, result} ->
{:ok, result}
{:ok, _, _} = ok ->
ok
In the Postgrex.execute/4
@spec
do not have these cases
We recently moved to ecto 3, and have suddenly had issues with concurrent index creation. The migration will hang indefinitely, and if you inspect the index in psql it is marked INVALID. This occurs even with non-unique indexes, which is unusual because it usually happens when a constraint is violated during index create. My guess is something is racing in the new multi-process migration system but couldn't piece it together.
Another possible wrinkle is we actually run our migration through distillery custom commands as seen here: https://github.com/bitwalker/distillery/blob/master/docs/guides/running_migrations.md. I tried fiddling with the pool sizes and diffing that code against the actual mix ecto.migrate task but nothing popped out.
Ecto migrate for concurrent indexes hangs and corrupts the index
Ecto can create indexes concurrently
Erlang/OTP 21 [erts-10.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]
Elixir 1.8.1 (compiled with Erlang/OTP 20)
* ecto (Hex package) (mix)
locked at 3.0.1 (ecto) a26605ee
* ecto_sql (Hex package) (mix)
locked at 3.0.0 (ecto_sql) 8d188337
* postgrex (Hex package) (mix)
locked at 0.14.0 (postgrex) f3d6ffea
So, we are trying to support ecto 3 on triplex (PR ateliware/triplex#61), and we stumbled in a problem: we cannot run a CREATE SCHEMA "a"
and the migrations for the prefix "a"
inside the same transaction, because now the migrations are ran inside a Task.async
, so the db connection will not be the same and then we get a ** (Postgrex.Error) ERROR 3F000 (invalid_schema_name) schema "a" does not exist
for the migrations.
A sample code would be:
Repo.transaction fn ->
Ecto.Adapters.SQL.query!(Repo, "CREATE SCHEMA \"a\"", [])
Ecto.Migrator.run(Repo, :up, all: true, prefix: "a")
end
And the error you will get on ecto 3 is:
** (Postgrex.Error) ERROR 3F000 (invalid_schema_name) schema "a" does not exist
(ecto_sql) lib/ecto/adapters/sql.ex:590: Ecto.Adapters.SQL.raise_sql_call_error/1
(elixir) lib/enum.ex:1327: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql) lib/ecto/adapters/sql.ex:677: Ecto.Adapters.SQL.execute_ddl/4
(ecto_sql) lib/ecto/migration/runner.ex:272: Ecto.Migration.Runner.log_and_execute_ddl/3
(ecto_sql) lib/ecto/migration/runner.ex:105: anonymous fn/2 in Ecto.Migration.Runner.flush/0
(elixir) lib/enum.ex:1940: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto_sql) lib/ecto/migration/runner.ex:103: Ecto.Migration.Runner.flush/0
(stdlib) timer.erl:181: :timer.tc/2
So with ecto 2 the sample code runs smoothly, and migrates everything, because all the migrations are run inside of the same process of the transaction.
My suggestion for a solution would be an async
option for Ecto.Migrator.up/4
, Ecto.Migrator.down/4
and Ecto.Migrator.run/3,4
, that could be true
by default, but if false
, it will not run the migrations on a Task.async
.
I can work on that if you want.
mix ecto.migrations fails with this error on 3.0.0-rc1:
** (ErlangError) Erlang error: :timeout_value
(stdlib) proc_lib.erl:933: :proc_lib.stop/3
(elixir) lib/gen_server.ex:880: GenServer.stop/3
(ecto_sql) lib/mix/tasks/ecto.migrations.ex:45: anonymous fn/3 in Mix.Tasks.Ecto.Migrations.run/3
(elixir) lib/enum.ex:1314: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql) lib/mix/tasks/ecto.migrations.ex:38: Mix.Tasks.Ecto.Migrations.run/3
(mix) lib/mix/task.ex:316: Mix.Task.run_task/3
(mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2
I think the call to repo.stop(pid) should be replaced to repo.stop()
https://github.com/elixir-ecto/ecto_sql/blob/master/lib/mix/tasks/ecto.migrations.ex#L45
Note, without --log-sql
everything is fine.
hexpm[wm-ecto-3]% mix ecto.migrate --log-sql
[debug] QUERY OK source="schema_migrations" db=1.1ms
SELECT s0."version"::bigint FROM "schema_migrations" AS s0 FOR UPDATE []
[info] == Running Hexpm.Repo.Migrations.AddServiceToUsers.up/0 forward
[info] alter table users
[error] an exception was raised logging %DBConnection.LogEntry{call: :prepare_execute, connection_time: 12329000, decode_time: 14000, params: [], pool_time: nil, query: %Postgrex.Query{cache: :reference, columns: nil, name: "", param_formats: nil, param_oids: nil, param_types: nil, ref: nil, result_formats: nil, result_oids: nil, result_types: nil, statement: ["ALTER TABLE ", [34, "users", 34], 32, [[], "ADD COLUMN ", [34, "service", 34], 32, "boolean", [[" DEFAULT ", "false"], []]], []], types: nil}, result: {:ok, %Postgrex.Query{cache: :reference, columns: nil, name: "", param_formats: [], param_oids: [], param_types: [], ref: #Reference<0.2121130605.262930433.40304>, result_formats: [], result_oids: nil, result_types: nil, statement: ["ALTER TABLE ", [34, "users", 34], 32, [[], "ADD COLUMN ", [34, "service", 34], 32, "boolean", [[" DEFAULT ", "false"], []]], []], types: {Postgrex.DefaultTypes, #Reference<0.2121130605.263061505.39872>}}, %Postgrex.Result{columns: nil, command: :alter_table, connection_id: 80873, messages: [], num_rows: 0, rows: nil}}}: ** (FunctionClauseError) no function clause matching in Logger.__should_log__/1
(logger) lib/logger.ex:663: Logger.__should_log__(true)
(ecto) lib/ecto/log_entry.ex:57: Ecto.LogEntry.log/2
(ecto_sql) lib/ecto/adapters/sql.ex:752: Ecto.Adapters.SQL.log/4
(db_connection) lib/db_connection.ex:1303: DBConnection.log/5
(postgrex) lib/postgrex.ex:168: Postgrex.query_prepare_execute/4
(ecto_sql) lib/ecto/adapters/sql.ex:288: Ecto.Adapters.SQL.query!/4
(elixir) lib/enum.ex:1326: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql) lib/ecto/adapters/sql.ex:671: Ecto.Adapters.SQL.execute_ddl/4
(ecto_sql) lib/ecto/migration/runner.ex:272: Ecto.Migration.Runner.log_and_execute_ddl/3
(ecto_sql) lib/ecto/migration/runner.ex:105: anonymous fn/2 in Ecto.Migration.Runner.flush/0
(elixir) lib/enum.ex:1936: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto_sql) lib/ecto/migration/runner.ex:103: Ecto.Migration.Runner.flush/0
(stdlib) timer.erl:181: :timer.tc/2
(ecto_sql) lib/ecto/migration/runner.ex:27: Ecto.Migration.Runner.run/6
(ecto_sql) lib/ecto/migrator.ex:195: Ecto.Migrator.attempt/6
(ecto_sql) lib/ecto/migrator.ex:112: anonymous fn/3 in Ecto.Migrator.do_up/4
(ecto_sql) lib/ecto/adapters/sql.ex:774: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection) lib/db_connection.ex:1341: DBConnection.run_transaction/4
(ecto_sql) lib/ecto/migrator.ex:183: Ecto.Migrator.do_run_maybe_in_transaction/3
(elixir) lib/task/supervised.ex:88: Task.Supervised.do_apply/2
[info] execute "ALTER TABLE users ALTER password DROP NOT NULL"
[error] an exception was raised logging %DBConnection.LogEntry{call: :prepare_execute, connection_time: 496000, decode_time: 9000, params: [], pool_time: nil, query: %Postgrex.Query{cache: :reference, columns: nil, name: "", param_formats: nil, param_oids: nil, param_types: nil, ref: nil, result_formats: nil, result_oids: nil, result_types: nil, statement: "ALTER TABLE users ALTER password DROP NOT NULL", types: nil}, result: {:ok, %Postgrex.Query{cache: :reference, columns: nil, name: "", param_formats: [], param_oids: [], param_types: [], ref: #Reference<0.2121130605.262930433.40374>, result_formats: [], result_oids: nil, result_types: nil, statement: "ALTER TABLE users ALTER password DROP NOT NULL", types: {Postgrex.DefaultTypes, #Reference<0.2121130605.263061505.39872>}}, %Postgrex.Result{columns: nil, command: :alter_table, connection_id: 80873, messages: [], num_rows: 0, rows: nil}}}: ** (FunctionClauseError) no function clause matching in Logger.__should_log__/1
(logger) lib/logger.ex:663: Logger.__should_log__(true)
(ecto) lib/ecto/log_entry.ex:57: Ecto.LogEntry.log/2
(ecto_sql) lib/ecto/adapters/sql.ex:752: Ecto.Adapters.SQL.log/4
(db_connection) lib/db_connection.ex:1303: DBConnection.log/5
(postgrex) lib/postgrex.ex:168: Postgrex.query_prepare_execute/4
(ecto_sql) lib/ecto/adapters/sql.ex:288: Ecto.Adapters.SQL.query!/4
(elixir) lib/enum.ex:1326: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql) lib/ecto/adapters/sql.ex:671: Ecto.Adapters.SQL.execute_ddl/4
(ecto_sql) lib/ecto/migration/runner.ex:272: Ecto.Migration.Runner.log_and_execute_ddl/3
(ecto_sql) lib/ecto/migration/runner.ex:105: anonymous fn/2 in Ecto.Migration.Runner.flush/0
(elixir) lib/enum.ex:1936: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto_sql) lib/ecto/migration/runner.ex:103: Ecto.Migration.Runner.flush/0
(stdlib) timer.erl:181: :timer.tc/2
(ecto_sql) lib/ecto/migration/runner.ex:27: Ecto.Migration.Runner.run/6
(ecto_sql) lib/ecto/migrator.ex:195: Ecto.Migrator.attempt/6
(ecto_sql) lib/ecto/migrator.ex:112: anonymous fn/3 in Ecto.Migrator.do_up/4
(ecto_sql) lib/ecto/adapters/sql.ex:774: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection) lib/db_connection.ex:1341: DBConnection.run_transaction/4
(ecto_sql) lib/ecto/migrator.ex:183: Ecto.Migrator.do_run_maybe_in_transaction/3
(elixir) lib/task/supervised.ex:88: Task.Supervised.do_apply/2
[info] execute "INSERT INTO users (username, service, inserted_at, updated_at)\nVALUES ('hexdocs', true, now(), now())\n"
[error] an exception was raised logging %DBConnection.LogEntry{call: :prepare_execute, connection_time: 1963000, decode_time: 16000, params: [], pool_time: nil, query: %Postgrex.Query{cache: :reference, columns: nil, name: "", param_formats: nil, param_oids: nil, param_types: nil, ref: nil, result_formats: nil, result_oids: nil, result_types: nil, statement: "INSERT INTO users (username, service, inserted_at, updated_at)\nVALUES ('hexdocs', true, now(), now())\n", types: nil}, result: {:ok, %Postgrex.Query{cache: :reference, columns: nil, name: "", param_formats: [], param_oids: [], param_types: [], ref: #Reference<0.2121130605.262930433.40411>, result_formats: [], result_oids: nil, result_types: nil, statement: "INSERT INTO users (username, service, inserted_at, updated_at)\nVALUES ('hexdocs', true, now(), now())\n", types: {Postgrex.DefaultTypes, #Reference<0.2121130605.263061505.39872>}}, %Postgrex.Result{columns: nil, command: :insert, connection_id: 80873, messages: [], num_rows: 1, rows: nil}}}: ** (FunctionClauseError) no function clause matching in Logger.__should_log__/1
(logger) lib/logger.ex:663: Logger.__should_log__(true)
(ecto) lib/ecto/log_entry.ex:57: Ecto.LogEntry.log/2
(ecto_sql) lib/ecto/adapters/sql.ex:752: Ecto.Adapters.SQL.log/4
(db_connection) lib/db_connection.ex:1303: DBConnection.log/5
(postgrex) lib/postgrex.ex:168: Postgrex.query_prepare_execute/4
(ecto_sql) lib/ecto/adapters/sql.ex:288: Ecto.Adapters.SQL.query!/4
(elixir) lib/enum.ex:1326: Enum."-map/2-lists^map/1-0-"/2
(ecto_sql) lib/ecto/adapters/sql.ex:671: Ecto.Adapters.SQL.execute_ddl/4
(ecto_sql) lib/ecto/migration/runner.ex:272: Ecto.Migration.Runner.log_and_execute_ddl/3
(ecto_sql) lib/ecto/migration/runner.ex:105: anonymous fn/2 in Ecto.Migration.Runner.flush/0
(elixir) lib/enum.ex:1936: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto_sql) lib/ecto/migration/runner.ex:103: Ecto.Migration.Runner.flush/0
(stdlib) timer.erl:181: :timer.tc/2
(ecto_sql) lib/ecto/migration/runner.ex:27: Ecto.Migration.Runner.run/6
(ecto_sql) lib/ecto/migrator.ex:195: Ecto.Migrator.attempt/6
(ecto_sql) lib/ecto/migrator.ex:112: anonymous fn/3 in Ecto.Migrator.do_up/4
(ecto_sql) lib/ecto/adapters/sql.ex:774: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection) lib/db_connection.ex:1341: DBConnection.run_transaction/4
(ecto_sql) lib/ecto/migrator.ex:183: Ecto.Migrator.do_run_maybe_in_transaction/3
(elixir) lib/task/supervised.ex:88: Task.Supervised.do_apply/2
[info] == Migrated in 0.0s
hexpm[wm-ecto-3]%
No error
Hello folks! I've been participating in the various conversations around telemetry
project, and I'd like to propose a change in the Event naming scheme that Ecto uses...
Right now it is configurable, so a user could change the event name to [:my_app, :query]
. The impact of this is that it's impossible to attach to Ecto events w/o some configuration input from the user (since the event name is needed up-front to subscribe to the event stream. This situation doesn't improve the ergonomics at all, but does introduce a burden on the user & the reporter library.
I'd like to suggest that the event name be static, so that all event name sent from Telemetry for Ecto instrumentation is always the same, for example:
:telemetry.execute([:ecto, :query], total, metadata)
This change would mean that the raw events are discoverable & consumable directly by Telemetry reporters, while still allowing customization via metadata or through telemetry_metrics
. Any user-specific info, like a custom metric name or the Ecto Repo belong in the metadata.
Thanks!
cc/ @josevalim @arkgil
FYI, I think this should also be the generic recommendation for building Telemetry instrumentation into libraries.
Could it be possible to make the log functions available so you can format the data you receive from telemetry?
I would like to reformat the uuids before printing the logging.
But since you can no longer update the data you can no longer make use of logging of the printing code from this library.
So now I have to copy all the code from this library with log: false setting. And print the logging myself with this code and all the functions that are referenced in it.
Logger.log(
level,
fn -> log_iodata(measurements, metadata) end,
ansi_color: sql_color(query_string)
)
There has recently been an update to plug (v1.8.0) and it looks like there is a conflict with the telemetry library for ecto_sql. It looks like everything is up to date in master but a release has not been cut for a while. Is there any way we can get out patch release?
Considering this table
defmodule MyApp.Repo.CreatePosts do
use Ecto.Migration
def change do
create table(:likes) do
add :user_id, references(:users, on_delete: :nothing)
add :post_id, references(:posts, on_delete: :nothing)
timestamps()
end
create index(:likes, [:user_id])
create index(:likes, [:post_id])
create unique_index(:likes, [:user_id, :post_id], name: :unique_user_post_index)
end
end
and a follow-up migration that just aims to change the on_delete
behaviour:
defmoule MyApp.Repo.ChangeOnDeleteOfUsersAndPostsOnLikes do
use Ecto.Migration
def change do
alter table(:likes) do
modify :user_id, references(:users, on_delete: :delete_all), from: references(:users, on_delete: :nothing)
modify :post_id, references(:posts, on_delete: :delete_all), from: references(:posts, on_delete: :nothing)
end
end
end
Now fails in PostgreSQL since 11.4, but worked fine in previous versions. The error message given is:
* (Postgrex.Error) ERROR 42P07 (duplicate_table) relation "unique_user_post_index" already exists
Not sure if this isn't just a bug introduced with this release of PostgreSQL or if this is a bug on ecto_sql. What I did find out however is that the following SQL is being generated:
ALTER TABLE "likes"
DROP CONSTRAINT "likes_user_id_fkey",
ALTER COLUMN "user_id" TYPE bigint,
ADD CONSTRAINT "likes_user_id_fkey" FOREIGN KEY ("user_id") REFERENCES "users"("id") ON DELETE CASCADE,
DROP CONSTRAINT "likes_post_id_fkey",
ALTER COLUMN "post_id" TYPE bigint,
ADD CONSTRAINT "likes_post_id_fkey" FOREIGN KEY ("post_id") REFERENCES "posts"("id") ON DELETE CASCADE
Which fails when run on the psql
console as well, but the following SQL executes and has the desired result:
ALTER TABLE "likes"
DROP CONSTRAINT "likes_user_id_fkey",
ADD CONSTRAINT "likes_user_id_fkey" FOREIGN KEY ("user_id") REFERENCES "users"("id") ON DELETE CASCADE,
DROP CONSTRAINT "likes_post_id_fkey",
ADD CONSTRAINT "likes_post_id_fkey" FOREIGN KEY ("post_id") REFERENCES "posts"("id") ON DELETE CASCADE
So, the question is: Should a migration that changes/resets the foreign key reference on a column also declare the column type that doesn't change? I understand that changing the reference to another table might result in a change of type for the column and that it's kinda hard to figure out if this does beforehand. However, the required information is all there in the from:
option....
On our production system we run migrations as pre startup task. We deployed a new version of our app with several migrations to be applied (Let's call them M1, M2, M3, ..). From what was logged we can see that M1 was applied successfully. M2 started and took too long to complete before our app health check failed and the app was killed. After restart the migrations were supposed to rerun but failed due to inconsistent state in schema_migrations.
Logs of the first run
Running migrations
Loading my_app..
Starting dependencies..
Starting repos..
Running migrations for my_app
11:24:22.407 [info] == Running 20190131122210 MyApp.Repo.Migrations.M1.change/0 forward
11:24:22.408 [info] create table table_1
11:24:22.436 [info] create index table_1_column_1_index
11:24:22.440 [info] alter table table_2
11:24:22.447 [info] execute "DROP MATERIALIZED VIEW view_1;"
11:24:22.456 [info] == Migrated 20190131122210 in 0.0s
11:24:22.523 [info] == Running 20190131122211 MyApp.Repo.Migrations.M2.change/0 forward
11:24:22.524 [info] execute "update table_3 set status = 'status_1' where status = 'status_2';"
(killed by health check policy)
Logs of the second run
Loading my_app..
Starting dependencies..
Starting repos..
Running migrations for my_app
11:28:48.228 [info] == Running 20190131122210 MyApp.Repo.Migrations.M1.change/0 forward
11:28:48.229 [info] create table table_1
[31m▸ Evaluation failed with: ERROR 42P07 (duplicate_table) relation "table_1" already exists[0m
(returned non 0 exit code)
Migration should be marked as applied even if app terminates during applying of following migrations
I would expect the following to create a cross-schema reference:
create table(:comments, prefix: :child_app) do
add :user_id, references("parent.users")
end
Alternatively:
create table(:comments, prefix: :child_app) do
add :user_id, references("users", prefix: :parent)
end
I feel like the docs here are a bit confusing.
I think this could be read as execute/2
is one of those commands that are non-reversible in change/0
(even though it provides an up/0
and down/0
).
This confused me quite a bit, actually, and I wasted a bunch of time trying to find some examples of execute/2
's usage in migrations.
Thoughts? Should we make it more clear?
Or am I just terrible at reading comprehension? 🤣
Since upgrading to Ecto 3.0.3 we randomly get this error in tests:
(DBConnection.ConnectionError) connection not available and request was dropped from queue after 254ms
It seems to only happen sometimes, but once it happens all subsequent tests are affected.
timestamp
vs timestamptz
columns, and am aware of (and agree with) Ecto just using timestamp
If I have a legacy table (defined in another system) with a column of type timestamptz
, I cannot use Ecto to write to that column.
TzTest.changeset(%TzTest{}, %{tz_col: DateTime.utc_now()}) |> Repo.insert()
results in this in the log:
11:44:59.028 [debug] QUERY ERROR db=13.7ms queue=1.1ms
INSERT INTO "tz_test" ("tz_col") VALUES ($1) RETURNING "id" [~N[2018-11-05 16:44:58.996802]]
even though schema is defined as
schema "tz_test" do
field(:tz_col, :utc_datetime_usec)
end
Full source available in test project at https://github.com/jamesvl/ecto_tz_test
Insert or updates write a DateTime value.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.