Giter VIP home page Giter VIP logo

mysql-otp's Introduction

MySQL/OTP

Build Status 🔗 Test coverage (EUnit) 🔗 API documentation (EDoc) 🔗 Hex package

MySQL/OTP is a driver for connecting Erlang/OTP applications to MySQL and MariaDB databases. It is a native implementation of the MySQL protocol in Erlang.

Some of the features:

  • Mnesia style transactions:
    • Nested transactions are implemented using SQL savepoints.
    • Transactions are automatically retried when deadlocks are detected.
  • Each connection is a gen_server, which makes it compatible with Poolboy (for connection pooling) and ordinary OTP supervisors.
  • SSL.
  • Authentication methods caching_sha2_password (default from MySQL 8.0.4) and mysql_native_password (default from MySQL 4.1).
  • Parametrized queries using cached unnamed prepared statements (What?)
  • Slow queries are interrupted without killing the connection (MySQL version ≥ 5.0.0)
  • Implements both protocols: the binary protocol for prepared statements and the text protocol for plain queries.

Requirements:

  • Erlang/OTP version R16B or later
  • MySQL database version 4.1 or later or MariaDB
  • GNU Make or Rebar or any other tool for building Erlang/OTP applications

Synopsis

%% Connect (ssl is optional)
{ok, Pid} = mysql:start_link([{host, "localhost"}, {user, "foo"},
                              {password, "hello"}, {database, "test"},
                              {ssl, [{server_name_indication, disable},
                                     {cacertfile, "/path/to/ca.pem"}]}]),

%% Select
{ok, ColumnNames, Rows} =
    mysql:query(Pid, <<"SELECT * FROM mytable WHERE id = ?">>, [1]),

%% Manipulate data
ok = mysql:query(Pid, "INSERT INTO mytable (id, bar) VALUES (?, ?)", [1, 42]),

%% Separate calls to fetch more info about the last query
LastInsertId = mysql:insert_id(Pid),
AffectedRows = mysql:affected_rows(Pid),
WarningCount = mysql:warning_count(Pid),

%% Mnesia style transaction (nestable)
Result = mysql:transaction(Pid, fun () ->
    ok = mysql:query(Pid, "INSERT INTO mytable (foo) VALUES (1)"),
    throw(foo),
    ok = mysql:query(Pid, "INSERT INTO mytable (foo) VALUES (1)")
end),
case Result of
    {atomic, ResultOfFun} ->
        io:format("Inserted 2 rows.~n");
    {aborted, Reason} ->
        io:format("Inserted 0 rows.~n")
end,

%% Multiple queries and multiple result sets
{ok, [{[<<"foo">>], [[42]]}, {[<<"bar">>], [[<<"baz">>]]}]} =
    mysql:query(Pid, "SELECT 42 AS foo; SELECT 'baz' AS bar;"),

%% Graceful timeout handling: SLEEP() returns 1 when interrupted
{ok, [<<"SLEEP(5)">>], [[1]]} =
    mysql:query(Pid, <<"SELECT SLEEP(5)">>, 1000),

%% Close the connection
mysql:stop(Pid).

Usage as a dependency

Using erlang.mk:

DEPS = mysql
dep_mysql = git https://github.com/mysql-otp/mysql-otp 1.8.0

Using rebar (version 2 or 3):

{deps, [
  {mysql, ".*", {git, "https://github.com/mysql-otp/mysql-otp",
                {tag, "1.8.0"}}}
]}.

Using mix:

{:mysql, git: "https://github.com/mysql-otp/mysql-otp", tag: "1.8.0"},

There's also a Hex package called mysql.

Tests

EUnit tests are executed using make tests or make eunit.

To run individual test suites, use make eunit t=SUITE where SUITE is one of mysql_encode_tests, mysql_protocol_tests, mysql_tests, ssl_tests or transaction_tests.

The encode and protocol test suites does not require a running MySQL server on localhost.

For the suites mysql_tests, ssl_tests and transaction_tests you need to start MySQL on localhost and give privileges to the users otptest, otptest2 and (for ssl_tests) to the user otptestssl:

CREATE USER otptest@localhost IDENTIFIED BY 'OtpTest--123';
GRANT ALL PRIVILEGES ON otptest.* TO otptest@localhost;

CREATE USER otptest2@localhost IDENTIFIED BY 'OtpTest2--123';
GRANT ALL PRIVILEGES ON otptest.* TO otptest2@localhost;

-- in MySQL < 5.7, REQUIRE SSL must be given in GRANT
CREATE USER otptestssl@localhost IDENTIFIED BY 'OtpTestSSL--123';
GRANT ALL PRIVILEGES ON otptest.* TO otptestssl@localhost REQUIRE SSL;

-- in MySQL >= 8.0, REQUIRE SSL must be given in CREATE USER
CREATE USER otptestssl@localhost IDENTIFIED BY 'OtpTestSSL--123' REQUIRE SSL;
GRANT ALL PRIVILEGES ON otptest.* TO otptestssl@localhost;

Before running the test suite ssl_tests you'll also need to generate SSL files and MySQL extra config file. In order to do so, please execute make tests-prep.

The MySQL server configuration must include my-ssl.cnf file, which can be found in test/ssl/. Do not run make tests-prep after you start MySQL, because CA certificates will no longer match.

If you run make tests COVER=1 a coverage report will be generated. Open cover/index.html to see that any lines you have added or modified are covered by a test.

Contributing

Run the tests and also dialyzer using make dialyze.

Linebreak code to 80 characters per line and follow a coding style similar to that of existing code.

Keep commit messages short and descriptive. Each commit message should describe the purpose of the commit, the feature added or bug fixed, so that the commit log can be used as a comprehensive change log. CHANGELOG.md is generated from the commit messages.

Maintaining

This is for the project's maintainer(s) only.

Tagging a new version:

  1. Before tagging, update src/mysql.app.src and README.md with the new version.
  2. Tag and push tags using git push --tags.
  3. After tagging a new version:
  • Update the changelog using make CHANGELOG.md and commit it.
  • Update the online documentation and coverage reports using make gh-pages. Then push the gh-pages branch using git push origin gh-pages.

Updating the Hex package (requires Mix):

make publish-hex

License

GNU Lesser General Public License (LGPL) version 3 or any later version. Since the LGPL is a set of additional permissions on top of the GPL, both license texts are included in the files COPYING and COPYING.LESSER respectively.

mysql-otp's People

Contributors

choptastic avatar fenek avatar geraldxv avatar getong avatar hunkhl avatar imrivera avatar ingwinlu avatar juhlig avatar kianmeng avatar krishnakumar4a4 avatar kzemek avatar michaellenaghan avatar qingchuwudi avatar sergejjurecko avatar silviucpp avatar skaee avatar suexcxine avatar sztheory avatar umatomba avatar weisslj avatar wk8 avatar zuiderkwast avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mysql-otp's Issues

Mysql command timeout is really short, should be longer

?cmd_timeout (https://github.com/mysql-otp/mysql-otp/blob/master/src/mysql.erl#L48) is set to 3 seconds. It seems a bit short when looking at other clients that has 30 seconds.

Example clients: https://dev.mysql.com/doc/connector-net/en/connector-net-programming-mysqlcommand.html defaults to 30 seconds, but is configurable.

We discovered the issue when issuing a large commit and the database had big load.

So should we only bump the timeout to 30 seconds?
Or should it be configurable.

1.3.2 Hex package

Is there any change for the hex page for version 1.3.2? Currently the newest hex package is 1.3.1

Support Stored Procedure.

The error message comes after the call stored procedure.
mysql_pool:query(pool_game, "CALL sp_get_user_info(?)", [UID]).
{error,{1312,<<"0A000">>,
<<"PROCEDURE gamedb.sp_get_user_info can't return a result set in the given context">>}}

Replication protocol

If we implement the replication protocol, we can effectively subscribe to all changes in a database. An application of this would be to manage a complete or partial mirror of the database in mnesia.

Several unit tests failing

@zuiderkwast As I mentioned in my previous PR several unit tests are failing for me.
I'm running MySQL 5.7.13 (installed via brew) on Mac OS X 10.11.4.
Here's the full output from make tests on a pristine master checkout:

$ make tests
 APP    mysql.app.src
 GEN    test-dir
 GEN    eunit
mysql_tests: query_test_ (Text protocol)...*failed*
in function mysql_tests:text_protocol/1 (test/mysql_tests.erl, line 278)
**error:{badmatch,{error,{1364,<<"HY000">>,
                  <<"Field 'tx' doesn't have a default value">>}}}
  output:<<"">>

mysql_tests: query_test_ (Binary protocol)...*failed*
in function mysql_tests:binary_protocol/1 (test/mysql_tests.erl, line 298)
**error:{badmatch,{error,{1050,<<"42S01">>,<<"Table 't' already exists">>}}}
  output:<<"">>

mysql_tests: query_test_ (INT)...*failed*
in function mysql_tests:write_read_text_binary/5 (test/mysql_tests.erl, line 527)
in call from mysql_tests:int/1 (test/mysql_tests.erl, line 412)
**error:{badmatch,{error,{1264,<<"22003">>,
                  <<"Out of range value for column 'i' at row 1">>}}}
  output:<<"">>

mysql_tests: query_test_ (DATE)...*failed*
in function mysql_tests:write_read_text_binary/5 (test/mysql_tests.erl, line 527)
in call from lists:foreach/2 (lists.erl, line 1337)
in call from mysql_tests:date/1 (test/mysql_tests.erl, line 439)
**error:{badmatch,{error,{1292,<<"22007">>,
                  <<"Incorrect date value: '0000-00-00' for column 'd"...>>}}}
  output:<<"">>

mysql_tests: query_test_ (DATETIME)...*failed*
in function mysql_tests:write_read_text_binary/5 (test/mysql_tests.erl, line 527)
in call from lists:foreach/2 (lists.erl, line 1337)
in call from mysql_tests:datetime/1 (test/mysql_tests.erl, line 473)
**error:{badmatch,{error,{1292,<<"22007">>,
                  <<"Incorrect datetime value: '0000-00-00 00:00:00' "...>>}}}
  output:<<"">>

mysql_tests: query_test_ (Microseconds)...*failed*
in function mysql_tests:test_datetime_microseconds/1 (test/mysql_tests.erl, line 512)
**error:{badmatch,{error,{1050,<<"42S01">>,<<"Table 'dt' already exists">>}}}
  output:<<"">>

mysql_tests: log_warnings_test...*failed*
in function mysql_tests:log_warnings_test/0 (test/mysql_tests.erl, line 192)
**error:{badmatch,{error,{badmatch,{error,{1364,<<"HY000">>,
                                   <<"Field 'x' doesn't have a default"...>>}}},
                 [{mysql_tests,'-log_warnings_test/0-fun-0-',1,
                               [{file,"test/mysql_tests.erl"},{line,193}]},
                  {error_logger_acc,capture,1,
                                    [{file,"test/error_logger_acc.erl"},
                                     {line,28}]},
                  {mysql_tests,log_warnings_test,0,
                               [{file,"test/mysql_tests.erl"},{line,192}]},
                  {eunit_test,'-mf_wrapper/2-fun-0-',2,
                              [{file,"eunit_test.erl"},{line,266}]},
                  {eunit_test,run_testfun,1,
                              [{file,"eunit_test.erl"},{line,65}]},
                  {eunit_proc,run_test,1,[{file,"eunit_proc.erl"},{line,503}]},
                  {eunit_proc,with_timeout,3,[{file,[...]},{line,...}]},
                  {eunit_proc,handle_test,2,[{file,...},{...}]}],
                 []}}
  output:<<"">>

=======================================================
  Failed: 7.  Skipped: 0.  Passed: 51.
make: *** [eunit] Error 1

Some errors (maybe all?) seem related to sql_mode settings. The eunit tests assume that it will get warnings for e.g. missing values on insert, but an error is issued. The mysql behaviour for these cases are controlled by sql_mode settings.
These are my settings: (query: SELECT @@GLOBAL.sql_mode;)

ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

Those are the settings I got when installing mysql, I don't know if different distros can have different settings or if it is defined by the mysql version. But maybe it's a good idea for the unit tests to explicitly set the sql_mode variable on a session basis to get the expected behaviour?

not match {error, close}

start_link() at <0.142.0> exit with reason {{{case_clause,{error,closed}},[{mysql_protocol,fetch_response,5,[{file,"src/mysql_protocol.erl"},{line,303}]},
{mysql,execute_stmt,4,[{file,"src/mysql.erl"},{line,763}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,607}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,639}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},{gen_server,call,[<0.124.0>,{execute,log_ccu_stmt,[0,0,0]},infinity]}} in context child_terminated

not match {error, close}

Support connection with no password

Hi!!
is it possible, can you realize support for connection with no password?
like in mysql, it work in this example:

mysql -u root -h host database

but not work in your driver

{ok, Pid} = mysql:start_link([{host, "host"}, {user, "root"},{database, "database"},{password, []}]).
or
{ok, Pid} = mysql:start_link([{host, "host"}, {user, "root"},{database, "database"}}]).

it's not work

Add support for caching-sha2-password

caching-sha2-password became the default authentication plugin in MySQL 8.0.4

Trying to connect to a server with new default, the connection process crashes:

2018-04-24 13:43:18.523 [error] <0.2253.0> CRASH REPORT Process <0.2253.0> with 0 neighbours exited with reason: {auth_method,<<"caching_sha2_password">>} in mysql_protocol:build_handshake_response/5 line 310 in gen_server:init_it/6 line 352
2018-04-24 13:43:18.529 [error] <0.2250.0>@mongoose_rdbms:connect:450 Database connection attempt with {mysql,"localhost","ejabberd","ejabberd","mongooseim_secret",[{verify,verify_peer},{cacertfile,"priv/ssl/cacert.pem"}]} resulted in {error,{{auth_method,<<"caching_sha2_password">>},[{mysql_protocol,build_handshake_response,5,[{file,"/home/travis/build/esl/MongooseIM/_build/default/lib/mysql/src/mysql_protocol.erl"},{line,310}]},{mysql_protocol,handshake,7,[{file,"/home/travis/build/esl/MongooseIM/_build/default/lib/mysql/src/mysql_protocol.erl"},{line,64}]},{mysql,init,1,[{file,"/home/travis/build/esl/MongooseIM/_build/default/lib/mysql/src/mysql.erl"},{line,499}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}. Retrying in 2 seconds.

Problems with single qoute in a binary.

When encoding a binary that has a single quote in it, only the first instance of the single quote gets replaced by a double single quote.

Example
<<"My name is 'joey'.">>
Result:
<<"My name is ''joey'.">> (which will lead to an sql error).

Think the binary:replace is missing the global option in mysql_encode:encode/1 for the binary case.

Cannot connect to mysql 5.1.73

I get this error when trying to connect to a database running mysql 5.1.73. The same works fine on 5.5.42 and 5.6.23.

2> mysql:start_link([{user, "otptest"}, {password, "otptest"}]).
** exception exit: badarg
in function binary:last/1
called as binary:last(<<>>)
in call from mysql_protocol:parse_handshake/1 (src/mysql_protocol.erl, line 241)
in call from mysql_protocol:handshake/5 (src/mysql_protocol.erl, line 52)
in call from mysql:init/1 (src/mysql.erl, line 476)
in call from gen_server:init_it/6 (gen_server.erl, line 306)
in call from proc_lib:init_p_do_apply/3 (proc_lib.erl, line 237)

Support LOAD DATA LOCAL INFILE

Hello,
First, thanks for this repo, I replace mysql driver via mysql-otp. But, I found this driver can not support LOAD DATA LOCAL INFILE, just like:

LOAD DATA LOCAL INFILE '/path/myfile' INTO TABLE table_name;

is it possible, can you realize support for LOAD DATA LOCAL INFILE?

Thanks again.

CHAR/VARCHAR vs BINARY/VARBINARY

Is there a method to accurately distinguish between a CHAR/VARCHAR and a BINARY/VARBINARY column.

A BINARY column in mysql is often defined as TYPE_STRING within mysql-otp (debugging msql_protocol.erl) while VARBINARY is often defined as TYPE_VAR_STRING.

The type seems to be determined from the element following the column name as shown where the 253 matches to TYPE_VAR_STRING.

ColDef={col,<<"columnname">>,253,33,150,0,0};

IMPORTANT:
Is the element following the type some sort of sub-type. As seen, 33 shows up for CHAR/VARCHAR while 63 for BINARY/VARBINARY columns.
Can anyone confirm this sub-type??

CHAR, VARCHAR and TEXT appear to be consistently 33; while all others are 63.

Data types according to protocol.hrl

%% --- Types ---

-define(TYPE_DECIMAL, 16#00).
-define(TYPE_TINY, 16#01).
-define(TYPE_SHORT, 16#02).
-define(TYPE_LONG, 16#03).
-define(TYPE_FLOAT, 16#04).
-define(TYPE_DOUBLE, 16#05).
-define(TYPE_NULL, 16#06).
-define(TYPE_TIMESTAMP, 16#07).
-define(TYPE_LONGLONG, 16#08).
-define(TYPE_INT24, 16#09).
-define(TYPE_DATE, 16#0a).
-define(TYPE_TIME, 16#0b).
-define(TYPE_DATETIME, 16#0c).
-define(TYPE_YEAR, 16#0d).
-define(TYPE_VARCHAR, 16#0f).
-define(TYPE_BIT, 16#10).
-define(TYPE_JSON, 16#f5).
-define(TYPE_NEWDECIMAL, 16#f6).
-define(TYPE_ENUM, 16#f7).
-define(TYPE_SET, 16#f8).
-define(TYPE_TINY_BLOB, 16#f9).
-define(TYPE_MEDIUM_BLOB, 16#fa).
-define(TYPE_LONG_BLOB, 16#fb).
-define(TYPE_BLOB, 16#fc).
-define(TYPE_VAR_STRING, 16#fd).
-define(TYPE_STRING, 16#fe).
-define(TYPE_GEOMETRY, 16#ff).

When I ran the tests i got the error time out

I ran the tests with $ make tests

Operating System: Fedora 27

Erlang Version: 20

mysql_tests: query_test_ (INT)...*timed out*

=ERROR REPORT==== 14-Feb-2018::12:46:13 ===
** Generic server <0.85.0> terminating 
** Last message in was {'EXIT',<0.78.0>,killed}
** When Server state == {state,[5,7,21],
                               31,#Port<0.951>,mysql_sock_tcp,undefined,
                               "localhost",3306,"otptest","otptest",false,
                               60000,infinity,60000,1,2,0,0,0,
                               #Ref<0.2829144779.414973953.108221>,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}},
                               empty,true}
** Reason for termination == 
** killed
undefined

=INFO REPORT==== 14-Feb-2018::12:46:20 ===
TLS client: In state certify at ssl_handshake.erl:1624 generated CLIENT ALERT: Fatal - Unknown CA

ssl_tests: successful_ssl_connect_test (module 'ssl_tests')...*skipped*
undefined
*unexpected termination of test process*
::{failed_to_upgrade_socket,{tls_alert,"unknown ca"}}

=======================================================
  Failed: 0.  Skipped: 0.  Passed: 49.
One or more tests were cancelled.
make: *** [erlang.mk:762: eunit] Error 2

The whole application is shutdowned when using illegal parameter for query

(eLockAccount@johnzeng-virtual-machine)1> mysql:query(pool_mysql, <<"select * from account where id =?">>, [undefine]).

=ERROR REPORT==== 14-Apr-2017::11:05:01 ===
** Generic server <0.1261.0> terminating
** Last message in was {param_query,<<"select * from account where id =?">>,
                                    [undefine]}
** When Server state == {state,[5,5,54],
                               95,#Port<0.3137>,"localhost",3306,"root",
                               "Removed",true,infinity,infinity,60000,0,
                               2,0,0,0,undefined,
                               {dict,0,16,16,8,80,48,
                                     {[],[],[],[],[],[],[],[],[],[],[],[],[],
                                      [],[],[]},
                                     {{[],[],[],[],[],[],[],[],[],[],[],[],[],
                                       [],[],[]}}},
                               empty,false}
** Reason for termination ==
** {function_clause,[{mysql_protocol,encode_param,
                                     [undefine],
                                     [{file,"src/mysql_protocol.erl"},
                                      {line,745}]},
                     {lists,map,2,[{file,"lists.erl"},{line,1238}]},
                     {mysql_protocol,execute,5,
                                     [{file,"src/mysql_protocol.erl"},
                                      {line,161}]},
                     {mysql,execute_stmt,4,
                            [{file,"src/mysql.erl"},{line,804}]},
                     {gen_server,try_handle_call,4,
                                 [{file,"gen_server.erl"},{line,629}]},
                     {gen_server,handle_msg,5,
                                 [{file,"gen_server.erl"},{line,661}]},
                     {proc_lib,init_p_do_apply,3,
                               [{file,"proc_lib.erl"},{line,240}]}]}
** exception exit: {{function_clause,[{mysql_protocol,encode_param,
                                                      [undefine],
                                                      [{file,"src/mysql_protocol.erl"},{line,745}]},
                                      {lists,map,2,[{file,"lists.erl"},{line,1238}]},
                                      {mysql_protocol,execute,5,
                                                      [{file,"src/mysql_protocol.erl"},{line,161}]},
                                      {mysql,execute_stmt,4,[{file,"src/mysql.erl"},{line,804}]},
                                      {gen_server,try_handle_call,4,
                                                  [{file,"gen_server.erl"},{line,629}]},
                                      {gen_server,handle_msg,5,
                                                  [{file,"gen_server.erl"},{line,661}]},
                                      {proc_lib,init_p_do_apply,3,
                                                [{file,"proc_lib.erl"},{line,240}]}]},
                    {gen_server,call,
                                [pool_mysql,
                                 {param_query,<<"select * from account where id =?">>,
                                              [undefine]},
                                 infinity]}}
     in function  gen_server:call/3 (gen_server.erl, line 212)
     in call from mysql:query_call/2 (src/mysql.erl, line 792)
(eLockAccount@johnzeng-virtual-machine)2>
=INFO REPORT==== 14-Apr-2017::11:05:01 ===
    application: eLockAccount
    exited: shutdown
    type: permanent
{"Kernel pid terminated",application_controller,"{application_terminated,eLockAccount,shutdown}"}

Crash dump is being written to: erl_crash.dump...done
Kernel pid terminated (application_controller) ({application_terminated,eLockAccount,shutdown})
make: *** [console] Error 1

Is it possilbe to add param check to avoid it?? This it unacceptable for an erlang application.

I did some search on it and looks like the crash comes from the mysql:terminate/2 callback.

I modified the callback to do some logging:

terminate(Reason, #state{socket = Socket})
  when Reason == normal; Reason == shutdown ->
      io:format("now shutdown for reason:~p~n", [Reason]),
      %% Send the goodbye message for politeness.
      inet:setopts(Socket, [{active, false}]),
      io:format("setopts is done~n"),
      R = mysql_protocol:quit(gen_tcp, Socket),
      io:format("quit gen_tcp is done~n"),
      inet:setopts(Socket, [{active, once}]),
      io:format("set opts done again~n"),
      R;

and the output is like this:

now shutdown for reason:shutdown
(eLockAccount@johnzeng-virtual-machine)2>
=INFO REPORT==== 14-Apr-2017::14:56:17 ===
    application: eLockAccount
    exited: shutdown
    type: permanent
{"Kernel pid terminated",application_controller,"{application_terminated,eLockAccount,shutdown}"}

So the exception happens when calling

inet:setopts(Socket, [{active, false}]),

Please let me know what's wrong here. I don't wanna dig into it anymore.

How to skip checking statements in prepare config

hi, i use mysql-otp with a pool named pooler, url:https://github.com/seth/pooler

a question occured as that:
the pooler call mysql:start_link() and get the pid of mysql connection, the example config is that:

{pooler, [
           {pools, [
                    [{name, logger_pool},
                     {max_count, 50},
                     {init_count, 20},
                     {start_mfa,
                      {mysql, start_link, [[{host,"127.0.0.1"}, {port, 3306}, {user, "mycomm"}, {password, "mycomm123"}, {database, "c8config"},
                                            {prepare,
                                             [
                                              {'heartbeat.select', "select 1 from dual"}
                                             ]
                                            }
                                           ]]}}]
                   ]}
          ]}

when the statements are wrong, such as unknown column, mysql-otp throw an error of badmatch in

Prepare = proplists:get_value(prepare, Options, []),
            lists:foreach(fun ({Name, Stmt}) ->
                              {ok, Name} = mysql:prepare(Pid, Name, Stmt)
                          end,
                          Prepare);

pool 'logger_pool' failed to start member:{error,{'EXIT',{{badmatch,{error,{1054,<<"42S22">>,<<"Unknown column 'work_start_time11' in 'where clause'">>}}},[{mysql,'-start_link/1-fun-1-',2,[{file,"/root/workspace/cti_dialing_svr_test/_build/default/lib/mysql/src/mysql.erl"},{line,163}]},{lists,foreach,2,[{file,"lists.erl"},{line,1336}]},{mysql,start_link,1,[{file,"/root/workspace/cti_dialing_svr_test/_build/default/lib/mysql/src/mysql.erl"},{line,162}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,330}]},{supervisor,handle_call,3,[{file,"supervisor.erl"},{line,355}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,607}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,639}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]}}}

Pooler thought the connection was created failed, but mysql connection was created success.
I have a timer to execute stmt, so pooler always created a new connection.After a while the mysql connections reached the upper limit.
Use show processlist in mysql
I think it's not Pooler's fault, right?
What should I do? thx

Log slow and unindexed queries (options)

In each result we get some status bits from the server with the meanings "query was slow", "no index was used", "no good index was used". When this happens, log the query using error_logger:warning_msg("MySQL query was slow: ~s", [Query]), etc.

Options:

  • {log_slow_queries, boolean()}, default false
  • {log_no_index_used, boolean()} default false? Performing queries with no index is always bad.
  • {log_no_good_index_used, boolean()} default false?

I think we should log only slow queries and add the "index used" info to the "slow query" log entry. No index querys that are not slow are not logged. Examples:

"MySQL query (with no index) was slow: SELECT ..."
"MySQL query (with no good index) was slow: SELECT ..."
"MySQL query was slow: SELECT ..."

[Edited 20 March 2018]

Application name conflict

Hi, I am currently testing that everything compiles, in the process adding them all to the package index, and would like to add your application as well, however I am running into a problem. Your application's name is the same as the much older https://github.com/dizzyd/erlang-mysql-driver

... and that's all I got, really.

I don't know if they are API and feature compatible (in which case maybe better to switch to this one), and if they're not, then I don't really know how to handle that kind of case.

Thoughts?

License inconsistency

The README says LGPL is used, but GPL is shown in the github "status bar" (or whatever it is). You need to update the project settings in github to make it LGPL?

mysql centos fetch_response crash

start_link() at <0.142.0> exit with reason {{{case_clause,{error,closed}},[{mysql_protocol,fetch_response,5,[{file,"src/mysql_protocol.erl"},{line,303}]},
{mysql,execute_stmt,4,[{file,"src/mysql.erl"},{line,763}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,607}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,639}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,237}]}]},{gen_server,call,[<0.124.0>,{execute,log_ccu_stmt,[0,0,0]},infinity]}} in context child_terminated

not match {error, close}

Optionally replace Erlang error_logger with custom logger

When logging warnings, the library uses Erlang's error_logger which is known to have scalability issues. A lot of Erlang applications tend to use other logging solutions such as lager. Would like to know if it would be acceptable to add a feature where a user of this library can override the logging function through a config? Perhaps leave the default as error_logger if not specified?

I can work/implement this feature but thought it would be good to bring it up in case there are opinions on this.

Unprepare statements

There is a prepare function but no corrensponding unprepare or deallocate_prepare yet.

I'm not sure which of the names I like the most. In Emysql it's named unprepare. The corresponding SQL is DEALLOCATE PREPARE.

MariaDB 10.2.6 error in handshake

Hello there,

today I have updated MariaDB from 10.1.24 to 10.2.6 and now mysql:start_link([{name, {local,db}},{database,D},{user,U},{password,P}]) raises an exception:

** exception exit: {badmatch,2}
     in function  mysql_protocol:handshake/6 (/usr/home/u/project/_build/default/lib/mysql/src/mysql_protocol.erl, line 60)
     in call from mysql:init/1 (/usr/home/u/project/_build/default/lib/mysql/src/mysql.erl, line 497)
     in call from gen_server:init_it/6 (gen_server.erl, line 328)
     in call from proc_lib:init_p_do_apply/3 (proc_lib.erl, line 247)

When I comment out that line 60 in mysql_protocol.erl everything seems to work fine...

Does not support JSON type field

MySQL 5.7.8 forward supports JSON type fields. But mysql-otp does not seem to support it. I got error message such as the following when retrieving a record having JSON field:

** Reason for termination ==
** {function_clause,[{mysql_protocol,decode_text,
[{col,<<"blob">>,245,63,4294967295,0,144},
<<"["javascript", "es2015", "json"]">>],
[{file,"src/mysql_protocol.erl"},
{line,454}]},
{mysql_protocol,decode_text_row_acc,3,
[{file,"src/mysql_protocol.erl"},
{line,446}]},
{mysql_protocol,'-parse_resultset/3-lc$^0/1-0-',3,
[{file,"src/mysql_protocol.erl"},
{line,359}]},
{mysql_protocol,parse_resultset,3,
[{file,"src/mysql_protocol.erl"},
{line,359}]},
{mysql_protocol,fetch_response,5,
[{file,"src/mysql_protocol.erl"},
{line,323}]},

boolean atoms encoding

Hello,

In most of my projects most of the time I need to convert the true | false atoms into 1 or 0 before sending to mysql-otp.

I think will be nice if mysql-opt will do this on it's own. Right now anyway is crashing

Silviu

replace error(old_server_version)

so I recently had the pleasure to deploy to an older mysql version(5.1) and encountered the old_server_version error.

It would be much nicer to try and continue if possible and just not offer some of the features that are tested during the handshake (I only needed 1 simple query to get some data).

If that is not reasonable, it would at least be good to know which versions of popular sql servers are supported and what features are missing for mysql-otp to work in the error output.

bulk insert

Is there a way to pass multiple records in a single INSERT query?

6> mysql:query( C, "INSERT INTO mytable (id, bar) VALUES (?, ?)", [33, 33]).
ok
7> mysql:query( C, "INSERT INTO mytable (id, bar) VALUES (?, ?)", [[33, 33], [44,44]]). 
ok
8> 
=WARNING REPORT==== 13-Sep-2015::23:47:22 ===
Warning 1366: Incorrect integer value: '!!' for column 'id' at row 1
Warning 1366: Incorrect integer value: ',,' for column 'bar' at row 1
 in INSERT INTO mytable (id, bar) VALUES (?, ?)

Command 7 just inserts (0,0 )-:

I am asking because in another project (java) I observed that a few bulk inserts caused less CPU load time than many single INSERTs.

transactions

Hi there, just want to clarify how transactions will look like.

Nested transactions are specifically interesting, since each START TRANSACTION
statement implicitly commits previously started transcation. I'd like to have
some ability to check, if there is transaction in progress or, as shortcut,
start new or continue existing transaction.
What about START TRANSACTION/COMMIT/ROLLBACK statements explicitly passed
by user?
Do you consider to use savepoints?

Detect server shutdown

When the server shuts down cleanly it sends a TCP message to notify us that it is closing the connection. We do not detect this because our gen_tcp connection is in passive mode.

Solution: Set the connectionn to {active, once} and listen for info message {tcp_closed, _Socket}. Stop the gen_server when this happens (so that some supervisor or pool can start a new connection). Before any communication with the server (such as when we send a query) we should set the connection back to passive mode, do the communication and afterwards set it back to active once.

Note that we cannot detect a network split or a server crash. In this case there is no TCP message that tells us that the connection is gone.

Related issues: #22, #25 and mysql-otp/mysql-otp-poolboy#14

Log warnings (option)

Add an option to fetch and forward MySQL warnings and notes to error_logger. If warning_count > 0, fetch the warnings and log them using error_logger:warning_msg("MySQL ~s ~p: ~s in query ~s", [Type, Code, Message, Query]) where Type :: warning | note.

Option: {log_warnings, boolean()}, default true?

Question: Does connect timeout infinity works perfectly?

I have couple of questions regarding the connect timeout value of infinity.

  1. Will the infinity timeout works even if there is no query request for a long time?
  2. I have saved the Pid returned by start_link() at the start of the service which is passed to Erlang processes when a request is received. When the MySQL node goes down, and then I restart the MySQL node, the connector will not work without restarting my service, thereby getting a new Pid. How to deal with such a case using the connector without restarting my service?

Thanks.

Floats: What is correct behaviour?

How to reproduce:

  • Let's say we have a table with a column of type FLOAT. That means a 32 bit float.
  • Insert a value using a simple query: INSERT INTO tab (x) VALUES (3.14).
  • Text protocol (normal queries): We receive the value as a binary string, e.g. <<"3.14">>. We parse this using binary_to_float which gives us a double that is rendered as 3.14.
  • Binary protocol (prepared statements): The value is sent as a 32 bit float, just as it is stored in the database. We parse this using a binary pattern like <<Value:32/float-little>>. This value is rendered as 3.140000104904175. Obviously we have a problem.

What behaviour do we want here?

A property that I would like is that we should get the same value regardless of how we fetch the value. That seems reasonable.

I have only one idea how to solve it right now: In the text protocol, after parsing the value, convert it to a 32-bit float and back to get the same precision loss.

1> Value = binary_to_float(<<"3.14">>).
3.14
2> <<LossyValue:32/float-little>> =  <<Value:32/float-little>>.
<<"ÃõH@">>
3> LossyValue.
3.140000104904175

Comments on this? Any other ideas?

Multiple Statements fails

Hi,

first of all thank you for your very nice work with this driver.

I am trying to run your example of multiple statement and it fails with:

19> mysql:query(PID, "SELECT 42 AS foo; SELECT 'baz' AS bar;").
{error,{1064,<<"42000">>,
        <<"You have an error in your SQL syntax; check the manual that corresponds to your MySQL server ver"...>>}}

I am running against a RDS MySQL instance

Best

Pedro

Cached anonymous prepared statements

A query on the form

mysql:query(Pid, <<"SELECT * FROM mytable WHERE id=?">>, [42]).

needs to be executed as a prepared statement (unless we parse the SQL code and substitute the parameter values for the quetion marks, but that we don't).

Instead of just preparing a statement, executing it and unpreparing it (three roundtrips to the server), we could cache the prepared statement in the connection state or an ETS table for a certain time. We gain:

  • The same query will already be prepared the next time, i.e. only one roundtrip for cached queries
  • Even if the same query is not executed again, the unprepare call becomes asynchronous so there are only two roundtrip in the blocking call.

Todo:

  • An option for the lifetime of this cache or a maximum size or both. Name and default values for these options.
  • Possibly a way to access and update these options with gen_server calls. (Can be left as future work.)
  • Decide whether to use an ETS table or a dict in the connection gen_server state and how and when to expire entries. (Minor implementation design decisions.)

param undefined exception

no function clause matching mysql_protocol:encode_param(undefined) line 737

I think undefined should handle the null.

Prepared statements

Suggestion:

%% Unnamed
{ok, Ref} = mysql:prepare(Connection, <<"SELECT * FROM foo WHERE id = ?">>),
{ok, Columns, Rows} = mysql:query(Connection, Ref, [42]).

%% Named
{ok, foo} = mysql:prepare(Connection, foo, <<"SELECT * FROM foo WHERE id = ?">>),
{ok, Columns, Rows} = mysql:query(Connection, foo, [42]).

When a statement is prepared, various metadata is returned such as column count and the types of each column. I suggest this information be stored within the connection state (in a dict or ets table) and only an integer or an atom be returned as the reference.

This issue involves implementing the binary protocol.

Rows affected vs Rows matched in update

I see the rows affected function; is it possible to add a "rows_matched" function so its feasible to distinguish between a query not found and a query found but an update not changed. Currently one must do two ops.

Here is the response inside mysql shell.
Query OK, 0 rows affected (0.00 sec)
Rows matched: 1 Changed: 0 Warnings: 0

Tests

Tests for all datatypes and sizes:

  • fetching using the text protocol
  • fetching using the binary protocol (preparad statement)
  • sending using the binary protocol (parameters for a prepared statement)

Check the coverage report to make sure we cover all cases such as negative TIME values, DATETIMEs with and without micro seconds, etc.

Below are some other ideas for tests, but these are too general for a specific task:

  • Add hexdumps from examples in the documentation "MySQL Internals" as unit tests.
  • Run tests against different versions of MySQL, MariaDB, etc.
  • Handshake with a username/password where the server requires "old" or some other authentication.

binary <->

Hello,
First, thanks for this repo. i want to store a record into the BLOB column, and then read back from mysql as a record. how can i do this? thx

Problems with decimal columns

Hello,

I have the following test schema:

CREATE TABLE `test_decimals` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `balance` decimal(13,4) NOT NULL DEFAULT '0.0000',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=15 DEFAULT CHARSET=utf8;

INSERT INTO test_decimals(id, balance) VALUES(1, 5000);

And the following code snippet:

-module(test_decimal).

-export([start/0, exec/1]).

start() ->
    {ok, Pid} = mysql:start_link([
        {host, "127.0.0.1"},
        {port, 3306},
        {user, "root"},
        {password, "root"},
        {database, "test_db"},
        {keep_alive, 300000},
        {queries, [<<"set names 'utf8';">>]},
        {prepare, [{debit_account, <<"UPDATE test_decimals SET balance = balance - ? WHERE id = ?">>}]}
    ]),

    Pid.

exec(Pid)->
    mysql:execute(Pid, debit_account, [10.2, 1]).

If you run Pid = test_decimal:start(). to create a connection and then run several times test_decimal:exec(Pid). you will notice from time to time the following warning:

[warning] Note 1265: Data truncated for column 'balance' at row 1
 in UPDATE test_decimals SET balance = balance - ? WHERE id = ?

I tested the same query with mysql workbench, emysql client, and also the mysql console and I cannot see this warning.

I'm sure is related on how floats are encoded in mysql-otp in prepared statements.

Any Idea how I can get rid of this problem ?

Silviu

Timouts using KILL QUERY

A slow query hangs when we have sent the query and we are waiting for the first packet to return. We could use a timeout on the recv here and send "KILL QUERY ". This way we don't have to kill the connection and if we are inside a transaction, it is not lost.

API: Add an optional last parameter Timeout to mysql:query/2,3 and mysql:execute/3. These return {error, timeout} on timeout.

The command line mysql client does this when ^C is pressed while waiting for a query. Example:

mysql> SELECT SLEEP(5);
^CCtrl-C -- sending "KILL QUERY 28" to server ...
Ctrl-C -- query aborted.
+----------+
| SLEEP(5) |
+----------+
+----------+
1 row in set (0,33 sec)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.