mymedsandme / marco_polo Goto Github PK

When many requests are made (in my case, synchronously), I eventually get the following error message:

** (exit) exited in: :gen_server.call(#PID<0.644.0>, {:operation, :command, [{:raw, "s"}, 194, {:raw, <<0, 0, 0, 1, 99>>}, {:raw, [<<0, 0, 0, 159, 67, 82, 69, 65, 84, 69, 32, 69, 68, 71, 69, 32, 104, 97, 115, 95, 100, 101, 109, 111, 103, 114, 97, 112, 104, 105, 99, 95, 114, 101, 115, 112, 111, 110, 115, 101, 10, 70, 82, ...>>, <<1>>, [<<0, 0, 0, 20>>, [0, [[<<0>>, ""], [[[[<<20>>, "parameters"], <<0, 0, 0, 19>>, 12]], 0, [[<<0>>, [], []]]]]]], <<0>>]}]}, 5000)
    ** (EXIT) time out
        (stdlib) gen_server.erl:212: :gen_server.call/3
    (marco_polo) lib/marco_polo.ex:831: MarcoPolo.refetching_schema/2
     (ex_orient) lib/ex_orient/db.ex:29: anonymous fn/3 in ExOrient.DB.command/2
       (poolboy) src/poolboy.erl:76: :poolboy.transaction/3
      (hydrogen) lib/migrator/agents_agentattribute.ex:12: anonymous fn/1 in Migrator.AgentsAgentAttribute.run/1
        (elixir) lib/enum.ex:610: anonymous fn/3 in Enum.each/2
        (elixir) lib/enum.ex:1478: anonymous fn/3 in Enum.reduce/3
        (elixir) lib/stream.ex:1227: anonymous fn/3 in Enumerable.Stream.reduce/3
        (elixir) lib/enum.ex:2744: Enumerable.List.reduce/3
        (elixir) lib/stream.ex:732: Stream.do_list_transform/9
        (elixir) lib/stream.ex:1247: Enumerable.Stream.do_each/4
        (elixir) lib/enum.ex:1477: Enum.reduce/3
        (elixir) lib/enum.ex:609: Enum.each/2
      (hydrogen) lib/migrator/all.ex:69: Migrator.All.migrate/1

I can submit a PR for this, but just wanted to test the waters beforehand.

Would it be useful to use https://github.com/avdi/dotenv_elixir to manage environment variables in test?

When using marco_polo in a very simple phoenix app I get very slow response times. Returning 10 results (without any associations) as json takes about 200ms. The same request using Ecto and postgresql takes around 10ms.

OrientDB is running on the same machine, so network lag cannot be the issue. When I use the OrientDB Studio (Webinterface) the request takes around 30ms to render. I tested this using a simple worker and a pool of workers using poolboy. The latter was the slowest with ~250ms per request.

200ms might sound acceptable. But part of the reason I'm using Elixir / Phoenix is speed. And if the database connection slows my app down by factor 20 it is a problem.

My testing enviroment

orientdb-community-2.1.8(Mac OSX 10.11.3)
orientdb-community-2.1.13(CentOS6.7)

Elixir version:1.2.3
marco_polo:0.2.1

Error exception below were there

  #mix test  test/
** (ErlangError) erlang error: :enoent
    (elixir) lib/system.ex:450: System.cmd("orientdb-console", ["SET ignoreErrors true;\nDROP DATABASE remote:localhost/MarcoPoloTest root root;\nDROP DATABASE remote:localhost/MarcoPoloTestGenerated root root;\nDROP DATABASE remote:localhost/MarcoPoloToDrop root root;\nSET ignoreErrors false;\n\nCREATE DATABASE remote:localhost/MarcoPoloTest root root plocal;\nCREATE DATABASE remote:localhost/MarcoPoloToDrop root root memory;\n"], [stderr_to_stdout: true])
    test/test_helper.exs:47: anonymous fn/1 in :elixir_compiler_1.__FILE__/1
    test/test_helper.exs:82: (file)
    (elixir) lib/code.ex:363: Code.require_file/2
    (elixir) lib/enum.ex:604: Enum."-each/2-lists^foreach/1-0-"/2

please respond to this matter...

try marco_polo with 0.2.2, always get time out error.

ex(5)> {:ok, conn} = MarcoPolo.start_link(user: "reader", password: "reader", connection: :server)
{:ok, #PID<0.280.0>}
** (EXIT from #PID<0.272.0>) evaluator process exited with reason: time out
        
Interactive Elixir (1.5.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(6)> 18:42:46.100 [error]        GenServer #PID<0.280.0> terminating
** (stop) time out
Last message: nil
State: %{live_query_tokens: #HashDict<[]>, opts: [host: "localhost", port: 2424, user: "reader", password: "reader", connection: :server], protocol_version: 36, queue: {[], []}, schema: nil, session_id: nil, socket: #Port<0.8089>, tail: "", transaction_id: 1}
iex(1)>

Right now, if our connection gets a {:tcp, socket, msg} message and the message contains more than on full responses (e.g., 2 full responses and some leftover data), then I'll spend eternity in hell I'm sure.

Basically, a {:tcp, socket, msg} is the only "trigger" that says to the connection "hey, reply to a waiting client". What we should do is parse a response and (if it's complete) recursively parse the rest of the tail, until we have an incomplete response or an empty queue.

Installed OrientDB on OSX via docker with:

docker run -d --name orientdb -p 2424:2424 -p 2480:2480 -e ORIENTDB_ROOT_PASSWORD=root orientdb:latest

Created the otp_ice database through the web console on port 2480.

When trying to start_link I get a timeout error:

iex(1)> {:ok, conn} = MarcoPolo.start_link(user: "admin", password: "admin", connection: {:db, "otp_ice", :graph})
** (EXIT from #PID<0.265.0>) time out

Interactive Elixir (1.3.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> [error] GenServer #PID<0.267.0> terminating
** (stop) time out
Last message: nil
State: %{live_query_tokens: #HashDict<[]>, opts: [host: "localhost", port: 2424, user: "admin", password: "admin", connection: {:db, "otp_ice", :graph}], protocol_version: 36, queue: {[], []}, schema: nil, session_id: nil, socket: #Port<0.7584>, tail: "", transaction_id: 1}```

Hi,

Does this driver works with Erlang ?

Thanks

Trying to copy the examples, for example:
{:ok, conn} = MarcoPolo.start_link(user: "admin", password: "admin", connection: {:db, "GratefulDeadConcerts"})

Does not work, and users might be wondering what is wrong, the "connection" map needs either a :graph or :document atom provided.

Ref:
`Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.2.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> {:ok, conn} = MarcoPolo.start_link(user: "admin",
...(1)> password: "admin",
...(1)> connection: {:db, "GratefulDeadConcerts"})
{:ok, #PID<0.130.0>}
** (EXIT from #PID<0.126.0>) an exception was raised:
** (ArgumentError) invalid connection type, valid ones are :server or {:db, name, type}
lib/marco_polo/connection/auth.ex:84: MarcoPolo.Connection.Auth.op_and_args_from_connection_type/1
lib/marco_polo/connection/auth.ex:58: MarcoPolo.Connection.Auth.op_and_connection_args/1
lib/marco_polo/connection/auth.ex:46: MarcoPolo.Connection.Auth.authenticate/1
lib/marco_polo/connection.ex:131: MarcoPolo.Connection.connect/2
lib/connection.ex:623: Connection.enter_connect/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

Interactive Elixir (1.2.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
21:06:22.445 [error] GenServer #PID<0.130.0> terminating
** (ArgumentError) invalid connection type, valid ones are :server or {:db, name, type}
lib/marco_polo/connection/auth.ex:84: MarcoPolo.Connection.Auth.op_and_args_from_connection_type/1
lib/marco_polo/connection/auth.ex:58: MarcoPolo.Connection.Auth.op_and_connection_args/1
lib/marco_polo/connection/auth.ex:46: MarcoPolo.Connection.Auth.authenticate/1
lib/marco_polo/connection.ex:131: MarcoPolo.Connection.connect/2
lib/connection.ex:623: Connection.enter_connect/5
(stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Last message: nil
State: %{live_query_tokens: #HashDict<[]>, opts: [host: "localhost", port: 2424, user: "admin", password: "admin", connection: {:db, "GratefulDeadConcerts"}], protocol_version: nil, queue: {[], []}, schema: nil, session_id: nil, socket: nil, tail: "", transaction_id: 1}`

OrientDB 2.1 introduces some index-related operations in the binary protocol, namely:

• REQUEST_INDEX_GET
• REQUEST_INDEX_PUT
• REQUEST_INDEX_REMOVE

For now, we're not implementing these operations in the driver because they're experimental as of now (at the time of writing, OrientDB is at version 2.1-rc-something so 2.1 isn't out yet). That said, all the functionalities available through these low-level operations can be achieved using regular SQL commands:

MarcoPolo.command(conn, "SELECT FROM index:myIndex WHERE key = 'foo'")

A suggestion: close all issues and pending PRs and make this repository read-only so folks cannot open new issues and PRs.

Hi @whatyouhide,

I was wondering if you could update us on the status of v0.3.0, since OrientDB 2.2 is expected by the end of the month? Are you planning to support #11?

Thanks for all the great work!

Not sure if an issue is the best place to ask this, but here it is:

Is this repo still being maintained? Are there future plans for it? I'm looking into using it for work, but I need to ensure that it's not going to go out of date in a couple months.

Is there any reason MarcoPolo doesn't use the native Elixir DateTime and Date types? Was it just done before they were added?

Would seem to be more convenient, and I'd be happy to PR to fix, just don't want to spend time on it if there was some other reason not to.

When upgrading OrientDB to 2.2.5 and trying to connect I get the following error:

** (Mix) Could not start application my_api: MyApi.start(:normal, []) returned an error: shutdown: failed to start child: MyApi.OrientdbPool
    ** (EXIT) shutdown: failed to start child: :orientdb_poolboy
        ** (EXIT) an exception was raised:
            ** (MatchError) no match of right hand side value: {:error, {:EXIT, {{:badarg, [{MarcoPolo.Protocol.Types, :encode, 1, [file: 'lib/marco_polo/protocol/types.ex', line: 48]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1088]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line: 1088]}, {MarcoPolo.Protocol, :encode_op, 2, [file: 'lib/marco_polo/protocol.ex', line: 31]}, {MarcoPolo.Connection.Auth, :authenticate, 1, [file: 'lib/marco_polo/connection/auth.ex', line: 48]}, {MarcoPolo.Connection, :connect, 2, [file: 'lib/marco_polo/connection.ex', line: 134]}, {Connection, :enter_connect, 5, [file: 'lib/connection.ex', line: 622]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}]}, {:gen_server, :call, [#PID<0.365.0>, :fetch_schema]}}}}
                src/poolboy.erl:275: :poolboy.new_worker/1
                src/poolboy.erl:296: :poolboy.prepopulate/3
                src/poolboy.erl:145: :poolboy.init/3
                (stdlib) gen_server.erl:328: :gen_server.init_it/6
                (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3

Edit: Just to clarify: I use poolboy to start the a pool of MarcoPolo processes. When trying to connect directly in the console, I get a timeout error:

iex(1)> {:ok, conn} = MarcoPolo.start_link(user: "root", password: "password", connection: {:db, "mydb_dev", :graph})
** (exit) exited in: :gen_server.call(#PID<0.612.0>, :fetch_schema)
    ** (EXIT) time out
    (stdlib) gen_server.erl:204: :gen_server.call/2
             lib/marco_polo/connection.ex:55: MarcoPolo.Connection.start_link/1
iex(1)> [error] GenServer #PID<0.612.0> terminating
** (stop) time out
Last message: nil
State: %{live_query_tokens: #HashDict<[]>, opts: [host: "localhost", port: 2424, user: "root", password: "password", connection: {:db, "mydb_dev", :graph}], protocol_version: 36, queue: {[], []}, schema: nil, session_id: nil, socket: #Port<0.22367>, tail: "", transaction_id: 1}
** (EXIT from #PID<0.610.0>) time out

@whatyouhide You referenced work on Ecto support for this driver in #19. Curious if that is still in the works or not? It is an effort I'd be happy to contribute to.

Fwiw, I'm currently using ex_orient (mainly for it's poolboy integration out of the box), but it doesn't offer any Ecto support.

As of now (around commit 4d22186), tree-based RidBags are not supported. When OrientDB returns a tree-based RidBag, MarcoPolo.Protocol.RecordSerialization.decode_type/2 fails with an error message saying that tree-based RidBags are not supported. The error message also suggests what to do to force OrientDB to always use embedded RidBags.

We have to support them :)

Hi,

Firstly, thanks for the awesome package. Made me look at some very interesting features of orientdb.

Could this package be used in a phoenix framework project, with configuration on config.exs ?

Thanks!

It should be straightforward: initiating the query is just a particular kind of REQUEST_COMMAND, so nothing new to handle that. It returns a response that the parser for REQUEST_COMMAND can parse so everything already works wrt initiating the query.

After that, OrientDB sends messages to the client every time a change happens in a given live query. Tokens are used to match which query changed. These messages start with <<3, ...>> so they're easy to identify as "live query messages", but the format is still undocumented and that's what's holding this feature off.

Sketch of the API

MarcoPolo.command(conn, "CREATE CLASS Language")

{:ok, token} = MarcoPolo.live_query(conn, "LIVE SELECT FROM Language", self())

# Through other means:
# INSERT INTO Language(name) VALUES ('Elixir', 'Erlang')

receive do
  {:orientdb_live_query, ^token, changes} ->
    # we still have to see what `changes` looks like
    changes
end
#=> changes

/cc @yelps

I was wondering if you could push a new release to hex. The 0.2.1 version is hard coded to the connection 1.0.0 release candidate, that is now released and this hard coded dependency is now incompatible with current projects...