elixir-mongo / mongodb Goto Github PK

Hi,

I want to suggest you to move BSON lib to a separate HEX package, since your implementation works much (more than 50%) faster and predictable than bson package that have 10k downloads right now.

At the moment I'm used to copy-paste your code and make some cleanup, since making DB driver as dependency to parse BSON would be an ugly solution.

I'm sure this is due to my inexperience, but I'm trying to find an example of setting up a stream from a cursor.

I can see a number flaws in my approach and figure there is likely a better way to do this.

def doc_stream(some_criteria) do
  Stream.resource(
    fn -> iteration end,
    fn(iteration) ->
      IO.puts "iteration: #{iteration}"
      data = Mongo.find(SomePool, "collection_name", %{"field" => some_criteria, limit: 1, skip: iteration)
      if Enum.empty?(data) do
        {:halt, iteration}
      else
        {[Enum.at(data, 0)], iteration + 1}
      end
    end,
    fn(iteration) -> iteration end
  )
end

doc_stream |> Enum.take(10)

My goal is to not put a ton of pressure on the mongo servers when trying to work through massive collections in batches. I'd like to utilize streams and async processes to work on small chunks and only pull the next n-number of documents when its time to use them.

My issue with what I have is that if I take 10, its going to fire 10 queries at the database I assume. That is probably not ideal.

But before I sink more time into this and reinvent the wheel, I figured one of you may have a better approach or maybe this is silly altogether?

Add a .drop() function to drop a collection.

We performed it manually using a run_command but if the collection does not exist, an uncaught exception occurs...

** (CaseClauseError) no case clause matching: %{"errmsg" => "ns not found", "ok" => 0.0}

As a guidepost, within our wrapper lib, we're handling like this presently..

def drop_collection(collection) do
try do
%{"ok" => 1.0} = Mongo.run_command(@pool, %{:drop => collection})
:ok
rescue
e in CaseClauseError ->
Logger.debug "#{inspect e}"
:error
end
end

The code below would throw a duplicate ID error, but the rescue doesn't work.
What's the right way to catch/rescue an operation so that I can log the message?

try do
  Mongo.insert_one(MongoPool, "articles",%{"_id" => "aaa"})
  Mongo.insert_one(MongoPool, "articles",%{"_id" => "aaa"})
rescue
  e in RuntimeError -> IO.inspect e
end

I'm sure I may be missing something here, but after an insert I get back a BSON.ObjectId like #BSON.ObjectId<1e0b239999a1014ed1b35a84>.

What is the easiest way to get a string representation of the actual _id for later use?

I've tried something like %BSON.ObjectId{value: object_id} = #BSON.ObjectId<1e0b239999a1014ed1b35a84>, but then I have a binary.

Feels like I'm missing something real simple here. Thanks!

Take a look at elrmongo

It seems there are intermittent failures in the test/mongo/connection_test.exs file, for the "auth wrong" scenario on line 31.

As an example travis run:
https://travis-ci.org/ericmj/mongodb/jobs/141147103

Is it possible this is the kind of message ordering issue that should be handled using special process tests, as described here:
http://blog.carbonfive.com/2016/06/28/special-processes-in-otp/ ?

BSON encoding/decoding of values does not pay attention to the endianness of the storage when doing binary pattern matching. All types which have a possible endianness should be serialized to little endian per the BSON spec.

Hi @ericmj. Thanks for this. I was wondering if you would be interested in a PR to support authentication via an Authentication Database? If so, would master be the correct branch to submit the PR to? I've noticed some tests are failing on master and that there is another branch that seems to be active. Thanks :)

I saw a previous issue that was the same as mine but the solution for them did not work for me (adding a name parameter to start_link). This is my first time using the library (version 2.0.0) and I followed the instructions in the README. I've got a MongoPool module:

defmodule MongoPool do
  use Mongo.Pool, name: __MODULE__, adapter: Mongo.Pool.Poolboy
end

and when the app tries to compile I get the following error:

** (CompileError) lib/intake_zen/mongo_pool.ex:2: module Mongo.Pool is not loaded and could not be found
    (elixir) expanding macro: Kernel.use/2
    lib/intake_zen/mongo_pool.ex:2: MongoPool (module)
    (elixir) lib/kernel/parallel_compiler.ex:116: anonymous fn/4 in Kernel.ParallelCompiler.spawn_compilers/1

I searched through the driver repo and can't find Mongo.Pool anywhere. Is this module no longer available?

distinct/find/insert_one/insert_many/delete_one/delete_many/replace_one/update_one/update_many/save_one/save_many including MongoCursors should support :database option to specify a database to talk with.

Hi @ericmj! I have been recently working with massive datasets and more or less complex aggregate functions to retrieve data, operations that sometimes last 2-3 minutes. I've set the HTTP timeout high but even with this option if my process lasts more than five secs the GenServer.call raises an exception because it is always called with the default timeout for GenServer operations(5000ms), do you think is worth to send the same time as in HTTP timeout for mongo? For the moment I have this in my local project:

GenServer.call(conn, {:find, coll, query, select, opts}, 60_000)

60_000 is a hardcoded limit I put only to try, and it worked.

Sort is made with atoms, and can't do this with them. I think they should remain as a string keys.

Mongo.find(MongoPool, %{}, [sort: [game.name: -1]])

Thanks 😃

Hello, I'm trying to connect a MongoDB version 3.2 but it seems it isn't using SCRAM authentication, how can I specify that in my connection options?

{:ok, _} = MongoPool.start_link(
            hostname: server,
            username: username,
            password: password,
            database: dbname
        )

** (Mix) Could not start application ejobs: exited in: JobsMonitor.Start.start(:normal, [])
    ** (EXIT) exited in: GenServer.call(#PID<0.149.0>, {:find, "bz_administrators", %{}, nil, [batch_size: 1000]}, 5000)
        ** (EXIT) %Mongo.Error{code: 18, message: "auth failed for 'admin': Authentication failed."}

Values passed as connection parameters are ok

How can I query for document based on ObjectID as string? The only way I could get it to work is
to use Mongo.Ecto.ObjectID.dump from Mongo.Ecto project, like this:

Mongo.find(MongoPool, "some_collection", %{_id: elem(Mongo.Ecto.ObjectID.dump("55e724bf44f4173c876919f9"),1) })

is there a way to do it and not depend on an external project?

When performing a sum aggregation and excluding _id from the final output I get an exception. The same operation can be completed from Mongo Client with out issue.

MongoDB shell version: 2.6.10

Mongo Client

db.www_test_com.aggregate( 
  [ 
    { $group: { _id: '', total: { $sum: "$bytes_sent" } } }, 
    { $project: { _id: 0, total: '$total' } } 
  ] 
);

{ "total" : 85000 }

Using Mongo.aggregate with project

resp = Mongo.aggregate(MongoPool, "www_test_com",
      [
        %{'$group': %{ '_id': false, 'total': %{ '$sum': "$bytes_sent" }}},
        %{'$project': %{ '_id': 0, 'total': '$total' }}
      ]
    ) |> Enum.to_list

** (EXIT) an exception was raised:
   ** (CaseClauseError) no case clause matching: {:ok, %Mongo.ReadResult{cursor_id: 0, docs: [%{"code" => 15992, "errmsg" => "exception: disallowed field type Array in object expression (at 'total')", "ok" => 0.0}], from: 0, num: 1}}

Using Mongo.aggregate without project

resp = Mongo.aggregate(MongoPool, "www_test_com",
      [
        %{'$group': %{ '_id': false, 'total': %{ '$sum': "$bytes_sent" }}}
      ]
    ) |> Enum.to_list

IO.inspect(resp)
[%{"_id" => false, "total" => 85000}]

Was writing some more documentation and noticed naming of "remove" from Mongo is named as "delete_*". The main Mongo API uses remove(), so I was looking for that in the code. Connection is using remove though, so not sure why its like this. Could just be something that's going to change later.

Summary: Why is Mongo using delete_* instead of remove?

I'm getting an error running queries against the oplog on a MongoDB replica set. MongoDB v 2.6.5 / Elixir 1.0.5.

An iex session that illustrates this:

-> % iex -S mix
Erlang/OTP 18 [erts-7.0.2] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.0.5) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> defmodule MongoPool do use Mongo.Pool, name: __MODULE__, adapter: Mongo.Pool.Poolboy end                        {:module, MongoPool,
 <<70, 79, 82, 49, 0, 0, 9, 80, 66, 69, 65, 77, 69, 120, 68, 99, 0, 0, 1, 111, 131, 104, 2, 100, 0, 14, 101, 108, 105, 120, 105, 114, 95, 100, 111, 99, 115, 95, 118, 49, 108, 0, 0, 0, 2, 104, 2, ...>>,
 :ok}
iex(2)> {:ok, _} = MongoPool.start_link(database: "local")
{:ok, #PID<0.134.0>}
iex(3)> Mongo.find(MongoPool, "oplog.rs", %{}) |> Enum.take(1)
** (exit) exited in: GenServer.call(#PID<0.150.0>, {:find, "oplog.rs", %{}, nil, [batch_size: 1000]}, 5000)
    ** (EXIT) an exception was raised:
        ** (FunctionClauseError) no function clause matching in Mongo.Connection.handle_info/2
            lib/mongo/connection.ex:346: Mongo.Connection.handle_info({:tcp, #Port<0.5509>, <<0, 3, 0, 0, 0, 82, 85, 0, 2, 49, 56, 48, 0, 3, 0, 0, 0, 82, 87, 0, 2, 49, 56, 49, 0, 3, 0, 0, 0, 82, 69, 0, 2, 49, 56, 50, 0, 3, 0, 0, 0, 66, 76, 0, 2, 49, 56, 51, ...>>}, {:ok, %{auth: [], database: "local", opts: [backoff: 1000, port: 27017, hostname: 'localhost'], queue: %{0 => %{command: :find, from: {#PID<0.127.0>, #Reference<0.0.3.911>}, params: nil}}, request_id: 1, socket: #Port<0.5509>, tail: <<84, 120, 47, 0, 85, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 138, 133, 8, 97, 208, 0, 0, 0, 0, 0, 0, 0, 232, 3, 0, 0, 242, 11, 0, 0, 17, 116, ...>>, timeout: 5000, wire_version: 2, write_concern: [w: 1]}})
            lib/connection.ex:798: Connection.handle_async/3
            (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
            (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
            (stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3

18:55:47.367 [error] GenServer #PID<0.150.0> terminating
Last message: {:tcp, #Port<0.5509>, <<0, 3, 0, 0, 0, 82, 85, 0, 2, 49, 56, 48, 0, 3, 0, 0, 0, 82, 87, 0, 2, 49, 56, 49, 0, 3, 0, 0, 0, 82, 69, 0, 2, 49, 56, 50, 0, 3, 0, 0, 0, 66, 76, 0, 2, 49, 56, 51, ...>>}
State: {:ok, %{auth: [], database: "local", opts: [backoff: 1000, port: 27017, hostname: 'localhost'], queue: %{0 => %{command: :find, from: {#PID<0.127.0>, #Reference<0.0.3.911>}, params: nil}}, request_id: 1, socket: #Port<0.5509>, tail: <<84, 120, 47, 0, 85, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 138, 133, 8, 97, 208, 0, 0, 0, 0, 0, 0, 0, 232, 3, 0, 0, 242, 11, 0, 0, 17, 116, ...>>, timeout: 5000, wire_version: 2, write_concern: [w: 1]}}
** (exit) an exception was raised:
    ** (FunctionClauseError) no function clause matching in Mongo.Connection.handle_info/2
        lib/mongo/connection.ex:346: Mongo.Connection.handle_info({:tcp, #Port<0.5509>, <<0, 3, 0, 0, 0, 82, 85, 0, 2, 49, 56, 48, 0, 3, 0, 0, 0, 82, 87, 0, 2, 49, 56, 49, 0, 3, 0, 0, 0, 82, 69, 0, 2, 49, 56, 50, 0, 3, 0, 0, 0, 66, 76, 0, 2, 49, 56, 51, ...>>}, {:ok, %{auth: [], database: "local", opts: [backoff: 1000, port: 27017, hostname: 'localhost'], queue: %{0 => %{command: :find, from: {#PID<0.127.0>, #Reference<0.0.3.911>}, params: nil}}, request_id: 1, socket: #Port<0.5509>, tail: <<84, 120, 47, 0, 85, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 138, 133, 8, 97, 208, 0, 0, 0, 0, 0, 0, 0, 232, 3, 0, 0, 242, 11, 0, 0, 17, 116, ...>>, timeout: 5000, wire_version: 2, write_concern: [w: 1]}})
        lib/connection.ex:798: Connection.handle_async/3
        (stdlib) gen_server.erl:615: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:681: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3
    (elixir) lib/gen_server.ex:356: GenServer.call/3
    (stdlib) timer.erl:181: :timer.tc/2
             lib/mongo/pool/poolboy.ex:34: Mongo.Pool.Poolboy.run/2
             lib/mongo/pool.ex:140: Mongo.Pool.run_with_log/5
             lib/mongo/cursor.ex:33: anonymous fn/6 in Enumerable.Mongo.Cursor.start_fun/6
    (elixir) lib/stream.ex:1014: anonymous fn/5 in Stream.resource/3
    (elixir) lib/enum.ex:1740: Enum.take/2

Would like the ability to also define the database when defining the pool definition itself, instead of specifying the database on start_link. Since the pool is directly tied to the database, this keeps it cleaner.

The existing approach is:

defmodule DatabasePool do
use Mongo.Pool, name: MODULE, adapter: Mongo.Pool.Poolboy
end

but then elsewhere having to perform a DatabasePool.start_link(database: "my_database").

Would also like the ability to do this instead...

defmodule DatabasePool do
use Mongo.Pool, name: MODULE, adapter: Mongo.Pool.Poolboy, database: "my_database"
end

DatabasePool.start_link

http://docs.mongodb.org/manual/reference/method/db.collection.findAndModify/

https://github.com/mongodb/specifications/blob/master/source/crud/crud.rst#find-and-modify

If you are just recursion processing results, and pass a blank result set, it crashes on this.

** (FunctionClauseError) no function clause matching in Mongo.many_docs/1
             (mongodb) lib/mongo.ex:449: Mongo.many_docs([])
             (mongodb) lib/mongo.ex:130: Mongo.insert_many/4
    (twitch_discovery) lib/index/stream.ex:2: TwitchDiscovery.Index.Stream.mongo_save_many/1
    (twitch_discovery) lib/index/stream.ex:2: TwitchDiscovery.Index.Stream.get_next/1
    (twitch_discovery) lib/index.ex:41: TwitchDiscovery.Index.index/0

Hi!
I'm trying to use the package in a project that I'm working on, but if I try to use DBConnection.Poolboy as specified in the instructions, I get the following error:

iex(11)> Mongo.find(:mongo, "products", %{}, limit: 20, pool: DBConnection.Poolboy) |> Enum.to_list
** (UndefinedFunctionError) function :invalid_message.exception/1 is undefined (module :invalid_message is not available)
                    :invalid_message.exception([])
    (db_connection) lib/db_connection.ex:925: DBConnection.checkout/2
    (db_connection) lib/db_connection.ex:741: DBConnection.run/3
    (db_connection) lib/db_connection.ex:1132: DBConnection.run_meter/3
    (db_connection) lib/db_connection.ex:636: DBConnection.execute/4
          (mongodb) lib/mongo.ex:386: Mongo.kill_cursors/3
           (elixir) lib/stream.ex:1129: Stream.do_resource/5
           (elixir) lib/enum.ex:1627: Enum.reduce/3
           (elixir) lib/enum.ex:2346: Enum.to_list/1

These are my deps:

* db_connection 1.1.0 (Hex package) (mix)
  locked at 1.1.0 (db_connection) b2b88db6
  ok
* mongodb 0.2.0 (Hex package) (mix)
  locked at 0.2.0 (mongodb) 95f85074
  ok
* poolboy 1.5.1 (Hex package) (rebar)
  locked at 1.5.1 (poolboy) 6b461639
  ok

And this is how I have added the Mongo worker to the main supervision tree:

defp children do
    [
        ...
        worker(Mongo, [mongo_config]),
        ...
    ]
  end

defp mongo_config do
    [
      name:     :mongo,
      pool:     DBConnection.Poolboy,
      username: System.get_env("MONGODB_USERNAME"),
      password: System.get_env("MONGODB_PASSWORD"),
      database: System.get_env("MONGODB_DBNAME"),
    ]
  end

The thing is that if I remove the pool from the config, and run the same query without the pool, it actually works. What might be going wrong?
Thanks in advance!

hi, is there a way to make BSON.ObjectId.decode!/1 available?, I'm passing a string as parameter

15:01:10.138 [error] #PID<0.287.0> running Estorage.Router terminated
Server: localhost:4001 (http)
Request: POST /uploadfile
** (exit) an exception was raised:
    ** (UndefinedFunctionError) function BSON.ObjectId.decode!/1 is undefined or private
        (mongodb) BSON.ObjectId.decode!("579a2daf189fe3491028a3c6")
        (estorage) lib/estorage/users.ex:7: Estorage.Users.getById/1
        (estorage) lib/estorage/manager.ex:8: Estorage.Manager.upload_file/5
        (estorage) lib/estorage/router.ex:18: anonymous fn/1 in Estorage.Router.do_match/4
        (estorage) lib/estorage/router.ex:2: Estorage.Router.plug_builder_call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

Hi there. Thanks for this great library! I have been struggling with a few other libraries that are basically dead. Everything works as expected until this issue came up. I can't figure out how to get past it, and using the code for find that you're using in the tests. I understand this is basically pre-alpha, but my requirements are limited at the moment.

  defmodule MongoPool do
    use Mongo.Pool, name: __MODULE__, adapter: Mongo.Pool.Poolboy
  end

collection = "broadcasts0"
Mongo.find(MongoPool, collection, %{} |> Enum.to_list

** (exit) an exception was raised:
    ** (Protocol.UndefinedError) protocol Enumerable not implemented for %Mongo.Cursor{coll: "broadcasts-0", opts: [], pool: MongoPool, query: %{}, select: nil}
        (elixir) lib/enum.ex:1: Enumerable.impl_for!/1
        (elixir) lib/enum.ex:112: Enumerable.reduce/3
        (elixir) lib/enum.ex:1265: Enum.reduce/3
        (elixir) lib/enum.ex:1807: Enum.to_list/1

Thanks 😃

Please, add a LICENSE file for this project.

BTW, congrats! 👍

This would be pretty cool. They also have a nice guide for it.

Elixir (and Erlang) doesn't natively support those special floating point values so if we see any such value from Mongo, we'll simply throw up an error.

        ** (FunctionClauseError) no function clause matching in BSON.Decoder.type/2
            (mongodb) lib/bson/decoder.ex:10: BSON.Decoder.type(1, <<0, 0, 0, 0, 0, 0, 248, 127>>)

It might be possible to binary match to detect those values and return our own special atoms as discussed here: https://groups.google.com/forum/#!topic/capnproto/T6jEFu38rtI

and here: kaos/ecapnp@a4e3221#diff-d28a6e8472db5dc7149ae3347d12566aR84

This is for the emj-dbconn branch

I believe my recent PR (#74) broke something. When trying to do simple queries, I am getting a protocol error.

However, if I execute the commands fast enough (copy / paste into iex) the query works properly.

Let me know if there is anything else I can do to debug this.

12:44:41 (emj-dbconn) mongodb$ iex -S mix
Erlang/OTP 19 [erts-8.0] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false]

Interactive Elixir (1.3.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> {:ok, pid} = Mongo.start_link(database: "some_database")
{:ok, #PID<0.154.0>}
iex(2)> c = Mongo.find(pid, "users", %{})
%Mongo.Cursor{coll: "users", conn: #PID<0.154.0>, opts: [], query: %{},
 select: nil}
iex(3)> Enum.to_list(c)
** (Mongo.Error) tcp recv: invalid argument - :einval

12:45:27.973 [info]  Mongo.Protocol (#PID<0.154.0>) missed message: {:tcp, #Port<0.5043>, <<167, 25, 7, 0, 1, 245, 4, 0, 0, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 143, 1, 0, 0, 50, 24, 0, 0, 7, 95, 105, 100, 0, 82, 129, 95, ...>>}

12:45:27.973 [error] Mongo.Protocol (#PID<0.154.0>) disconnected: ** (Mongo.Error) tcp recv: invalid argument - :einval
    (mongodb) lib/mongo/cursor.ex:40: anonymous fn/6 in Enumerable.Mongo.Cursor.start_fun/6
     (elixir) lib/stream.ex:1121: anonymous fn/5 in Stream.resource/3
     (elixir) lib/enum.ex:1626: Enum.reduce/3
     (elixir) lib/enum.ex:2345: Enum.to_list/1
** (EXIT from #PID<0.152.0>) an exception was raised:
    ** (RuntimeError) disconnect/2 not implemented
        (mongodb) lib/db_connection.ex:240: Mongo.Protocol.disconnect/2
        (db_connection) lib/db_connection/connection.ex:178: DBConnection.Connection.disconnect/2
        (connection) lib/connection.ex:767: Connection.disconnect/3
        (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

Interactive Elixir (1.3.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>
12:45:27.978 [error] GenServer #PID<0.154.0> terminating
** (RuntimeError) disconnect/2 not implemented
    (mongodb) lib/db_connection.ex:240: Mongo.Protocol.disconnect/2
    (db_connection) lib/db_connection/connection.ex:178: DBConnection.Connection.disconnect/2
    (connection) lib/connection.ex:767: Connection.disconnect/3
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:disconnect, #Reference<0.0.6.118>, %Mongo.Error{code: nil, message: "tcp recv: invalid argument - :einval"}, %{database: "some_database", request_id: 0, socket: #Port<0.5043>, timeout: 5000, wire_version: 4, write_concern: %{w: 1}}}}
State: {Mongo.Protocol, %{database: "some_database", request_id: 0, socket: #Port<0.5043>, timeout: 5000, wire_version: 4, write_concern: %{w: 1}}}

Hi, on one of my collections I get the following error:

 ** (FunctionClauseError) no function clause matching in BSON.Decoder.type/2
            (mongodb) lib/bson/decoder.ex:10: BSON.Decoder.type(1, <<1, 0, 0, 0, 0, 0, 240, 127, 1, 49, 0, 1, 0, 0, 0, 0, 0, 240, 127>>)

 lib/bson/decoder.ex:10: BSON.Decoder.type(1, <<0, 0, 0, 0, 0, 0, 240, 127, 1, 112, 114, 111, 106, 101, 99, 116, 101, 100, 69, 84, 99, 70, 114, 111, 109, 104, 105, 115, 116, 111, 114, 121, 0, 170, 241, 210, 77, 98, 16, 201, 63, 2, 67, 73, 77, 73, 83, 83, 116, 97, ...>>)

I am pulling from master since it seems a similar issue was resolved recently, but it still breaks. I'm connecting to an existing database so I need to be able to access the values. Is there a workaround for this type of issue before it is fixed?

I think it is caused by a 2 element array that has nans with typedouble.

Thanks.

So, if you create a tailable cursor, you can use the cursor as a Stream; however, the functionality that a tailable cursor adds (allowing you to see newly inserted documents for a cursor as they are added) doesn't really support streams. We'd need some way to have a supervised process that sends new results to another process as they are available. Here's the code for how RethinkDB handles this:

https://github.com/hamiltop/rethinkdb_changefeed

@ericmj I think that adding replica set support will really boost the elixir / phoenix community.
Using mongodb in production without replica set it's not really possible. My company has a project that I really would like to migrate to phoenix and it's impossible without replica set support. I think there is a lot of developers in the same position. If you need help I can try and add the replica set support myself. If you already working on it I would like to help anyway I can.

Say I have an incremental id, in elixir i set "_id"=> "incrId()", "timest"=>"db.eval("Date()")"

            Mongo.insert_one(MongoPool, "test", %{
                    "_id"=> "incrId()",
                    "timest"=> "db.eval(\"Date()\")"
                })

This makes the record saved without evaluating the function.Neither way works. Using the commandline for mongo this behavior does not happen as the functions get evaluated as they are inserted.

When I try to do a basic search as per the examples, I get the above error.

When I follow the instructions on the Readme I get an error doing an insert:

iex(4)> Mongo.insert_one(MongoPool, "crawl_report", %{"field" => 10})
** (ArgumentError) argument error
(stdlib) :ets.lookup_element(Mongo.IdServer, :machineprocid, 2)
lib/mongo/id_server.ex:52: Mongo.IdServer.new/0
lib/mongo/connection.ex:134: Mongo.Connection.add_id/1
lib/mongo/connection.ex:102: Mongo.Connection.assign_ids/1
lib/mongo/connection.ex:79: Mongo.Connection.insert/4
(stdlib) timer.erl:181: :timer.tc/2
lib/mongo/pool/poolboy.ex:36: Mongo.Pool.Poolboy.run/2
lib/mongo/pool.ex:142: Mongo.Pool.run_with_log/5

However if I manually run Mongo.IdServer.start_link the insert works:

iex(3)> Mongo.insert_one(MongoPool, "crawl_report", %{"field" => 10})
** (ArgumentError) argument error
(stdlib) :ets.lookup_element(Mongo.IdServer, :machineprocid, 2)
lib/mongo/id_server.ex:52: Mongo.IdServer.new/0
lib/mongo/connection.ex:134: Mongo.Connection.add_id/1
lib/mongo/connection.ex:102: Mongo.Connection.assign_ids/1
lib/mongo/connection.ex:79: Mongo.Connection.insert/4
(stdlib) timer.erl:181: :timer.tc/2
lib/mongo/pool/poolboy.ex:36: Mongo.Pool.Poolboy.run/2
lib/mongo/pool.ex:142: Mongo.Pool.run_with_log/5
iex(3)> Mongo.IdServer.start_link
{:ok, #PID<0.203.0>}
iex(4)> Mongo.insert_one(MongoPool, "crawl_report", %{"field" => 10})
{:ok,
%Mongo.InsertOneResult{inserted_id: #BSON.ObjectId<1dfbc07783152906278464a1>}}

Is this expected behavior or is something wrong?

I see this one is started about a year ago while checkiz/elixir-mongo is a lot older. What should one chose when starting fresh and for what reason? Thanks for your feedback :-)

Currently I'm migrating a lib which using elixir-mongo as dependency which it doesn't require a database when creating a connection(which should be much more nature as well). The reason why I'm doing so is because elixir-mongo did purge all values's key into atom which might cause a huge memory leakage if it is running in a long term,and looks like their project is unmaintained for a while as well.

Everything looks fine for mongoldb as well, but I can't find anyway to create a clean connection to the database both with Mongo.Connection or MongoPool, is there any way I can circumvent with this restricts.

And also I did notice all find/update or db actions are taking collection as the first argument, so from my understanding this is a designed pattern, but indeed I need to maintain a single connection and talks to multiple db in the same time, is there anyway I can use it except for fork a new version?

Would be useful to encode to JSON, e.g. map of strings:

BSON.encode_json(doc)
%{"_id" => "559c3c676b92c9bf5c0898fd", "name" => "Travis"}

I'm pretty new to Elixir and this is above my level still, but when I can do it I'll make a PR if there isn't already.

If mongo is inaccessible (offline), an attempt to use any Mongo functions results in an exception.

The Connection is returning a {:error, :closed} upstream, but the local function handler tries to raise all errors of the pattern {:error, e} causing the exception.

The Connection error should be converted to some kind of %Mongo.Error{} instead, so the raise function consistently reports these exceptions.

After I insert some data I get this:

%Mongo.InsertOneResult{inserted_id: #BSON.ObjectId<57dc602dfa2d310e03f95be8>}

That's great, but I have no idea what to do with this BSON.Objectid I would like to get back maybe a map of all the data I just inserted or the Mongo ID so I can look up the record I just inserted.

Can anyone help me figure out how to do this?

I want to use this on Heroku where I am provided a full connection string as an environment variable. Is there any way to connect with a string in the following format?

mongodb://heroku_****:****@***.mlab.com:17545,***.mlab.com:17545/heroku_*****?replicaSet=rs-****

You can treat a cursor as an Elixir Stream, but this should really be documented. I'll make a PR for this soon.

Is pool still available? Can't seem to use it with v2.1.. And if it's not used anymore, what is the alternative?

If I run a Mongo.find query on a large mongo collection in a busy server with:

cursor = Mongo.find(MongoPool, "userdata", %{state: "ready"}, batch_size: 1000)

After a few hours of running I always get a timeout into :get_more:

** (exit) exited in: GenServer.call(#PID<0.169.0>, {:get_more, "userdata", 17595059170, [batch_size: 1000]}, 5000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:564: GenServer.call/3
    (stdlib) timer.erl:181: :timer.tc/2
    lib/mongo/pool/poolboy.ex:36: Mongo.Pool.Poolboy.run/2
    lib/mongo/pool.ex:142: Mongo.Pool.run_with_log/5
    lib/mongo/cursor.ex:65: anonymous fn/3 in Enumerable.Mongo.Cursor.next_fun/2
    (elixir) lib/stream.ex:1099: Stream.do_resource/5
    (elixir) lib/enum.ex:1477: Enum.reduce/3
    (elixir) lib/enum.ex:1092: Enum.map/2

The :timeout connection parameter is not used for :get_more so the only way to fix it is to manually edit this line: https://github.com/ericmj/mongodb/blob/943f6bcefff916e2e8b572fad1c9d79d55235490/lib/mongo/connection.ex#L64

and change it to:

GenServer.call(conn, {:get_more, coll, cursor_id, opts}, 50000)

With 50000 everything works fine and I no longer get any timeouts. So I think the :timeout connection parameter should also be use here or maybe create a new parameter.

Hi there, I'm an elixir noob trying to insert a date into mongo, but I can't figure out to to create the BSON.DateTime object properly. I'm currently doing this:

utc = :calendar.datetime_to_gregorian_seconds(:calendar.universal_time)
%BSON.DateTime{utc: get_utc_timestamp}

When it saves to mongo, it shows up as 1972-01-07....

What's the correct way to create a DateTime for now?

Hi,
I'd like to query the connected database to get a list of collections?
Can you give an example of how to do that?
Thanks
-John

I get an error when the BSON decoder tries to decode the machine identifier bytes of an ObjectId.

** (MatchError) no match of right hand side value: <<111, 230, 80, 74, 154, 74, 223, 8>>
stacktrace:
  (mongodb) lib/bson/decoder.ex:98: BSON.Decoder.document/1
  (mongodb) lib/bson/decoder.ex:6: BSON.Decoder.decode/1
  test/controllers/user_controller_test.exs:27

The original binary was <<86, 206, 224, 104, 111, 230, 80, 74, 154, 74, 223, 8>>