mongodb / bson-rust Goto Github PK

The ordered::OrderedDocument is a temporary solution while the linked-hash-map crate can only be compiled with nightly. But now, linked-hash-map can be compiled with the latest stable rustc v1.7 (contain-rs/linked-hash-map#28).

So I suggest that we could migrate it to linked-hash-map.

I just noticed that there is no license file in the repo. Maybe you could add the MIT license or something similar?

Although the BSON spec says that symbols (which start with \x0E) are deprecated, Mongo itself will happily encode and store symbols. Unfortunately, that means bson-rs should also support symbols.

I am getting the error "cannot find macro 'doc!' in this scope". I have added the extern crate

 #[use_macro(bson, doc)]
 extern crate bson;

 use bson::{Bson, Document, encode_document, decode_document};

fn get_user_email(self: &Self, email: Value) {
    let coll = self.mongo.db("tnv").collection("person");
    let doc = doc! { "email" => email };
    let cursor = coll.find(doc, None);
    println!("{:?}", cursor );
}

error: cannot find macro `doc!` in this scope
  --> src/main.rs:92:19
   |

92 | let doc = doc! { "email" => email };
| ^^^

error: aborting due to previous error(s)

I know this is probably something I am doing, but I am lost for a fix.

main.txt

Could you bump the version? :-)

Hello!

I am trying to create an OrderedDocument from a bson::Bson, but it already contains keys and values, so I'd just like to be able to insert it as is without having to specify the keys. Is there a good way of doing that?

value is dynamic and is always of type Bson. Example:

{
   "city": "Berlin",
   "street": "Oranienstraße 123",
   "postal_code": 10945
}

Create a document with Document::new() and then .insert() ?

let mut subdoc = Document::new();
subdoc.insert(None, value) // Edited to add 'None'
generate_schema_from_doc(subdoc)

or do that with doc! macro ?:

let mut subdoc = doc! { value };
generate_schema_from_doc(subdoc)

Thank you in advance!

Every crate is a Rust crate, so this should be named bson. It's also available on crates.io.

Could you please bump the version? Then we can use the latest ergonomics improvements in combination with the Mongo driver.

It seems to work 2 out of 3 times. It fails with:

thread 'oid::count_is_big_endian'
panicked at 'assertion failed: `(left == right)` (left: `17`, right: `0`)', src/oid.rs:333

I'm currently trying to port an existing microservice from java to rust to create a proof of concept. One of the thing is that in my current java microservice, the deserializer always forcefully coerce the type to whatever the java class is capable of holding, while it throws an error in rust

For example

struct definition

#[derive(Deserialize, Serialize, Debug)]
#[serde(rename_all = "camelCase")]
pub struct Test {
    #[serde(rename = "_id")]
    test_id : i32,
    amount : i32
}

bson data

{ 
    "_id" : NumberInt(12345), 
    "amount" : 1.0
}

will give me a

BsonSerializationError(InvalidType("i32"))

Is there a way we can enable/implement a lax serialization on bson-rs ?

According to BSON 1.1 Spec, it introduced a new type decimal128.

But currently Rust doesn't support d128 type natively.

Related links

1. d128 crate: https://github.com/alkis/decimal
2. Official support RFC: rust-num/num#8

Hi,

I'm implementing an auth mechanism based on bson and sodiumoxide, and I'm having trouble decoding the challenge. Sodiumoxide treats plaintexts as a &[u8], so I'd like to use a binary string instead of a Unicode string for the challenge, but I can't seem to get bson to decode it correctly.

Code

With a "standard" string, everything works fine:

#[derive(Serialize, Deserialize, Debug)]
pub struct AuthChallenge {
    pub challenge: String,
}

// [...]

    let challenge: AuthChallenge = bson::from_bson(
        bson::Bson::Document(
            bson::decode_document(
                &mut Cursor::new(&message.payload[..])
            ).chain_err(|| "Could not decode bson")?
        )
    ).expect("Decoding failed");
    info!("Challenge: {:?}", challenge.challenge);

This yields:

2018-03-08T08:22:04+01:00 - INFO - Received message ReqAuthentication (payload: [53, 0, 0, 0, 2, 99, 104, 97, 108, 108, 101, 110, 103, 101, 0, 33, 0, 0, 0, 49, 56, 55, 54, 50, 98, 57, 56, 98, 55, 99, 51, 52, 99, 50, 53, 98, 102, 57, 100, 99, 51, 49, 53, 52, 101, 52, 97, 53, 99, 97, 51, 0, 0])
2018-03-08T08:22:04+01:00 - INFO - Challenge: "18762b98b7c34c25bf9dc3154e4a5ca3"

However, if I changeString to Vec<u8> in the struct and change the server side to send a binary string (5) instead of a standard string (2), I get this:

2018-03-08T08:28:12+01:00 - INFO - Received message ReqAuthentication (payload: [53, 0, 0, 0, 5, 99, 104, 97, 108, 108, 101, 110, 103, 101, 0, 32, 0, 0, 0, 0, 54, 55, 98, 54, 100, 53, 50, 99, 50, 101, 48, 51, 52, 52, 56, 49, 98, 52, 57, 101, 102, 51, 56, 56, 101, 100, 100, 54, 51, 98, 50, 102, 0])
thread 'main' panicked at 'Decoding failed: InvalidType("a sequence")', /checkout/src/libcore/result.rs:906:4

I'm also having trouble encoding the signature, because to_bson would always complain that there are no unsigned types in bson, so I ended up doing the encoding manually:

    bson::Bson::Binary(
        bson::spec::BinarySubtype::Generic,
        Vec::from(&signature[..])
    )

Am I doing it wrong, or does bson not currently support binary strings correctly? Can I help in fixing it somehow?

the last 0.2 version, this function return lowercase hex, and the mongodb shell use lowsercase too, In order to unify the style

maybe modify to:

hex::encode(&self.id).to_lowercase()

Currently we only implement From<Array> and From<Document>; however, we should be able to implement the more generalized FromIterator<Bson> and FromIterator<(String, Bson)> traits for these two objects.

It would be nice to have a suitable Encoder/Decoder (or Serializer/Deserializer) implementation for Bson type in order to automatically serialize serializable types to BSON.

Hi, I think I found a problem there.

#![feature(proc_macro)]

extern crate serde;
#[macro_use]
extern crate serde_derive;

extern crate bson;

#[derive(Serialize,Deserialize, Debug, PartialEq)]
pub enum E1 { A(u32) }

#[test]
fn e1_bson() {
    let e = E1::A(12);
    let bson = bson::to_bson(&e).unwrap();
    let after:E1 = bson::from_bson(bson.clone()).unwrap();
    assert_eq!(e, after);
}

This will fail on the deser step with:

thread 'e1_bson' panicked at 'called `Result::unwrap()` on an `Err` value: Unknown("Invalid type. Expected `Seq`")', ../src/libcore/result.rs:799

I can probably hunt it down and offer a fix. I just noticed that bson-rs/serde-tests are kinda broken, but maybe it's a temporary thing with serde being so much in flux. (And also that they are not run by travis).

There is a lot of useful information at https://docs.rs/bson/0.3.2/bson/ but there are no links to it from the readme or crates.io. New users don't know where to get started.

cc @sssilver

I want to convert JSON to BSON, but I stuck on integers.

extern crate bson;
extern crate serde_json;

use bson::Document;

fn main() {
    let _doc_1: Document = serde_json::from_str("{\"a\": -1}").unwrap();

    let _doc_2: Document = serde_json::from_str("{\"a\": 1}").unwrap();
}

The code above panics at line 11 because bson seems to consider 1 as an unsigned integer. Is there a way to deserialize positive integers to a signed integer instead of an unsigned integer?

This came up from some fuzz testing of the crate. I don't know bson's wire protocol but from the output of bsondump it looks like the issue is it declares an illegal BSONSize and the crate still tries to allocate it and crashes.

The offending binary is: [0x06, 0xcc, 0xf9, 0x0a, 0x05, 0x00, 0x00, 0x03, 0x00, 0xff, 0xff]

I've been using mongo_driver that uses bson to import some data that I've exported with mongoexport (so in Strict mode). First I had problems with date parsing (#107), then I've noticed that normal numbers are needlessly converted to NumberLong version.

I might look into it tomorrow, but if anyone has any guesses where to look, it would help a lot.

BTW. I'm either confused, or the code is incorrectly calling "Strict Mode" , "the extended" one. From what I understand the "Extended JSON" is the one that MongoDB uses everywhere with ObjectID etc. It's extended because it's nor really correct JSON per-se. The one that converts these fancy type-wrapper to proper JSON, is "Strict Mode". So from_extended_document should probably be called from_strict_mode_document.

👋

After using the new serde json! macro, it feels a little smarter about delineating/parsing keys and values. Improving our macro definition to allow more arbitrary values would be 👌

doc!{ "files_id" => (self.doc.id.clone()) }
json!{ "files_id": self.doc.id.clone() }

doc!{ "success" => (code == 200) }
json!{ "success": code == 200 }

bson!(["trailing", "commas", "are", "bad"])
json!(["trailing", "commas", "are", "okay",])

We might also consider moving syntax over from => to : (or allowing both of them to co-exist, if possible, for backwards-compatibility). Our initial reason for using => was that macros couldn't use :, but this doesn't seem to be the case anymore.

I'm cross-posting this here from mongodb-labs/mongo-rust-driver-prototype#262 in case it's more a BSON issue

I'm trying to convert a service from Java/Kotlin to Rust, but I'm having issues with some "complex" aggregation steps.

I've been unable to find how to do it on both this repo and the bson-rs repo. Can someone help me?

How can I translate something like the below into a Rust aggregation stage?

const aggregationStages = [{
  "$match": {
    "_id": {
      "$in": [ObjectId("5b4289600000000000000000"), ObjectId("5b1afc600000000000000000")]
    }
  },
  "$project": {
    "_id": 1,
    "field": 1,
    "hours": [
      {
        "starts": {
          "$arrayElemAt": ["$times.start", 0]
        }
      }, 
      {
        "ends": {
          "$arrayElemAt": ["$times.end", -1]
        }
      }
    ]
  }
}]

I suppose I don't know how to convert arrays into valid Bson documents.

I tried the following, but have <bson macros> errors.

// get the colelction from the db
let coll = client.db("mydb").collection("places");
let mut aggregation_stages: Vec<bson::Document> = vec![];
let mut aggregation_options = mongodb::coll::options::AggregateOptions::new();
aggregation_options.allow_disk_use = Some(true);
aggregation_options.batch_size = 1000;

let object_ids = ["5b4289600000000000000000", "5b1afc600000000000000000"];
let mut ids: Vec<bson::oid::ObjectId> = vec![];

  for x in object_ids.iter() {
    ids.push(bson::oid::ObjectId::with_string(x).unwrap())
  }

    aggregation_stages.push(doc! { "$match": doc! { "_id": doc! { "$in": ids} }});aggregation_stages.push(doc! {"$project": doc! {
  "_id" : 1,
  "other_field" : 1,
  "hours": [
    doc! { "starts" : doc! {"$arrayElemAt" : ["$times.start", 0]} },
    doc! { "ends" : doc! {"$arrayElemAt" : ["$times.end", -1]} }
  ]
}});

I'm getting the below error:

error[E0277]: the trait bound `bson::Bson: std::convert::From<std::vec::Vec<bson::oid::ObjectId>>` is not satisfied
   --> src\main.rs:111:60
    |
111 |     aggregation_stages.push(doc! { "$match": doc! { "_id": doc! { "$in": ids} }});
    |                                                            ^^^^^^^^^^^^^^^^^^ the trait `std::convert::From<std::vec::Vec<bson::oid::ObjectId>>` is not implemented for `bson::Bson`
    |
    = help: the following implementations were found:
              <bson::Bson as std::convert::From<i32>>
              <bson::Bson as std::convert::From<f32>>
              <bson::Bson as std::convert::From<std::vec::Vec<bson::Bson>>>
              <bson::Bson as std::convert::From<(bson::spec::BinarySubtype, std::vec::Vec<u8>)>>
            and 16 others
    = note: required by `std::convert::From::from`
    = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

Hello. I have struct

pub struct Award{
    pub name:String,
    pub icon:Uuid,
    pub description:String,
    pub id:String,
}

And I write it to mongodb

                let award=Award{
                //id:id as u32,
                name:String::from(award_name),
                icon:icon,
                description:description,
                id:id .to_string(),
            };

            let serialized_award = bson::to_bson(&award)?;

            let find_filter=doc! {
                "_id" => user_id
            };

            let update_doc=doc! {//All right?
                "$push" => { "awards" => serialized_award }
            };

            mongo_users.find_one_and_update( find_filter ,update_doc, None );//TODO:None

And later I read:

let doc=mongo_users.find_one(Some(find_filter),Some(find_option))?;

        println!("{:?}",doc);

        match doc {
            Some( doc ) => {
               match doc.get("awards") {
                Some(&Bson::Array(ref mongo_awards)) => {

                let mut awards=Vec::with_capacity(mongo_awards.len());
                for mongo_award in mongo_awards.iter() {
                    match *mongo_award {
                        Bson::Document( ref doc ) => {
                            let award:Award = bson::from_bson( bson::Bson::Document(doc.clone()) ).unwrap();//NOTE : problem, and I DO NOT WANT CLONE document!

Part of mongo document

"awards": Array([Document({ name: "abc", icon: "a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11", description: "def", id: 1 })])

If id of struct Award is String, all works, but if id is u32 or usize I get InvalidType("u32")
Note that Display trait for this error writes only usize,u32 without any description like InvalidType:usize

UPD. Struct initialization in exemple gets &'static str instead String. This code do not works!

#[derive(Serialize, Deserialize, Debug)]
        pub struct Person {
            #[serde(rename = "_id")]  // Use MongoDB's special primary key field name when serializing
            pub id: String,
            pub name: String,
            pub age: u32
        }

        let person = Person {
            id: "12345".to_string(),
            name: "Emma".to_string(),
            age: 3
        };

        let serialized_person = bson::to_bson(&person)?;  // Serialize

        let person2:Person = bson::from_bson(serialized_person).unwrap();
}

Cargo.toml

serde="1.0.2"
serde_derive="*"
bincode="*"
redis = "*"
cdrs = "1.0.0-rc.1"
bson = ">=0.7.0"
mongodb = ">=0.2.7"

i32,i64 works. I guess, bson hates unsigned types.

Check out the release notes here: https://github.com/serde-rs/serde/releases/tag/v0.9.0

As per serde's conventions, a newtype struct or variant is supposed to be treated as an insignificant wrapper around the inner type. bson-rs does not respect this and will serialize newtype structs as 1-element arrays.

Why convert a Vec<u8> not in a Bson::Binary?

#[test]
fn test_byte_array() {
	
	#[derive(Serialize, Deserialize, PartialEq, Debug)]
    struct Foo {
        a: Vec<u8>,
    }
	
	let foo = Foo { a: vec![100u8; 400]};
	
	let e = Encoder::new();
    let r = match foo.serialize(e) {
        Ok(b) => b,
        Err(e) => panic!("Failed to serialize: {}", e),
    };
    
    println!("result {:?}", r);
}

Result in the error:
thread 'test_byte_array' panicked at 'Failed to serialize: BSON does not support unsigned type', test.rs:88:19
I would expect a Bson::Binary element to be created.

When answering questions about the BSON library on Stack Overflow, it's useful to be able to link to the code corresponding to a given version without having to divine the SHA from the history. In the absence of a CHANGELOG (or even in addition to one), this helps your users understand what has changed between versions and allows people to figure out if a commit is present in a given release.

bson 0.1.2 has a 161 MB ./db directory in its cargo package source. Consider releasing a new clean version.

This is a reminder that cargo includes all non-ignored files in your working directory when you publish — look at git status before you publish.

(I downloaded all crates.io crates and I started to grep for junk)

from_bson failed when some field is i32

Like serde_json, add support for:

format!("{:#}", bson);

[
    Object(
        {
            "contributors": Array(
                [
                    Object(
                        {
                            "date": String(
                                "2012-03-28"
                            ),
                            "role": String(
                                "author"
                            )
                        }
                    ),
...

More info about pretty-printed debug: https://www.reddit.com/r/rust/comments/3ceaui/psa_produces_prettyprinted_debug_output/

serde_json implementation and issue.

I can't work with https://serde.rs/custom-date-format.html.

Hi, I want to encode a HashMap to bson so that I can store it in MongoDB. My HashMap type is HashMap<u64, usize>. When I ran the code, it panic and told me that InvalidMapKeyType(FloatingPoint(....)). Can't I use this method to encode HashMap in such type?
Thanks in advance!

because of the mongodb use bson to store data, and the users just show and store the data, there is no need the json as middleware if bson support json string

I am trying to insert and find sodiumoxide cryptographic material in to a mongo database. Right now I encoding and decoding the crypto material in to base64. When I read a document and try to convert it to a rust structure, I have three unwrap() for each item.

I created from method, but the problem with this is I need to return Result<TmpLogin, Error>
impl From for TmpLogin {

Nonce::from_slice(&decode(doc.get_str("nonce").unwrap()).unwrap()).unwrap(),

What is the code snippet that would
let non = box_::gen_nonce();
let mut buf = Vec::new();
non.serialize(&mut rmp_serde::Serializer::new(&mut buf)).unwrap();

let src = (BinarySubtype::Generic, buf);
let tst = bson::????????(&src);

then later doc! { "nonce" => tst }
I am new at rust so any help would be appreciated

att as the title

see bson.rs: 270

is there support for GeoObjects? https://docs.mongodb.com/manual/reference/geojson/#point

I am trying to write a rust structure to a mongodb. One of the fields is an Option I have written a StructA to Document, but now I am trying the StructB to document function.

Is there a macro or something that supports Option?
Bson supports both Null and Document so it should be able to be done.

struct StructA {
    name: String,
    age:  i32,
    happy: bool,
}
struct StructB {
   atype: String,
   thinga: Option<StructA>,
}

impl From<StructB> from Document {
    fn from(val: StructB) -> Document {
        let doc = doc! {
            "atype" => (&val.atype),
            "thinga" => (???????),
       };
       doc
   }
}

After issues were found with the boolean and datetime encoding, it would be beneficial to add tests to ensure that all bson objects are encoded and decoded correctly, and to catch any changes that may break it.

I'm trying to make sense of the following error:

expected struct bson::ordered::OrderedDocument, found a different struct bson::ordered::OrderedDocument
note: expected type bson::ordered::OrderedDocument (struct bson::ordered::OrderedDocument)
found type bson::ordered::OrderedDocument (struct bson::ordered::OrderedDocument)
note: Perhaps two different versions of crate bson are being used?

Has the struct changed between versions?

From the version 0.7.0 to 0.7.1, chrono has been updated to a new minor version (from 0.2 to 0.3). Please stop doing this, it creates conflicts with other dependencies.

Thanks !

What would be the best way to convert a bson to a document that is suitable for inserting into the database?

Is there a specific reason ordered module is private?

This makes convenient macros such as try! unusable.

For example:

use bson::Document;

fn set_and_get_age() -> Result<i32, MyCustomError> {
  let mut doc = Document::new();
  doc.insert("age".to_string(), 32 as i32);

  let value = try!(doc.get_i32("age"));
  Ok(value)
}

This code does not compile because MyCustomError does not implement From<bson::ordered::ValueAccessError>. You can't provide a custom impl for From, because the module is private.

Found the following:

• "thread '' panicked at 'No such local time'"
  From: chrono-0.2.25/src/offset/mod.rs:151 via src/decoder/mod.rs:172
• "thread '' panicked at 'attempt to multiply with overflow'" - src/decoder/mod.rs:172
• "thread '' panicked at 'attempt to subtract with overflow'" src/decoder/mod.rs:45
• "AddressSanitizer failed to allocate 0xffffffff93000000 bytes" (whatever that means in real life)

Full logs: https://gist.github.com/killercup/5e8623e0d8b0fe9868b45eb223ef51d8 (See last few lines for inputs used, in bytes or base64)

See rust-fuzz/targets#51 for sources, I ran it with

$ env ASAN_OPTIONS="detect_odr_violation=0 allocator_may_return_null=1" ./run-fuzzer.sh bson read_bson

cc rust-fuzz/targets#39

Hello, Is it possible for bson-rs to gives the specific field in which the decoding fails ?

For example:

#[derive(Deserialize, Serialize, Debug)]
#[serde(rename_all = "camelCase")]
pub struct Test {
    #[serde(rename = "_id")]
    test_id : i32,

    amount : i32
}

doing

let object = bson::from_bson(bson::Bson::from_extended_document(doc_result?))?;

on a json of

{ 
    "_id" : NumberInt(12345), 
    "amount" : "asd"
}

will result in

BsonSerializationError(InvalidType("i32"))

Without specifying which field contains the invalid type

Now that custom derives are stable, rustc-serialize is deprecated in favor of serde. Rustc-serialize has not been actively developed for a long time, which means if anyone needs rustc-serialize support for some reason they can always use an old version of the bson crate - no need for us to keep the code around. Cleaning this up would remove a stumbling block for new users trying to use this crate with modern Rust.

What do you think about encoding u32 as i64?
I currently have the issue that I basically generate my Rust code from protobuf definitions, which I then want to decode and encode using bson. Now changing all protobuf defintions to i32 just because bson does not support unsigned ints does not seem really reasonable to me. I know this does not cover the u64, downside I guess is here the inconsistency.

fn serialize_u32(self, value: u32) -> EncoderResult<Bson> {
        Ok(Bson::I64(value as i64))
}

Hey!

Thanks for your great work on this crate. Could we get a release for the changes you made for serde 0.8 ?

Thanks :)