variety / variety Goto Github PK

Currently does not work with mongos
http://docs.mongodb.org/manual/reference/program/mongos/

Setting slaveOK() in the script doesn't resolve the problem.

hi

tell me you know how I can include variety.js in mongodb to use it from the mongo shell

When you use limit = * the percentage seems to be based on the number you enter for the limit even when the results yield less results than the limit number. I believe percentage should not be based on the number you are looking for but rather the number which is returned.

This is a sample command I use normally to connect to our remote test mongo db:
mongo rs.instance1/testDB -u username -p password

Here is the updated command to include variety;
mongo rs.instance1/testDB -u username -p password --eval "var collection = 'collection'" path/to/variety.js

When I try to execute variety on it it doesn't seem to initialize the db_name variable and I get this error

Variety: A MongoDB Schema Analyzer
Version 1.2.5, released 21 February 2014
Wed Feb 26 15:17:41.959 TypeError: Cannot call method 'forEach' of undefined at variety.js:23

Hi, your instructions say that you should tail the log file to see percentage complete if it is taking a long time. I don't see any log files appearing in the variety.js folder... Where is the log file in question and what is its filename?

it would be a nice touch to be able to set resultsDB an resultsCollectionName via command line as well.

Currently we just dump the JSON from the relevant varietyResults collection.

JavaScript execution failed: TypeError: Object # has no method 'toSource' at variety.js:L55

It is in the print("Using query of " + query.toSource()); line.

Commented out the line and the script worked perfectly.

see http://stackoverflow.com/questions/9338989/what-does-the-0-mean-in-mongodbs-bindata0-e8menzzofymmd7wshdnrfjyek8m

and

http://bsonspec.org/#/specification

"type" here should be in a form like
binData-generic
binData-function
binData-MD5

Tried running variety.js on a modestly-sized collection. After about 10min the script terminated:

Wed May  2 17:15:26 uncaught exception: map reduce failed:{
    "assertion" : "Invalid BSONObj size: 18709033 (0x297A1D01) first element: 0: { types: [ undefined, \"number\", \"array\" ] }",
    "assertionCode" : 10334,
    "errmsg" : "db assertion failure",
    "ok" : 0
}

Thoughts?

Sometime variety would out put stuff like this:

[66] oti@core-dev:~/Software/variety  [master]  $ /srv/mongodb/bin/mongo  --eval "var db_name = 'client'; var collection = 'rules'" variety.js
MongoDB shell version: 2.6.4
connecting to: test
Variety: A MongoDB Schema Analyzer
Version 1.4.1, released 14 Oct 2014
Using query of { }
Using limit of 84
Using maxDepth of 99
Using sort of { "_id" : -1 }
Using outputFormat of ascii
Using persistResults of false
removing leaf arrays in results collection, and getting percentages
+----------------------------------------------------------------------------------------------+
| key                                      | types          | occurrences | percents           |
| ---------------------------------------- | -------------- | ----------- | ------------------ |
| rules.XX.filter                          | Object         | 5907        | 7032.142857142857  |
| rules.XX.filter.account                  | String         | 5907        | 7032.142857142857  |
| rules.XX.modify                          | Object         | 5907        | 7032.142857142857  |
| rules.XX.name                            | String         | 5907        | 7032.142857142857  |
| rules.XX.modify.channel                  | String         | 5012        | 5966.666666666666  |
| rules.XX.modify.sub_channel              | String         | 5012        | 5966.666666666666  |
| rules.XX.modify.sub2_channel             | String         | 4976        | 5923.809523809524  |
| rules.XX.modify.top_channel              | String         | 4592        | 5466.666666666666  |
| rules.XX.filter.channel                  | String         | 4278        | 5092.857142857143  |
| rules.XX.filter.campaign                 | String         | 3816        | 4542.857142857143  |
| rules.XX.modify.paid_performance_channel | Boolean,String | 1987        | 2365.4761904761904 |
| rules.XX.modify.exclude                  | Boolean        | 1164        | 1385.7142857142858 |
| rules.XX.modify.sea_brand                | String,Boolean | 567         | 675                |
| rules.XX.modify.cpc                      | Number         | 476         | 566.6666666666667  |
| rules.XX.modify.cost                     | Number         | 281         | 334.5238095238095  |
| rules.XX.modify.campaignId               | String         | 168         | 200                |
| rules.XX.modify.crr                      | Number         | 159         | 189.28571428571428 |
| _id                                      | ObjectId       | 84          | 100                |
| author                                   | String         | 84          | 100                |
| rules                                    | Array          | 84          | 100                |
| date                                     | String         | 27          | 32.142857142857146 |
| rules.XX.modify.sea_brand_non_brand      | Boolean        | 4           | 4.761904761904762  |
+----------------------------------------------------------------------------------------------+

This is true for values inside arrays. Notice here the rules.XX.modify is appearing 5907 times in 84 records? This is exactly 5907/84*100=7032.14285714285.

The README displays Variety terminal output in several places, and is still using the old, pre-ASCII, invalid-JSON-style. We should update the README when we have a moment so that the documentation is up-to-date and not confusing (as well as showing off our new and highly readable ASCII tables!).

"limit" output should not appear unless database exists.

Hi,
I would like to ask, what is the original reason for creating Variety results database and __Keys collection. Currently results are printed either in readable form to stdout or formatted as a JSON and piped to another tool. In those situations, creating results database and collection seems like unexpected side effect to me.

Also some users are running Variety without admin or write permissions (requested in #40).

Does anyone use the collected data later? What is the typical use case?

Thank you,
Tomas

It may make life easier for us, if we only support the latest production release of MongoDB. I think we can actually get away with this, because Variety is intended to be run on development machines, where version restrictions are often much less harsh than production environments.

It could be soft, in that Variety could still agree to work, but with a deprecation warning explaining that only, say, version 2.2.2 (current as of writing) is officially supported.

It's not a "perfect world" fix, sure, but it would help make things run smoothly for Variety's little community, given our limited resources.

Any thoughts?

I keep getting the following error when running against a collection with about 14,000 documents. It creates 274 accountKeys and it looks like it is probably done processing because the last entry is an attribute that only occurs once.

Tue Dec 17 08:38:42.345 mongo got signal 11 (Segmentation fault: 11), stack trace: 

Tue Dec 17 08:38:42.508 0x10c208740 0x10c0f9ac3 0x7fff88f505aa 0x10c3f5a6d 0x10c2e5047 0x10c3b2649 0x10c3b24b1 0x10c1c51ab 0x10c1c506f 0x10c1010e6 0x10c101e0f 0x7fff8d5b05fd 0x5 
 0   mongo                               0x000000010c208740 _ZN5mongo15printStackTraceERSo + 64
 1   mongo                               0x000000010c0f9ac3 _Z12quitAbruptlyi + 323
 2   libsystem_platform.dylib            0x00007fff88f505aa _sigtramp + 26
 3   mongo                               0x000000010c3f5a6d _ZN2v88internal6Object23GetPropertyWithReceiverEPS1_PNS0_6StringEP18PropertyAttributes + 125
 4   mongo                               0x000000010c2e5047 _ZN2v88internal15DeoptimizerDataD1Ev + 55
 5   mongo                               0x000000010c3b2649 _ZN2v88internal7Isolate6DeinitEv + 105
 6   mongo                               0x000000010c3b24b1 _ZN2v88internal7Isolate8TearDownEv + 81
 7   mongo                               0x000000010c1c51ab _ZN5mongo7V8ScopeD2Ev + 267
 8   mongo                               0x000000010c1c506f _ZN5mongo7V8ScopeD0Ev + 15
 9   mongo                               0x000000010c1010e6 _Z5_mainiPPcS0_ + 24566
 10  mongo                               0x000000010c101e0f main + 95
 11  libdyld.dylib                       0x00007fff8d5b05fd start + 1
 12  ???                                 0x0000000000000005 0x0 + 5

I'd be nice to specify using a tree which fields should be analysed:

subtree = {f1: 1, f2: 1, f3: {f31: 1, f32: 2}}

(subtree is just a random name; something else might be better)

Even more, it'd be nice to specify which fields to leave out!

mask = {f2: {f24: {f24_field_with_dynamic_data: 1}}}

This is because maxDepth just cuts off at a certain level but this might not be applicable to all use cases (incl. mine).

This feature would also allow for analysis of just specific subdocuments.

I'm thinking of something perhaps less flexible like
variety <usual mongo shell parameters for connecting to a db>

variety could be a (unix) shell script... just sayin'

Hello all,

Tomáš (@tovdora) has suggested the following:

We can also consider Github release infrastructure. Plus we get download functionality for free with it - easier for users to download and use Variety. What do you think?

He's referring to Github Releases (note the capital R indicates a proper noun): https://github.com/blog/1547-release-your-software

I barely glanced at it, but it looks cool. Anybody used it? Any thoughts?

Tomáš, do you think it looks like a good move?

-James

Just something I thought we might ponder, due to #56.

Anybody have any thoughts?

Followed the easy example - type for pets only showing as "String". Tried on two versions of MongoDB 2.0.4 and 2.2.1. Below is the the output of 2.0.4. Both versions are on a 64-bit Ubuntu 12.04 machine. Any ideas? Thanks!

$ mongo test --eval "var collection = 'users'" variety.js
MongoDB shell version: 2.0.4
connecting to: test
Variety: A MongoDB Schema Analyzer
Version 1.2.2, released 04 November 2012
Using limit of 5
Using maxDepth of 99
creating results collection: usersKeys
removing leaf arrays in results collection, and getting percentages
{ "_id" : { "key" : "_id" }, "value" : { "type" : "ObjectId" }, "totalOccurrences" : 5, "percentContaining" : 100 }
{ "_id" : { "key" : "name" }, "value" : { "type" : "String" }, "totalOccurrences" : 5, "percentContaining" : 100 }
{ "_id" : { "key" : "bio" }, "value" : { "type" : "String" }, "totalOccurrences" : 3, "percentContaining" : 60 }
{ "_id" : { "key" : "pets" }, "value" : { "type" : "String" }, "totalOccurrences" : 2, "percentContaining" : 40 }
{ "_id" : { "key" : "birthday" }, "value" : { "type" : "Date" }, "totalOccurrences" : 2, "percentContaining" : 40 }
{ "_id" : { "key" : "someBinData" }, "value" : { "type" : "BinData-old" }, "totalOccurrences" : 1, "percentContaining" : 20 }
{ "_id" : { "key" : "someWeirdLegacyKey" }, "value" : { "type" : "String" }, "totalOccurrences" : 1, "percentContaining" : 20 }

Tomáš,

I was thinking: our test suite is in Java. Does this mean we can very easily (or even already) have a Java wrapper Variety?

If it's already there or very easy to get to from here, I would like to add this to our README.

Let me know.

Thank You,
James

running variety.js with the following command line:
mongo host/database -u user -p password --eval "var collection = 'deviceDetail', maxDepth = 1" scripts/variety-master/variety.js
give the following output:
MongoDB shell version: 2.6.1
connecting to: host/database
Variety: A MongoDB Schema Analyzer
Version 1.3.0, released 30 May 2014
2014-09-03T13:42:25.987-0700 TypeError: Cannot call method 'forEach' of undefined at scripts/variety-master/variety.js:22
failed to load: scripts/variety-master/variety.js

Further checking leads to looking at the following line:
mongos> db.adminCommand('listDatabases')
{
"note" : "not authorized for command: listDatabases on database admin",
"ok" : 0,
"errmsg" : "unauthorized"
}

which is the beginning of line 22 in the source file.

Is there a way to use the script without access to the admin database? It is not open in all of our databases.

James,

You commented on my Snipplr post for finding out the keys used in a MongoDB collection, as seen here: http://snipplr.com/view/59334/list-of-keys-used-in-mongodb-collection/

I feel your Javascript library would be very valuable if it was converted into a PHP library.

Would you be interested in giving me the ability to commit, if I was to create the PHP class for doing what your JS class is doing?

Look forward to hearing from you,

David

@DL_JR on Twitter
[email protected]

Hi All,

I'm trying to run variety.js on a database collection and getting the following error.

Mon Dec 29 12:50:05 TypeError: Object.keys is not a function variety.js:186
failed to load: variety.js

It seems to be an issue when it's converting interim results. The error producing line is here:

newEntry['value'] = {'types':Object.keys(entry['types'])};

I'd be happy to investigate this a bit more, but I'm not sure what the Object is referencing. Any help would be appreciated.

Thanks,
Marc

Since Variety is command line based, I am using Node.JS's childprocess.exec to run it through Node. If "exec(cmd...)" is on its own line, the tool works fine. However, if I try to trigger it with a Node.JS eventemitter, the tool runs multiple times until it errors.

child_process.js:945 throw errnoException(process._errno, 'spawn'); ^ Error: spawn EMFILE

Does anyone have any thoughts? The way I'm structuring my eventemitter is as follows:

var events = require('events');
var eventEmitter = new events.EventEmitter();

var cmd = "mongo " + config.dbUrl + " --eval " + '\"var collection = \'' + config.collect[0] + '\'\"' + " variety.js "
function key_gen() {
    exec(cmd, puts);
}

eventEmitter.on('db_changes', key_gen);

var target = db[config.collect[0]]; //eg. varietyResults.test3
watcher.watch(target, function(event) { //Watch Mongo oplog for any changes to source
    eventEmitter.emit('db_changes');
});

I'm encountering an error trying to use this. Most likely I'm doing something wrong. I barely ever use the command line or "shell" for anything. I use the C# driver and MongoVue for all of my Mongo interactions. But this is compelling enough to want to take a look :)

On my local dev box I have Mongo installed on port 9123 and have a number of databases. Here is what I've typed at the commandline:

E:\mongo\binaries>mongo 127.0.0.1:9123 phillynjnet --eval "var collection = 'speaker'" variety.js

As far as I understand it, I'm running Mongo, connecting to the local server telling it to evaluate the phillynjnet database and the "speaker" collection. I'm getting the following output including the error:

MongoDB shell version: 2.0.2
connecting to: 127.0.0.1:9123/test
loading file: phillynjnet
Thu Apr 26 09:03:33 file [phillynjnet] doesn't exist
failed to load: phillynjnet

Can you please tell me what I'm doing wrong?

Hello there,

I just pulled this interesting project and run into this problem:

mongo mpl --eval "var collection = 'reciepes', limit = 1" variety.js
MongoDB shell version: 2.0.4
connecting to: mpl
Variety: A MongoDB Schema Analyzer
Version 1.4.1, released 14 Oct 2014
Using query of { }
Using limit of 1
Using maxDepth of 99
Using sort of { "_id" : -1 }
Using outputFormat of ascii
Sat Nov 22 22:51:24 TypeError: Array.isArray is not a function variety.js:121
failed to load: variety.js

Any ideas ?

Just noticed this. Using Variety to analyze the README's sample collection, I get the following:

james@laptop:~/variety$  mongo test --eval "var collection = 'users'" variety.js
MongoDB shell version: 2.4.9
connecting to: test
Variety: A MongoDB Schema Analyzer
Version 1.2.6, released 28 March 2014
Using query of { }
Using limit of 5
Using maxDepth of 99
Using sort of { "_id" : -1 }
creating results collection: usersKeys
removing leaf arrays in results collection, and getting percentages
{ "_id" : { "key" : "_id" }, "value" : { "type" : "ObjectId" }, "totalOccurrences" : 5, "percentContaining" : 100 }
{ "_id" : { "key" : "name" }, "value" : { "type" : "String" }, "totalOccurrences" : 5, "percentContaining" : 100 }
{ "_id" : { "key" : "bio" }, "value" : { "type" : "String" }, "totalOccurrences" : 3, "percentContaining" : 60 }
{ "_id" : { "key" : "pets" }, "value" : { "type" : "String" }, "totalOccurrences" : 2, "percentContaining" : 40 }
{ "_id" : { "key" : "birthday" }, "value" : { "type" : "Date" }, "totalOccurrences" : 2, "percentContaining" : 40 }
{ "_id" : { "key" : "someBinData" }, "value" : { "type" : "BinData-old" }, "totalOccurrences" : 1, "percentContaining" : 20 }
{ "_id" : { "key" : "someWeirdLegacyKey" }, "value" : { "type" : "String" }, "totalOccurrences" : 1, "percentContaining" : 20 }

As the README states, we are expecting, for example, types of values for the pets key to include both String and Array.

Not sure how this happened, or if I'm making a mistake.

I will say that it doesn't appear to be due to my 1.2.6 version bump via JSHint validation, as I get the same results after reverting that commit.

James

eval() is being deprecated. Is there any other way to run Variety that is future proof?

Mentioned on the official reference manual:
(http://docs.mongodb.org/manual/reference/method/db.eval/)

The reasoning behind the deprecation.

Different error than the first uncaught exception: map reduce failed bug, might be related. Mine is easily recreated using the following source code.

Error
$ mongo garden --eval "var collection = 'user.actions'" variety.js
MongoDB shell version: 2.0.3
connecting to: garden
Variety: A MongoDB Schema Analyzer
Version 1.1, released 03 June 2012
Using limit of 37
Using maxDepth of 99
Fri Jul 20 06:54:45 uncaught exception: map reduce failed:{
"assertion" : "could not create cursor over garden.user.actions for query : {} sort : { _id: -1.0 }",
"assertionCode" : 15876,
"errmsg" : "db assertion failure",
"ok" : 0
}
failed to load: variety.js

Source code - logging.rb

require 'rubygems' 
require 'mongo' 

VIEW_PRODUCT = 0 
ADD_TO_CART = 1 
CHECKOUT = 2 
PURCHASE = 3 

@con = Mongo::Connection.new 
@db = @con['garden'] 

@db.drop_collection("user.actions") 

@db.create_collection("user.actions", :capped => true, :size => 1024) 

@actions = @db['user.actions'] 

40.times do |n| 
  doc = { 
    :username => "kbanker", 
    :action_code => rand(4), 
    :time => Time.now.utc, 
    :n => n 
  } 
  @actions.insert(doc) 
end 

execution
$ ruby logging.rb
$ mongo garden --eval "var collection = 'user.actions'" variety.js

MongoDB shell version: 1.6.3

Variety: A MongoDB Schema Analyzer
Version 1.2.2, released 04 November 2012
Fri Jan 25 17:32:19 TypeError: db.adminCommand is not a function variety.js:21

anyone can help ?

tks

Guys I love this project alot and have been working hard to give some contributions in my latest push to my repo I have added the ability to specify an array for the collection name. It still works with the single string as well. I have made a couple fixes to ensure that json formatting still works, I may make some minor output formatting changes but it seems to work great. Please test and let me know what you think the syntax is now
mongo test --eval "var collection = 'users'" variety.js For a single collection
mongo test --eval "var collection = ['users', 'articles', 'collection3']" variety.js For an array

In the next day or two I should also have a working copy that will do all collections in the database I was thinking of using "var mode = recursive" Please test and give feedback then I will do a pull request if the feature is liked. Thanks have a wonderful day!!

Hello.

I ran this command,

/usr/bin/mongo log -eval "var collection='join', query={'date':{'$gte':'2013-09-17','$lt':'2013-09-19'}}, limit=10000, maxDepth=1" public/javascripts/variety.js

and got this result.

MongoDB shell version: 2.4.1
connecting to: log
Variety: A MongoDB Schema Analyzer
Version 1.2.3, released 01 September 2013
Using query of {"date":{"":"2013-09-19"}}
Using limit of 10000
Using maxDepth of 1
creating results collection: joinKeys
removing leaf arrays in results collection, and getting percentages

The query string of the result is different from my command string.

Please help me.

In results of analysis I see key names with added '.XX' that do not exist in real collection.
I created collection with commands:

db.blog.insert({title:"Article 1", comments:[{author:"John", body:"it works"}]});
db.blog.insert({title:"Article 2", comments:[{author:"Tom", body:"thanks"}]});

I think result should contain keys _id, title, comments, comments.author, comments.body

The real result looks like this:

tomas@ac100:~/variety-master$ mongo test --eval "var collection = 'blog';" variety.js
MongoDB shell version: 2.2.4
connecting to: test
Variety: A MongoDB Schema Analyzer
Version 1.2.3, released 01 September 2013
Using query of { }
Using limit of 2
Using maxDepth of 99
creating results collection: blogKeys
removing leaf arrays in results collection, and getting percentages
{ "_id" : { "key" : "_id" }, "value" : { "type" : "ObjectId" }, "totalOccurrences" : 2, "percentContaining" : 100 }
{ "_id" : { "key" : "title" }, "value" : { "type" : "String" }, "totalOccurrences" : 2, "percentContaining" : 100 }
{ "_id" : { "key" : "comments" }, "value" : { "type" : "Array" }, "totalOccurrences" : 2, "percentContaining" : 100 }
{ "_id" : { "key" : "comments.XX.author" }, "value" : { "type" : "String" }, "totalOccurrences" : 2, "percentContaining" : 100 }
{ "_id" : { "key" : "comments.XX.body" }, "value" : { "type" : "String" }, "totalOccurrences" : 2, "percentContaining" : 100 }

The '.XX' is added on line 147 of variety.js and not removed.
Is it bug or expected behaviour? Thank you.

This suggestion comes from the inimitable Tomáš Dvořák, via #57 (comment).

Sample output on a mongo collection of 3.13m records and and unknown number of unique keys:

[root@leonardo variety]# mongo **** --eval "var collection = '****', maxDepth = 3" variety.js
MongoDB shell version: 2.4.1
connecting to: ****
Variety: A MongoDB Schema Analyzer
Version 1.2.2, released 04 November 2012
Using limit of 3132369
Using maxDepth of 3
creating results collection: ****Keys
removing leaf arrays in results collection, and getting percentages
Sat Mar 30 15:13:22.341 JavaScript execution failed: error: {
        "$err" : "too much data for sort() with no index.  add an index or specify a smaller limit",
        "code" : 10128
} at src/mongo/shell/query.js:L128
failed to load: variety.js

maxDepth=2 works fine, but anything beyond that (namely 3 levels) results in this error.

Do you need any other information? If it can be easily resolved with a modification to mongodb's core config, I'd be willing to do that since this is just a dev server.

We currently display tables like:

+------------------------------------------------------------+
| key                | types        | occurrences | percents |
| ------------------ | ------------ | ----------- | -------- |
| _id                | ObjectId     | 127         | 100      |
| name               | String       | 15          | 100      |
| bio                | String       | 3           | 60       |
| birthday           | String       | 2           | 40       |
| pets               | String,Array | 2           | 40       |
| someBinData        | BinData-old  | 1           | 20       |
| someWeirdLegacyKey | String       | 1           | 6        |
+------------------------------------------------------------+

For the sake of quick reading/scanning, it would be a significant improvement to display tables like:

+------------------------------------------------------------+
| key                | types        | occurrences | percents |
| ------------------ | ------------ | ----------- | -------- |
| _id                | ObjectId     | 127         | 100      |
| name               | String       |  15         | 100      |
| bio                | String       |   3         |  60      |
| birthday           | String       |   2         |  40      |
| pets               | String,Array |   2         |  40      |
| someBinData        | BinData-old  |   1         |  20      |
| someWeirdLegacyKey | String       |   1         |   6      |
+------------------------------------------------------------+

-James

When printing the output nowhere in the ouput does it show which collection the data is for. When you print the collections you then have to rely on the filename to tell you which collection you are looking at. It would be benificial to add the name of the collection into the output of the program for easier documentation.

From Tovdora (Tomas Dvorak):

Current version of variety is not able to handle unnamed inside objects (e.g. arrays). Earlier they were marked with XX. key prefix. Now the unnamed object are skipped and not analysed at all. Example of earlier version results can be seen in issue #29.

Hat tip to Tomas for finding this.

-James

Hey,

Do you all think we should have an NPM package for Variety, also distributing in this way?

Or perhaps, just to throw it out there, via another package manager?

-James

I think this would be adding commas to the end of each line except the last, and enclosing the whole grouping in square braces. In the meantime, I used this in python:

$ mongo dbname --eval \"var collection = 'collname'\" /path/to/variety.js | awk '/{/' > fname

data = []
with open(fname, 'r') as f:
    for line in f.readlines():
        l = json.loads(line)
        key = l['_id']['key']
        datatype = l['value']['type']
        occurence = l['totalOccurrences']
        coverage_pct = l['percentContaining']
        data.append(dict(key=key,datatype=datatype,occurence=occurence,coverage_pct=coverage_pct))

json.dumps(data)

I guess this is trivial for experienced users*, but never the less:

mongo <host_ip>/admin -u <USER_NAME> -p --eval "var db_name = 'databaseToUse'; var collection = 'collectionToScan'; other_option1; other_option2..." variety.js

• I am not really experienced with JS or with MongoDB, but reading the source code was helpful. Having this mentioned in the readme would spare me some minutes ;-)

Variety results can have multiple occurrences per record, thus inflating the percentages above 100% in some cases.

This bug was caused by my latest update--I'll fix it tomorrow.