veg / datamonkey-js Goto Github PK

Output from running mocha (after removing pesky .only statements):

Currently, queuing is done per analysis type.

We should switch to a more sophisticated type that takes into consideration:

• IP Addresses
• Analysis
• Sites and Sequence number
• Using heuristics gained from logs

• Single dropdown specifying whether all leaf nodes, internal nodes, or all branches are to be tested.
• Branches unable to be selected
• Allow viewing of all partitions

See attached movie. When hovering over a method menu item, the text that used to be in the tooltip is now appended permanently to the page. This could be the case for other tooltips as well.

Dear @stevenweaver,

The menu bar does not work for me (e.g. no items in the "Cluster menu") in HIV-TRACE result pages, and the template UI is several generation behind release for Secure HIV-TRACE as it looks like.

Best,
Sergei

Installation currently needs an additional npm install winston for logging

Dear @stevenweaver,

There seem to be issues with HIV-TRACE when edge filtering or contaminant removal is turned (exceptions returned). Could you please run a few tests to confirm? If you can't, I'll show you the problematic dataset tomorrow.

We want HIV-TRACE to be running smoothly because the paper is about to come out.

Best,
Sergei

datamonkey-js is not always closing mongodb connections, and is continually creating new ones. This can cause the node process to periodically restart.

Some become a bit concerned when there is no new output for a while. We should provide some sort of re-assurance that the job is still running and is healthy.

Under rare circumstances (< 1% of jobs), there will be instances where jobs will never be marked as either aborted or completed. The exact cause for this is currently not fully appreciated, but it seems to happen in concurrence with the restart of the datamonkey-js pm2 process due to an unhandled exception (e.g. failing to check if the results JSON is valid before an attempted parse with no exception handling, triggering configured system limits, etc.). We should create a periodic reaper that harvests the jobs marked as "running" for longer than a pre-determined interval and mark whether the job has been completed or didn't make it.

Oftentimes, the tree will be too large to do anything meaningful within the results page itself. We should allow for the tree to be viewed as an entirely separate window. In addition, we should have more export options such as all branch attributes associated with the tree.

Both HyPhy and HyPhy vision can accept a Nexus file with partitioned data when using BUSTED (see, for instance, https://github.com/veg/hyphy-vision/blob/gh-pages/methods/busted/data/BUSTED_2partitions_testbg.json).

However, Datamonkey currently errors out when uploading a Nexus file. At a minimum, the user should be presented with the option of selecting all, internal, or leaf branches to test for selection, as would be the case running HyPhy from the command line.

The Datamonkey redesign is great. I was wondering though why you decided to get rid of the phylogenetic tree option for each position in the alignment in the output - it was a nice way to see what the ancestral state might have been...thanks and keep up the great work!!

The CSV button is not placed in a particularly logical spot. I see two places for it

1. The button UI bar above the network plot (easy)
2. The fixed nav bar at the top of the page (this is where 'global' analysis exports/functions should probably go anyway).

Hi

I´ve been trying to use MEME for the past two weeks and sometimes it runs and sometimes it doesn´t but in the log file I do not get any error or warning saying why it stops. Here is the lattest job I tried to run.
log(1).txt

The fonts that were chosen by Michael Li for HyPhy Vision did not carry over to Datamonkey.

I was trying to do a FEL analysis with Datamonkey and even if I try with the example data set (the flu.fasta one), use defaults then select all or leaf nodes I get a "400 bad request" error (this is the link that appears http://datamonkey.org/fel/<%=%20fel._id%20%>/select-foreground, something doesn't seem kosher with that, it looks like the job ID is not getting properly interpolated?

The following error has occurred 23 times over the past year as of the opening of this ticket :

277303 Error: EMFILE: too many open files, uv_resident_set_memory
277304     at process.memoryUsage (internal/process.js:100:5)
277305     at Object.exception.getProcessInfo (/opt/datamonkey-js/node_modules/winston/lib/winston/exception.js:33:26)
277306     at Object.exception.getAllInfo (/opt/datamonkey-js/node_modules/winston/lib/winston/exception.js:17:24)
277307     at exports.Logger.Logger._uncaughtException (/opt/datamonkey-js/node_modules/winston/lib/winston/logger.js:602:24)
277308     at emitOne (events.js:115:13)
277309     at process.emit (events.js:210:7)
277310     at process._fatalException (bootstrap_node.js:329:26)

We should investigate which file handlers are not being properly closed.

@stevenweaver and @spond , nice job so far on DM-js. I'm curious to try it out, but npm install is broken on both Linux and OSX:

> [email protected] postinstall /Users/simon/Programs/datamonkey-js
> ./node_modules/.bin/lessc ./contrib/dm-bootstrap/overrides.less > ./public/assets/css/bootstrap.css

FileError: '../bootstrap/less/bootstrap.less' wasn't found in /Users/simon/Programs/datamonkey-js/contrib/dm-bootstrap/overrides.less on line 1, column 1:
1 @import "../bootstrap/less/bootstrap.less";
2 

npm ERR! weird error 1
npm ERR! not ok code 0

Update stats page to display average runtimes of each model

We should create an admin panel that allows administrators to

1. View number of jobs spawned in last day
2. Reports cancelled and aborted jobs.
3. Reports completed jobs.
4. Provides links to logs of HyPhy jobs that were unable to complete. The logs should only support stack traces.
5. Node process metrics
6. Cancel jobs

The datamonkey image as part of the footer is a bit large on screens with smaller resolutions. We either remove it or create a proper bootstrap footer with the image, maximum height 60 px.

Create a simple API that can:
Upload
Check status
Show Result

Results ejs files can now be a template. It should be the same for all methods besides which hyphyVision function is called.

Add a Totals tab that displays job information across all methods.

Add benchmarking statistics, such as runtime and resources used, to the results page.

The help page should also be edited and updated during transcription.

The stats page is currently very slow to load. If a native solution in MongoDB is not available, then we should look into caching stats in redis and serving from there.

Add an option to the set-up page (selected by default) to remove unsupported edges using the triangle phylogenetic trick.

The BUSTED results page currently does not correctly display the p-value for the analysis; see http://test.datamonkey.org/busted/54cfd392581018526dfe8633

Create a simple cluster of node processes for performance and reliability improvements.

test

hello,
I have run an analysis using aBSREL that creates the desired output data, but when I hover over the branches of the fitted tree with the mouse cursor, the branch information does not appear. I have run it on different browsers but encountered the same problem. thank you for your help.

In order for users to remember what runs are which, we should allow them to name them.

• On the job submission page, add a new field with a label such as "alias", "run name", etc.
• Place the name on the page under the method title in the progress and results pages
• Reference the name in the email as well.

Create routes with example results data that can be used for local development to see how hyphy-vision results pages will look in datamonkey.

Currently, researchers can still submit jobs no matter what the job dispatcher's status is. A warning banner should be displayed if the job dispatcher is not responsive.

With the newer version of HyPhy, it seems as though how annotated trees are parsed has changed. We should update datamonkey-js job submission to reflect his change.

### Branch sets for RELAX analysis
* Selected 43 branches as the _test_ set: `478488770, 149706207, 664699950, 664699948, Node252, Node250, Node248, 795152647, 724899480, Node259, 109068106, 635116136, 795402852, 795323436, 402864747, Node270, Node268, 795228246, 685533192, 545687090, Node275, Node273, Node267, Node265, Node263, 332224382, 297681450, 426357812, 110832709, 57114123, Node284, Node282, Node280, Node278, Node262, Node258, 296210687, 403256847, Node289, Node287, Node257, Node255, Node247`
* Selected 7 branches as the _reference_ set: `829888493, 831225769, 826351608, Node244, Node242, 640810396, 817317597`
* 353 branches are in the unclassified (nuisance) set: `12054987, 6687226, 700438741, 4027889, 17974228, Node9, 155008488, 38231558, 8671490, 183985969, 54311338, Node18, Node16, Node14, Node12, Node8, 530573866, 591386123, Node23, 884527912, 884527910, Node27, 12667704, 5565637, Node31, 327288995, 884527920, 73666470, 884527918, 884527916, 884527914, Node44, Node42, Node40, Node38, 884527908, 240254682, 602672386, 240254688, Node52, Node50, 884527922, 884527928, 884527924, 884527926, Node59, Node57, Node55, Node49, Node47, Node37, Node35, 884527906, Node34, Node30, Node26, Node22, 59895726, 59895724, Node65, 85720641, 86211179, 86211177, 612053414, Node73, Node71, Node69, 395539363, 38231560, Node77, 52857761, 52221229, 31322956, Node82, Node80, Node76, Node68, Node64, 585696530, 507555013, 507693866, 731236363, 533178379, 512989065, 289629292, Node98, Node96, Node94, Node92, 674074617, 532046007, 14192790, 625289930, Node106, 880974618, 589933402, 568939379, 568939385, 37194856, Node116, Node114, 213579634, 25742620, Node119, Node113, Node111, Node109, Node105, Node103, Node101, Node91, 852759624, 667308135, 562835163, Node125, 83630405, 532067275, Node129, 655819600, 504165095, Node132, Node128, Node124, Node122, Node90, Node88, 586461042, 634888841, Node139, 51101223, 471399281, Node142, Node138, 98956513, 617617805, Node145, Node137, 507667492, 47522990, 560959760, 560917086, 744551514, 743705125, Node159, Node157, Node155, Node153, 594657847, 593762353, 602719153, 466021507, 470597006, Node169, Node167, Node165, Node163, 1905885, 594102812, 555986909, 742236428, Node177, Node175, Node173, 556772617, 548463879, 803253841, Node183, Node181, 295007320, Node180, Node172, Node162, Node152, 591302466, 410952805, Node188, 511834482, 62275622, Node193, 671026532, 62275609, 301755248, Node198, Node196, Node192, 472356891, 73975826, Node201, Node191, Node187, Node151, 284813839, 284813829, 284813843, Node208, Node206, 641724698, 284813833, Node212, 284813841, 254838572, 558123362, 98956515, 554548891, Node222, 584053674, 284813831, 194400541, Node227, Node225, Node221, Node219, Node217, Node215, Node211, Node205, 759111738, 98956505, 586551086, 284813847, Node235, Node233, Node231, 505801345, 507931660, Node238, Node230, Node204, Node150, Node148, Node136, Node241, Node135, Node87, Node85, Node63, Node21, Node7, Node5, 698413180, 729741604, 768405188, 541992323, 529449849, Node299, Node297, Node295, Node293, 723552850, 697870545, Node304, 701343627, 700375237, 154813845, 704335932, 677991594, 727000194, 669286296, Node318, Node316, 675421290, 30408052, 543380551, 30408056, 30408054, Node329, Node327, Node325, 30408060, 734704490, 7981414, 30408058, 734704468, 734704494, 734704476, Node345, Node343, Node341, Node339, Node337, 734704466, 734704474, Node349, 734704478, 734704482, Node352, Node348, Node336, 734704488, 543297617, 115529268, 542185675, Node359, Node357, Node355, Node335, Node333, 30408062, 525028303, 30408064, Node364, Node362, Node332, Node324, 734704496, 734704508, 734704506, 734704502, 734704498, 734704500, Node375, Node373, Node371, Node369, Node367, Node323, Node321, Node315, Node313, Node311, Node309, Node307, Node303, 694850140, 697033715, 699660929, 696987556, Node384, Node382, Node380, 686608688, Node379, 45382143, 704289124, Node388, Node378, Node302, Node292, Node4, 296142098, 296142096, 296142094, Node394, Node392, 701322822, 2894432, 570517896, 296142092, Node401, Node399, Node397, Node391, Node3`

FUBAR results cause datamonkey-js memory to bloat due to the JSON.parse call in fubar.getPage.