scratcharchive / mountaintools Goto Github PK

View Code? Open in Web Editor NEW

2.0 5.0 1.0 6.14 MB

License: Apache License 2.0

Python 90.07% Dockerfile 0.24% JavaScript 9.52% Shell 0.16%

mountaintools's Introduction

MountainTools

A collection of tools for creating shareable and reproducible scientific workflows. It is used by SpikeForest.

MountainTools provides

A formal method for defining well-defined Python procedures that operate on input parameters and files and produce output files. These are known as Processors.
Automatic execution of Processors inside singularity containers
Automatic caching of results from Processor runs
Job management, batching, and queing mechanisms for Processor runs
Parallelization capabilities and automated running of processing batches on compute clusters
Functions for storing files, text, and objects (JSON-serializable Python dicts) in local and remote databases
Content-addressible storage databases (kacheries)
A key/value storage database (pairio)
Tools for working with sha1:// and sha1dir:// URIs

The Python package is installable via

pip install mountaintools

and comprises the following modules:

mlprocessors - utilities for MountainTools Processors, jobs, batches, parallelization, and running batches on a compute cluster
mountainclient - for working with sha1:// and sha1dir:// URIs, accessing the local MountainTools databases, and interacting with remote kacheries (content-addressible storage databases).
vdomr - for building GUIs using a mix of Python and JavaScript that may be run in notebooks, in the browser, or on the desktop. This will eventually be separated out and maintained as its own Python package.

This repository also contains JavaScript / NodeJS code for running pairio and kachery servers.

Installation

To install mountaintools the easiest approach is to use pip. Open your terminal emulator and type in the following command:

pip install mountaintools

Note: Depending on your environment you might need to substitute pip with pip3 as mountaintools works only with Python 3.

For information on using mountaintools see mountainclient, mlprocessors, and vdomr.

mountaintools's People

Contributors

Stargazers

Watchers

Forkers

stjordanis

mountaintools's Issues

kachery-token without any arguments should do kachery-token list

fix mountainclient docs

@wysota

mt.login() and mt.configRemote*() have been deprecated.

Instead use mt.configDownloadFrom()

In addition to getting/setting to pairio (formerly cairio) mountainclient now interacts with kachery servers for remote content upload/download

Also we will need to document how to set kachery upload/download tokens. For example, create a ~/.mountaintools/kachery_tokens file with the following content

spikeforest.kbucket download *******secret-token***********
spikeforest.kbucket upload ******secret-token************
http://some-ip-or-host download ******secret-token************

for repro code, get spikeforest and mountaintools version from database, not hard-coded

create unit test for kachery server

Witold, I have in mind a particular way I think this should be done. I can provide more details, but here's an outline:

Make a class similar to the following one that starts and stops a local kachery server via a "with" statement: https://github.com/flatironinstitute/mountaintools/blob/master/mlprocessors/localcomputeresource.py
Use the mountainclient to test read and write to that server in a test function that will be called by pytest.

Will need to worry about location of temporary directories for back-end server files, etc, that needs to get cleared out with each test. Keeping it simple.