Giter VIP home page Giter VIP logo

jtlibrary's Introduction

TissueMAPS

TissueMAPS is a software framework for distributed analysis and interactive visualization of large-scale, high-dimensional microscopy image datasets.

Online documentation is available at docs.tissuemaps.org.

Quickstart

TissueMAPS uses a distributed client-server model. The server exposes RESTful web services and clients interact with the server via the REST API using either the browser-based user interface or other client interfaces.

Server

The easiest way to set up your own server for development and testing is to use either the pre-build container images available on Docker Hub or the pre-build virtual machines images for Amazon Web Services (AWS):

To run the containerized TissueMAPS server, clone the repository and bring up the containers using Docker Compose:

git clone https://github.com/tissuemaps/tissuemaps ~/tissuemaps
cd ~/tissuemaps
docker-compose up -d

To run a TissueMAPS server in a cloud virtual machine, launch a new instance from one of the shared Amazon Machine Images (AMIs) using the Elastic Compute Cloud (EC2) console. To find the TissueMAPS images, filter public AMIs in the Frankfurt region for AMI Name: TissueMAPS server (see AWS documentation).

For further details and alternative installation options, please refer to the installation guide of the online documentation.

Client

Once you have access to a running server instance, you can interact with it via HTTP protocol.

Browser

The easiest way is to interact with the server is via the web-based user interface. To this end, point your web browser to http://localhost:8002 when running the server within a container. Otherwise, just point the browser to the public IP address of your cloud virtual machine. This will redirect you to the login page. Once you have authenticated yourself using your credentials, you will see a list of your existing experiments.

cURL

Alternatively, you could use cURL to access resources via the command line:

Authenticate with your credentials:

curl --data '{"username": "devuser", "password": "123456"}' --header "Content-Type:application/json" http://localhost:8002/auth

List your existing experiments using the received JSON web token (replace XXX with the actual access token):

curl --header "Content-Type:application/json" --header "Authorization:JWT XXX" http://localhost:8002/api/experiments

TmClient

TissueMAPS also provides a Python client that abstracts the HTTP interface and facilitas interaction with the server. You can install it via the pip Python package manager:

pip install tmclient

List your existing experiments using the tm_client command line tool:

tm_client -H localhost -P 8002 -u devuser -p 123456 experiment ls

Or using the tmclient Python package:

License

Client code is licensed under Apache 2.0 and server code under GNU Affero General Public License 3.0.

For more information please refer to the license section of the online documentation or the LICENSE.txt files in the individual Github repositories.

jtlibrary's People

Contributors

hackermd avatar jluethi avatar riccardomurri avatar saadiaiftikhar avatar scottberry avatar sparkvilla avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

jtlibrary's Issues

Wrong object count in identify_secondary_iw.m

For some images, the object count is wrong after applying the watershed segmentation. The reason is not clear to me yet. It is related to the shape of the primary objects, however, since smoothing the input mask solves the problem. It could be related to the labeling of objects, which is different in Matlab and Python.

segment secondary module test

Error in jtlib.segmentSecondary (line 502)
FinalLabelMatrixImagePre = LabelsUsed(ActualObjectsLabelMatrixImage3+1);
Index exceeds matrix dimensions.

segment secondary module test bug

https://github.com/TissueMAPS/JtLibrary/blob/saadia/python/jtlib/modules/Jt_Modules_Saadia/segment_secondary.py

This code works fine in the first iteration, but in the next iteration- it gives the error:

Traceback (most recent call last):

File "", line 1, in
runfile('/Users/saadiaiftikhar/miniconda2/bin/jlib/test_20160531.py', wdir='/Users/saadiaiftikhar/miniconda2/bin/jlib')

File "/Users/saadiaiftikhar/miniconda2/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 807, in runfile
execfile(filename, namespace)

File "/Users/saadiaiftikhar/miniconda2/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 88, in execfile
builtins.execfile(filename, *where)

File "/Users/saadiaiftikhar/miniconda2/bin/jlib/test_20160531.py", line 99, in
[secondaryLabelMatrixImage,edited_primary_binary_image,th_array] = identify_secondary_objects_iw.identify_secondary_objects_iw(label_image,input_image,correction_factors,min_threshold)

File "identify_secondary_objects_iw.py", line 48, in identify_secondary_objects_iw
min_threshold, max_threshold)

File "segment_secondary.py", line 190, in segment_secondary
labels_used1[actual_objects_label_matrix_image3]

IndexError: index 4 is out of bounds for axis 1 with size 4

module for ilastik integration

Creation of a jterator module for pixel based classification ร  la CellProfiler classifypixels module. The idea is to train a classifier with the Ilastik GUI, save the output (either the classifier as ilastik project HDF5 file or a probability map as an image file) and then apply the output to images in a jterator pipeline. To this end, we need to create a Python module that implements functions to call ilastik independent of CP2.

pipeline

I tried various possibilities, but the error remained the same for the pipeline:

() | ___ _ _ __ | | ___ _ _ tmlib.workflow.jterator (0.0.7)
| | / -) '/ ` | / _ \ '| Image analysis pipeline engine
/ |
|| **,|___/_| https://github.com/TissueMAPS/TmLibrary
|**/

2016-06-02 13:07:14 | INFO | tmlib.workflow.cli | delete previous job output
2016-06-02 13:07:14 | INFO | tmlib.workflow.jterator.api | delete existing mapobject types
2016-06-02 13:07:14 | INFO | tmlib.workflow.cli | create batches
The following error occurred while reading from file:
while parsing a block mapping
in "", line 2, column 1:
input:
^
expected , but found '-'
in "", line 8, column 1:
- name: filter_name
^ File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/project.py", line 212, in _create_handles
'description': f.read()
File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 182, in read
return load_yaml(self._stream.read())
File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 162, in load_yaml
return ruamel.yaml.load(string, ruamel.yaml.RoundTripLoader)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/main.py", line 75, in load
return loader.get_single_data()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 59, in get_single_data
node = self.get_single_node()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 48, in get_single_node
document = self.compose_document()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 68, in compose_document
node = self.compose_node(None, None)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 99, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 151, in compose_mapping_node
while not self.check_event(MappingEndEvent):
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 116, in check_event
self.current_event = self.state()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 505, in parse_block_mapping_key
token.start_mark)

Plotting is broken in current develop due to import error

Whenever plotting is set to True in a module in the current develop branch (14db3a8), the pipeline fails because of the following import error:
JOB FAILED: cannot import name plotting File "/home/tissuemaps/tmlibrary/tmlib/workflow/cli.py", line 283, in __main__ cli_instance(arguments) File "/home/tissuemaps/tmlibrary/tmlib/workflow/cli.py", line 342, in __call__ method(**method_args) File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/cli.py", line 102, in debug api.run_job(batch, assume_clean_state=False) File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/api.py", line 679, in run_job store = self._run_pipeline(store, site_id, batch['plot']) File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/api.py", line 363, in _run_pipeline module.run(self._engines[module.language]) File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/module.py", line 470, in run return self._exec_py_module() File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/module.py", line 196, in _exec_py_module py_out = func(**kwargs) File "/home/tissuemaps/jtlibrary/python/jtmodules/src/jtmodules/smooth.py", line 85, in main from jtlib import plotting

The "from jtlib import plotting" doesn't work anymore because of the refactor that reorganized the modules & jtlibrary code. @riccardomurri How can one import the plotting file which is located at "JtLibrary/python/jtlibrary/src/jtlib/"? This import needs to be changed for all modules that use plotting.

Retrieval of saved objects

The current approach for saving and retrieving objects is based on outline coordinates of objects.

  1. Outlines of objects in the labeled image are determined for each object: skimage.measure.find_contours()
  2. y, x coordinates are saved in the HDF5 file
  3. Coordinates are retrieved from the file
  4. The label image is reconstructed: cv2.fillPoly()

It needs to be tested whether this approach works reliably for different types of objects...

Debugging of modules

The way modules are currently imported makes it difficult to debug them.

They should probably be included into the jtlib package as subpackage modules. Thereby one could import them "normally" and set break points in the code.

Maybe we should re-think the whole approach of providing the absolute path to modules. The idea behind that was that one could easily test code located somewhere else, but this may not pay off.

iterator pipeline issue

ubuntu@frontend001:/data/storage/experiments/experiment_1/workflow/jterator_segment_2d_cells_40x$ jterator -v 1 -p segment_2d_cells_40x init

_ _                _

() | ___ _ _ __ | | ___ _ _ tmlib.workflow.jterator (0.0.7)
| | / -) '/ ` | / _ \ '| Image analysis pipeline engine
/ |
|| **,|___/_| https://github.com/TissueMAPS/TmLibrary
|**/

2016-06-01 13:24:40 | INFO | tmlib.workflow.cli | delete previous job output
2016-06-01 13:24:40 | INFO | tmlib.workflow.jterator.api | delete existing mapobject types
2016-06-01 13:24:40 | INFO | tmlib.workflow.cli | create batches
The following error occurred while reading from file:
while scanning for the next token
found character '\t' that cannot start any token
in "", line 6, column 16:
correct: true
^ File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/project.py", line 165, in _create_pipe
'description': f.read()
File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 182, in read
return load_yaml(self._stream.read())
File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 162, in load_yaml
return ruamel.yaml.load(string, ruamel.yaml.RoundTripLoader)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/main.py", line 75, in load
return loader.get_single_data()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 59, in get_single_data
node = self.get_single_node()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 48, in get_single_node
document = self.compose_document()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 68, in compose_document
node = self.compose_node(None, None)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 99, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 158, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 97, in compose_node
node = self.compose_sequence_node(anchor)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 128, in compose_sequence_node
node.value.append(self.compose_node(node, index))
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 99, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 158, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 77, in compose_node
if self.check_event(AliasEvent):
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 116, in check_event
self.current_event = self.state()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 518, in parse_block_mapping_value
token.move_comment(self.peek_token())
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/scanner.py", line 1543, in peek_token
self._gather_comments()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/scanner.py", line 1573, in _gather_comments
self.fetch_more_tokens()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/scanner.py", line 276, in fetch_more_tokens
% utf8(ch), self.get_mark())

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.