Giter VIP home page Giter VIP logo

tesseract's Introduction

Build Status Coverage Status License: MIT

tesseract

tesseract is a library that enables the remote execution of python code on systems implementing the GA4GH Task Execution API.

Install

Available on PyPi.

pip install py-tesseract

Quick Start

from __future__ import print_function

from tesseract import Tesseract, FileStore


def identity(n):
    return n


def say_hello(a, b):
    return "hello " + identity(a) + b


fs = FileStore("./test_store/")
r = Tesseract(fs, "http://localhost:8000")
r.with_resources(
    cpu_cores=1, ram_gb=4, disk_gb=None, 
    docker="python:2.7", libraries=["cloudpickle"]
)

future = r.run(say_hello, "world", b="!")
result = future.result()
print(result)

r2 = r.clone().with_resources(cpu_cores=4)
f2 = r2.run(say_hello, "more", b="cpus!")
r2 = f2.result()
print(r2)

Object store support

If you provide a swift, s3, or gs bucket url to your FileStore tesseract_ will attempt to automatically detect your credentials for each of these systems.

To setup your environment for this run the following commands:

  • Google Storage - gcloud auth application-default login
  • Amazon S3 - aws configure
  • Swift - source openrc.sh

Input files

If your function expects input files to be available at a given path then add:

r.with_input("s3://your-bucket/path/to/yourfile.txt", "/home/ubuntu/yourfile.txt")

The first argument specifies where the file is available, the second specifies where your function will expect to find the file.

Output files

If your function generates files during its run you can specify these files as shown below and tesseract will handle getting them uploaded to the designated bucket.

r.with_output("./relative/path/to/outputfile.txt")

Resources

tesseract's People

Contributors

adamstruck avatar kellrott avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tesseract's Issues

Add retries to object store support?

Failure in communication with swift storage resulted in pipeline failure.
Perhaps add re-try to compensate

 File "./run_training.py", line 68, in <module>
    scores = ts.run(do_learning, "/data/feature.matrix", "/data/label.matrix", "/data/adj.matrix", list(gene_set))
  File "build/bdist.macosx-10.10-intel/egg/tesseract/tesseract.py", line 160, in run
  File "build/bdist.macosx-10.10-intel/egg/tesseract/filestore.py", line 199, in upload
  File "/Users/ellrott/workspaces/nci-cptac-dream/venv/lib/python2.7/site-packages/apache_libcloud-2.2.1-py2.7.egg/libcloud/storage/drivers/cloudfiles.py", line 435, in upload_object
    verify_hash=verify_hash, headers=headers)
  File "/Users/ellrott/workspaces/nci-cptac-dream/venv/lib/python2.7/site-packages/apache_libcloud-2.2.1-py2.7.egg/libcloud/storage/drivers/cloudfiles.py", line 784, in _put_object
    driver=self)
libcloud.common.types.LibcloudError: <LibcloudError in <libcloud.storage.drivers.cloudfiles.OpenStackSwiftStorageDriver object at 0x112646410> "Server didn't return etag">

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.