fredhutch / easybuild-life-sciences Goto Github PK

commit triggers:

add easyconfigs to .gitattributes file with attribute fh_built set
re-generate gh-pages json file using fh_built attribute and full easyconfigs repo
use jstree to display json of all easyconfigs, indiciating which we have built, and allowing users to select and request builds/ask questions

bioconda is has recently seen some growth, it's more primitive than easybuild but has a lot of bioinformatics packages, lot's of them R or python. We should review if any of these are worth adding to our easybuild

git clone https://github.com/bioconda/bioconda-recipes
cd bioconda-recipes/recipes

grep -R "url: https://pypi.python.org" * | wc -l
346

grep -R "lib/R/lib/" * | wc -l
465

If one does module show <module> or module help <module>, the contents of modloadmsg are not displayed in the output. This information should be available via module show so that use of the module is clear to the end user prior to actually loading the module.

Example:

rhino3[~/Work/GATK](master): module help GATK

---------------- Module Specific Help for "GATK/3.5-Java-1.8.0_66" -----------------
 The Genome Analysis Toolkit or GATK is a software package developed at the Broad Institute 
 to analyse next-generation resequencing data. The toolkit offers a wide variety of tools,
 with a primary focus on variant discovery and genotyping as well as strong emphasis on 
 data quality assurance. Its robust architecture, powerful processing engine and 
 high-performance computing features make it capable of taking on projects of any size. - Homepage: http://www.broadinstitute.org/gatk/

This information should include the tidbit that to use the GATK tools, one must run it as:

java -jar $CLASSPATH -T <GATK command> <GATK command options>

from UCSD homer peak calling
http://homer.ucsd.edu/homer/download.html

2016-b

• GATK
• BWA
• SAMtools
• Picard
• IGV
• Tophat
• Cufflinks
• BEDtools
• seqtk
• annovar

getting now pretty far building R with bootstrapped install but it had problems with RGL package

== installing extension glmnet 2.0-5 (256/410)...
== installing extension rgl 0.95.1441 (257/410)...
== FAILED: Installation ended unsuccessfully (build directory: /home/eb/.local/easybuild/build/R/3.3.1/foss-2016b): build failed (first 300 chars): cmd "R CMD INSTALL /home/eb/.local/easybuild/sources/r/R/extensions/rgl_0.95.1441.tar.gz --library=/home/eb/.local/easybuild/software/R/3.3.1-foss-2016b/lib/R/library --no-clean-on-error" exited with exitcode 1 and output:

• installing source package 'rgl' ...
  ** package 'rgl' successfully unpac
  == Results of the build can be found in the log file(s) /tmp/eb-fofEI7/easybuild-R-3.3.1-20161119.185829.PpWEt.log.1, /tmp/eb-fofEI7/easybuild-R-3.3.1-20161119.185829.PpWEt.log
  ERROR: Build of /easybuild/software/EasyBuild/3.0.0/lib/python2.7/site-packages/easybuild_easyconfigs-3.0.0-py2.7.egg/easybuild/easyconfigs/r/R/R-3.3.1-foss-2016b.eb failed (err: 'build failed (first 300 chars): cmd "R CMD INSTALL /home/eb/.local/easybuild/sources/r/R/extensions/rgl_0.95.1441.tar.gz --library=/home/eb/.local/easybuild/software/R/3.3.1-foss-2016b/lib/R/library --no-clean-on-error" exited with exitcode 1 and output:\n* installing source package 'rgl' ...\n** package 'rgl' successfully unpac')

Seems to be a conflict between the lua used by julialang and the one needed by module itself:

rhino1[~]: module load julia
rhino1[~]: module list
/app/lua/5.1/bin/lua: symbol lookup error: /app/julia/0.3.0/usr/lib/libreadline.so.6: undefined symbol: UP

ticket: 94518
Requester: Alli Black

Smith-Waterman Library
can be installed via:
git clone https://github.com/mengyao/Complete-Striped-Smith-Waterman-Library.git && cd Complete-Striped-Smith-Waterman-Library/src && make
(not in easybuild)

• R and Python latest
• move FH easyconfigs to github
• current version of exts_update.py sent upstream
• Carl trained to run R and Python
• lmod central cache and global activation
• lmod logging to syslog
• successful builds pushed to automatically to github

Phase II

• build a portal / dashboard with offered packages
• intel optimization flags

• REDCapR (R-3.3.1-foss-2016b-fh3)
• partools (https://cran.r-project.org/web/packages/partools/partools.pdf)
• RPostgreSQL
• sqldf
• dplyr
• lambda.r
• futile.options
• futile.logger
• perm
• samr (samr was droped from CRAN and moved to Bioconductor)
• mixOmics

rhino1:~$ gdc-client
Traceback (most recent call last):
File "/app/easybuild/software/gdc-client/1.0.1-foss-2016b-Python-2.7.12-fh2/bin/gdc-client", line 4, in
import('pkg_resources').run_script('gdc-client==1.0.1', 'gdc-client')
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 3138, in
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 3124, in _call_aside
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 3163, in _initialize_master_working_set
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 975, in subscribe
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 3163, in
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 2661, in activate
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 2755, in insert_on
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 2785, in check_version_conflict
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 2654, in _get_metadata
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 1643, in get_metadata_lines
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 1635, in get_metadata
File "build/bdist.linux-x86_64/egg/pkg_resources/init.py", line 1746, in _get
IOError: [Errno 13] Permission denied: '/app/easybuild/software/Python/2.7.12-foss-2016b-fh1/lib/python2.7/site-packages/blist-1.3.6-py2.7-linux-x86_64.egg/EGG-INFO/top_level.txt'

root@rhino-d:/# ls -l /app/easybuild/software/Python/2.7.12-foss-2016b-fh1/lib/python2.7/site-packages/blist-1.3.6-py2.7-linux-x86_64.egg/EGG-INFO/
total 16
-rw-rw---- 1 jfdey scicompstaff 1 Oct 3 09:23 dependency_links.txt
-rw-rw-r-- 1 jfdey scicompstaff 16 Oct 3 09:23 native_libs.txt
-rw-rw---- 1 jfdey scicompstaff 1 Oct 3 09:23 not-zip-safe
-rw-rw---- 1 jfdey scicompstaff 8790 Oct 3 09:23 PKG-INFO
-rw-rw---- 1 jfdey scicompstaff 644 Oct 3 09:23 SOURCES.txt
-rw-rw---- 1 jfdey scicompstaff 6 Oct 3 09:23 top_level.txt
root@rhino-d:/#
root@rhino-d:/# chmod o+r /app/easybuild/software/Python/2.7.12-foss-2016b-fh1/lib/python2.7/site-packages/blist-1.3.6-py2.7-linux-x86_64.egg/EGG-INFO/*
root@rhino-d:/# ls -l /app/easybuild/software/Python/2.7.12-foss-2016b-fh1/lib/python2.7/site-packages/blist-1.3.6-py2.7-linux-x86_64.egg/EGG-INFO/
total 16
-rw-rw-r-- 1 jfdey scicompstaff 1 Oct 3 09:23 dependency_links.txt
-rw-rw-r-- 1 jfdey scicompstaff 16 Oct 3 09:23 native_libs.txt
-rw-rw-r-- 1 jfdey scicompstaff 1 Oct 3 09:23 not-zip-safe
-rw-rw-r-- 1 jfdey scicompstaff 8790 Oct 3 09:23 PKG-INFO
-rw-rw-r-- 1 jfdey scicompstaff 644 Oct 3 09:23 SOURCES.txt
-rw-rw-r-- 1 jfdey scicompstaff 6 Oct 3 09:23 top_level.txt
root@rhino-d:/#

exts_annotate.py Annotate the exts_list from Easybuild easyconfig files. Designed to annotate, easyconfig for R, Biocondutor and Python. Use API calls to retrieve project URL and descriptioin. Create a single html page with the name of every package.

Usage: exts_annotate.py [path to easyconfig file; R-X.X.X, R-bundle-Bioconductor-X, Python-2.X]
Output html file and is always written to one of 3 file names and is based on the name of the input file. The three file names are: R.html, Python2.html, Bioconductor.html. git will track changes to the package contents with each update of the easyconfig. No updates need to be made to index.html because the html links will not change.

Python-3.5.2-foss-2016b-fh1 updated and annotated.
https://fredhutch.github.io/easybuild-life-sciences/Python3.html

https://www.bioconductor.org/packages/release/bioc/html/openCyto.html

http://scikit-learn.org/stable/install.html

Requested in YEK-819-42468

slurm-pipeline

https://Paradigm4.github.io/SciDBR

BWA/0.7.10-foss-2015b R/3.3.2-foss-2016b-fh1 libXdamage/1.1.4-foss-2016a (D)
BWA/0.7.12-foss-2015b R/3.3.2-foss-2016b-fh2 libXdmcp/1.1.2-foss-2015b
BWA/0.7.15-foss-2016b (D) R-bundle-Bioconductor/3.3-foss-2016a-R-3.3.0 libXdmcp/1.1.2-foss-2016a
BamTools/2.4.0-foss-2016a R-bundle-Bioconductor/3.3-foss-2016b-R-3.3.1-fh1 libXdmcp/1.1.2-intel-2016a (D)

these folks have a good list of tools, we can review if these are actively used at FH

http://www.vcru.wisc.edu/simonlab/bioinformatics/programs/

ADTEx

Not in CRAN or Bioc AFAICT: http://adtex.sourceforge.net/

I created a list of packages that are in my legacy python 3.5.0 install and in anaconda3 and in easybuild python 3.5.2. This is a totlal 323 packages. I compared that to what is currently in eb-3.5.2 and this is the diff. If possible we could add that all to the new python 3.6 build ?

root@rhino-d:/app/python3/3.6.0# diff -u pip-transfer-from-easybuild-3.5.2 pip-installed-on-3.6.0
--- pip-transfer-from-easybuild-3.5.2 2016-12-13 18:25:23.856110000 -0800
+++ pip-installed-on-3.6.0 2016-12-13 18:31:26.710052000 -0800
@@ -1,37 +1,83 @@
+3to2-py3k
+abstract-rendering
+aenum
+alabaster
alembic
+anaconda-client
+apiwrapper
appdirs
APScheduler
+argcomplete
argh
astroid
+astropy
+asyncio
+autopep8
+awscli
Babel
backports.shutil-get-terminal-size
beautifulsoup4
biopython
+bitarray
bitstring
+blaze
+blessings
+blinker
blist
bokeh
+boto
boto3
botocore
+bpython
+bz2file
+CacheControl
+certifi
+cffi
changeo
+chardet
click
cliff
+cloudpickle
+clyent
+cmd2
+colorama
+ConfigArgParse
+configobj
+contextlib2
cryptography
+curtsies
+cutadapt
+cx-Freeze
cycler
Cython
+cytoolz
+daemonize
darkslide
dask
+datashape
+dbf
deap
debtcollector
decorator
deepTools
+distributed
+dnspython
docutils
dominate
drmaa
+easygui
+easygui-qt
ecdsa
+entrypoints
enum34
+epydoc
+et-xmlfile
+fastcache
+fasteners
Flask
flask-appconfig
Flask-Bootstrap
+Flask-Cors
Flask-Debug
Flask-Migrate
flask-nav
@@ -40,42 +86,87 @@
Flask-WTF
freezer
funcsigs
+future
+futures
+get
github3.py
google
-h5py
+greenlet
+HeapDict
html5lib
+httplib2
idna
+imagesize
inflection
+ip-associations-python-novaclient-ext
+ipdb
+ipykernel
+ipyparallel
+ipython
ipython-genutils
+ipywidgets
iso8601
+isort
itsdangerous
+jdcal
+jedi
Jinja2
jmespath
+json2html
jsonpatch
jsonpointer
jsonschema
+jupyter
+jupyter-client
+jupyter-console
+jupyter-core
+keyring
keystoneauth1
+Kivy-Garden
+lazr.uri
lazy-object-proxy
ldap3
liac-arff
+locket
lockfile
+logilab-common
+lxml
Mako
Markdown
MarkupSafe
matplotlib
+mccabe
+mincemeat
+mistune
mock
monotonic
+moves
mpi4py
+mpmath
msgpack-python
+multipledispatch
+natsort
+nbconvert
+nbformat
netaddr
netifaces
+networkx
+nltk
nose
+notebook
npyscreen
Nuitka
+numexpr
numpy
+numpydoc
oauth
+odo
+openpyxl
openstacksdk
os-client-config
+os-diskconfig-python-novaclient-ext
+os-networksv2-python-novaclient-ext
+os-virtual-interfacesv2-python-novaclient-ext
osc-lib
oslo.concurrency
oslo.config
@@ -84,68 +175,149 @@
oslo.log
oslo.serialization
oslo.utils
+pam
pandas
paramiko
+path.py
pathtools
+patsy
paycheck
pbr
+pep8
+pexpect
pickleshare
+Pillow
pip
+ply
positional
+post
presto
prettytable
prompt-toolkit
+protobuf
+psutil
psycopg2
+ptyprocess
+public
+py
+py2bit
pyasn1
pyBigWig
-PyClone
+pycosat
+pycparser
pycrypto
pycurl
-PyDP
+pyflakes
+PyGithub
Pygments
+PyGreSQL
+pyinotify
+pylint
pymongo
pymssql
PyMySQL
+pyodbc
+pyOpenSSL
pypandoc
pyparsing
+PyQtX
+pyrax
pysam
+pyserial
+pytest
python-cinderclient
python-dateutil
+python-debian
python-editor
python-glanceclient
+python-hostlist
python-keystoneclient
python-novaclient
python-openstackclient
python-pam
python-swiftclient
+python3-ldap
pytz
+pyxdg
PyYAML
+pyzmq
qrcode
+QtAwesome
+qtconsole
+QtPy
+query-string
+rackspace-auth-openstack
+rackspace-novaclient
+rax-default-network-flags-python-novaclient-ext
+rax-scheduled-images-python-novaclient-ext
+redis
+reportlab
+request
requests
requestsexceptions
rfc3986
+roman
+rope-py3k
rpy2
+rsa
+ruffus
+s3fs
s3transfer
+scandir
+scikit-bio
+scikit-image
+scikit-learn
scipy
+SecretStorage
+setupfiles
setuptools
simplegeneric
simplejson
six
+snowballstemmer
+sockjs-tornado
+Sphinx
+sphinx-rtd-theme
+spyder
SQLAlchemy
+ssh-import-id
+statsmodels
+stdeb
stevedore
+swift
+swiftly
+sympy
+tblib
+terminado
+Theano
+toolz
+tornado
traitlets
twobitreader
tzlocal
+ujson
+unicodecsv
+uritemplate
uritemplate.py
urwid
utils
virtualenv
visitor
+wadllib
warlock
watchdog
wcwidth
webencodings
Werkzeug
wheel
+widgetsnbextension
wrapt
WTForms
+xlrd
+XlsxWriter
+xlwt
+xopen
+ZConfig
+zict
+zmq
+zope.interface

https://github.com/genome/breakdancer

This one builds many dependent packages that are also present in EasyBuild - it may be worth unrolling the dependencies into EasyBuild, or it may be better to just let it build them - external conflicts with R/Python and other commonly loaded modules should determine path.

http://singularity.lbl.gov/install-linux

there is now an EB for spark2.x could we install Spark 2.1 ?

https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/s/Spark/Spark-2.0.2.eb

afterwards we can discuss how to integrate EB python and EB R

• ARB-5.5-goolf-1.4.10.eb (Ben) (use --try-toolchain=goolf,1.5.14)
• ARB-5.5-ictce-4.1.13.eb (Ben) (use --try-toolchain=intel,2015b)
• bcl2fastq-2.17-foss-2015b.eb
• beagle-lib-88-foss-2015b.eb
• BEAST-1.8.3.eb
• BEDOPS-2.4.2-GCC-4.8.2.eb
• BEDTools-2.23.0-goolf-1.4.10.eb
• BEDTools-2.23.0-intel-2015a.eb (failed - not worth trying to fix)
• BLAST-2.2.26-Linux_x86_64.eb
• BLAT-3.5-goolf-1.4.10.eb
• Boost-1.55.0-goolf-1.7.20.eb
• Boost-1.55.0-goolf-1.7.20.eb (use --try-software-version=1.56.0)
• Boost-1.58.0-foss-2015b.eb
• Bowtie-1.0.0-goolf-1.4.10.eb
• Bowtie-1.1.1-goolf-1.4.10.eb (fails to build)
• Bowtie-1.1.1-goolf-1.4.10.eb (use --try-toolchain=intel,2015b) (failed - not worth fixing)
• Bowtie2-2.2.2-goolf-1.4.10.eb (use --try-software-version=2.2.3)
• Bowtie2-2.2.4-goolf-1.4.10.eb
• Bowtie2-2.2.5-goolf-1.7.20.eb
• Bowtie2-2.2.5-intel-2015a.eb
• BWA-0.7.5a-goolf-1.4.10.eb (use --try-software-version=0.7.10)
• BWA-0.7.5a-goolf-1.4.10.eb (use --try-software-version=0.7.12)
• CD-HIT-4.6.1-goolf-1.4.10-2012-08-27.eb (use --try-software-version=4.6.3) (version changed to drop date, so --try-software-version does not work)
• ClustalW2-2.1-goolf-1.4.10.eb (use --try-software-version=2.16.0)
• Cufflinks-2.2.1-goolf-1.7.20.eb
• Cufflinks-2.2.1-intel-2015a.eb (failed on libgmp for python/2.7.8-intel,2015b)
• EMBOSS-6.5.7-goolf-1.4.10.eb (use --try-software-version=6.6.0) (failed - EMBOSS 6.6.0 is gone)
• fastqc-0.11.2.eb
• fastqc-0.11.3.eb
• FastTree-2.1.7-goolf-1.4.10.eb (use --try-software-version=2.1.8)
• GATK-3.3-0-Java-1.7.0_80.eb (use --try-software-version=3.4.0)
• GATK-3.5-Java-1.8.0_66.eb
• GMAP-GSNAP-2014-06-10-goolf-1.4.10.eb (use --try-software-version=2014-08-20)
• GMAP-GSNAP-2014-06-10-goolf-1.4.10.eb (use --try-software-version=2014-12-16)
• GMAP-GSNAP-2014-06-10-goolf-1.4.10.eb (use --try-software-version=2014-12-17)
• GMAP-GSNAP-2014-06-10-goolf-1.4.10.eb (use --try-software-version=2014-12-29)
• HDF5-1.8.13-foss-2015a.eb
• HDF5-1.8.13-intel-2015a.eb
• HDF5-1.8.15-foss-2015a.eb
• HDF5-1.8.15-intel-2015b.eb
• Java-1.7.0_80.eb
• Java-1.8.0_66.eb
• MATLAB-2015a.eb (licensed - will try to build later)
• MEME-4.8.0-goolf-1.4.10.eb (use --try-software-version=4.10.0) (failed - source URL has changed)
• MUSCLE-3.8.31-goolf-1.4.10.eb
• MUSCLE-3.8.31-ictce-5.5.0.eb (use --try-toolchain=intel,2015b)
• NAMD-2.9-goolf-1.4.10-ibverbs.eb
• NAMD-2.9-ictce-5.5.0-ibverbs.eb (use --try-toolchain=intel,2015b)
• NCO-4.4.4-intel-2014b.eb
• netCDF-4.3.2-foss-2015a.eb
• netCDF-4.3.2-intel-2015a.eb
• Perl-5.22.0-foss-2015b-bare.eb
• Perl-5.22.0-intel-2015b-bare.eb
• picard-1.119.eb (use --try-software-version=1.131)
• PLINK-1.07-x86_64.eb (use --try-software-version=1.90)
• Queue-3.5-Java-1.8.0_66.eb
• SAMtools/1.2-foss-2015b
• SAMtools-1.2-intel-2015a-HTSlib-1.2.1.eb (use --try-toolchain=intel,2015b)
• Tracer_v1.6.eb
• TopHat-2.0.13-goolf-1.7.20.eb (use --try-toolchain=foss,2015b)
• TopHat-2.0.14-goolf-1.7.20.eb (use --try-toolchain=foss,2015b)
• TopHat/2.1.1-foss-2015b
• zlib-1.2.8-foss-2015b.eb
• zlib-1.2.8-intel-2015b.eb

== 2016-11-27 19:09:06,746 environment.py:97 INFO Environment variable EBVARSCALAPACK_MT_STATIC_LIBS set to libscalapack.a,libopenblas.a,libgfortran.a,libpthread.a (previous value: 'libscalapack.a,libopenblas.a,libgfortran.a,libpthread.a')
== 2016-11-27 19:09:06,746 environment.py:97 INFO Environment variable SCALAPACK_STATIC_LIBS set to libscalapack.a,libopenblas.a,libgfortran.a (previous value: 'libscalapack.a,libopenblas.a,libgfortran.a')
== 2016-11-27 19:09:06,746 environment.py:97 INFO Environment variable EBVARSCALAPACK_STATIC_LIBS set to libscalapack.a,libopenblas.a,libgfortran.a (previous value: 'libscalapack.a,libopenblas.a,libgfortran.a')
== 2016-11-27 19:09:06,746 filetools.py:229 INFO Command python found at /easybuild/software/Python/2.7.12-foss-2016b-fh2/bin/python
== 2016-11-27 19:09:06,747 pythonpackage.py:289 INFO Python command being used: /easybuild/software/Python/2.7.12-foss-2016b-fh2/bin/python
== 2016-11-27 19:09:06,816 filetools.py:953 INFO Creating directory /easybuild/build/Python/2.7.12/foss-2016b-fh2/rpy2 (parents: True, set_gid: False, sticky: False)
== 2016-11-27 19:09:06,886 environment.py:97 INFO Environment variable PYTHONNOUSERSITE set to 1 (previous value: '1')
== 2016-11-27 19:09:07,217 build_log.py:149 ERROR EasyBuild crashed with an error (at easybuild/software/EasyBuild/3.0.0/lib/python2.7/site-packages/vsc_base-2.5.5-py2.7.egg/vsc/utils/exceptions.py:124 in init): cmd " /easybuild/software/Python/2.7.12-foss-2016b-fh2/bin/python setup.py build " exited with exitcode 1 and output:
Warning: Tried to guess R's HOME but no command (R) in the PATH.
(at easybuild/software/EasyBuild/3.0.0/lib/python2.7/site-packages/easybuild_framework-3.0.0-py2.7.egg/easybuild/tools/run.py:438 in parse_cmd_output)
== 2016-11-27 19:09:07,218 easyblock.py:2498 WARNING build failed (first 300 chars): cmd " /easybuild/software/Python/2.7.12-foss-2016b-fh2/bin/python setup.py build " exited with exitcode 1 and output:
Warning: Tried to guess R's HOME but no command (R) in the PATH.

• rhino-d Intel license
• toolchains better explained
• versionsuffix better explained
• easybuild upgrade process
• easyconfig authoring best practices
• prereq command line
• modules install

in Python-3.5.2-foss-2016b-fh1.eb Cython-0.23.4.tar.gz cannot be found

it is no longer hosted here : http://www.cython.org/release/Cython-0.23.4.tar.gz

This will be fixed with Python-3.5.2-foss-2016b-fh2.eb and Cython-0.25.1. Cython has moved to pypi and only the newest version is available from pypi. We can continue to build old versions from stored local downloads.
/app/easybuild/sources/p/Python/extensions/Cython-0.21.2.tar.gz
/app/easybuild/sources/p/Python/extensions/Cython-0.22.tar.gz
/app/easybuild/sources/p/Python/extensions/Cython-0.20.2.tar.gz
/app/easybuild/sources/p/Python/extensions/Cython-0.23.2.tar.gz
/app/easybuild/sources/p/Python/extensions/Cython-0.23.4.tar.gz
/app/easybuild/sources/p/Python/extensions/Cython-0.24.tar.gz

It is a Bioconductor packge.

requester: Li Hsu
ticket: Office Hours
sources: http://homer.ucsd.edu/homer/download.html
(not available from easybuild)

http://qiime.org/

http://www.thegpm.org/TANDEM/index.html

easybuild easyconfig is written and built: tandem-foss-2016b.eb
module load tandem
To execute tandum use: $EBROOTTANDEM/bin/tandem.exe

just saw that a bunch of packages built for python2 have not made it into python3 (e.g. qiime scikit-learn)

petersen@rhino2:/home…-life-sciences$ diff -u /tmp/p2.list /tmp/p3.list
--- /tmp/p2.list 2017-01-28 06:05:42.842944308 -0800
+++ /tmp/p3.list 2017-01-28 06:08:32.769380146 -0800
@@ -1,212 +1,283 @@
+abstract-rendering (0.5.1)
+aenum (1.4.5)
+alabaster (0.7.9)
alembic (0.8.10)
-alnvu (0.1.0)
+anaconda-client (1.2.2)
+apiwrapper (0.1.7)
appdirs (1.4.0)
APScheduler (3.3.1)
+argcomplete (1.7.0)
argh (0.26.2)
astroid (1.4.9)
-awscli (1.11.43)
+astropy (1.3)
+autopep8 (1.2.4)
+awscli (1.11.35)
Babel (2.3.4)
backports.shutil-get-terminal-size (1.0.0)
-bcbio-gff (0.6.4)
-beautifulsoup4 (4.5.1)
-biom-format (2.1.5)
+beautifulsoup4 (4.5.3)
biopython (1.68)
+bitarray (0.8.1)
bitstring (3.1.5)
-bokeh (0.12.3)
-boto3 (1.4.3)
-botocore (1.5.6)
-burrito (0.9.1)
-burrito-fillings (0.1.1)
+blaze (0.10.1)
+bleach (1.5.0)
+blessings (1.6)
+blinker (1.4)
+blist (1.3.6)
+bokeh (0.12.4)
+boto3 (1.4.4)
+botocore (1.4.92)
+bpython (0.16)
+bz2file (0.98)
+CacheControl (0.11.7)
certifi (2017.1.23)
cffi (1.9.1)
+chardet (2.3.0)
click (6.7)
-cliff (2.3.0)
+cliff (2.4.0)
+cloudpickle (0.2.2)
+clyent (1.2.1)
cmd2 (0.6.9)
-cogent (1.5.3)
colorama (0.3.7)
-colorbrewer (0.1.1)
ConfigArgParse (0.11.0)
+configobj (5.0.6)
+contextlib2 (0.5.4)
cryptography (1.7.1)
+curtsies (0.2.11)
cutadapt (1.12)
cycler (0.10.0)
Cython (0.25.2)
+cytoolz (0.8.2)
+daemonize (2.4.7)
darkslide (2.3.3)
-datacache (0.4.20)
+dask (0.13.0)
+datashape (0.5.2)
+dbf (0.96.8)
deap (1.0.2)
debtcollector (1.10.0)
decorator (4.0.10)
deepTools (2.4.2)
+distributed (1.15.0)
+dnspython (1.15.0)
docutils (0.13.1)
dominate (2.3.1)
drmaa (0.7.6)
-easygui (0.98.0)
+easygui (0.98.1)
ecdsa (0.13)
-emperor (0.9.60)
-enum34 (1.1.6)
+epydoc (3.0.1)
+et-xmlfile (1.0.1)
+fastcache (1.0.2)
fasteners (0.14.1)
Flask (0.12)
flask-appconfig (0.11.1)
-Flask-Bootstrap (3.3.7.0)
+Flask-Bootstrap (3.3.7.1)
+Flask-Cors (3.0.2)
Flask-Debug (0.4.3)
Flask-Migrate (2.0.2)
flask-nav (0.6)
Flask-Script (2.0.5)
Flask-SQLAlchemy (2.1)
-Flask-WTF (0.13.1)
+Flask-WTF (0.14.2)
freezer (4.0.0.0b2)
funcsigs (1.0.2)
-functools32 (3.2.3.post2)
future (0.16.0)
-futures (3.0.5)
-gdata (2.0.18)
-gitdb2 (2.0.0)
-GitPython (2.1.1)
-google (1.9.3)
-gtfparse (0.0.6)
+greenlet (0.4.11)
h5py (2.7.0rc2)
-html5lib (0.999999999)
-HTSeq (0.6.1)
+HeapDict (1.0.0)
+html5lib (0.9999999)
+httplib2 (0.9.2)
idna (2.2)
+imagesize (0.7.1)
inflection (0.3.1)
-ipaddress (1.0.18)
+ip-associations-python-novaclient-ext (0.2)
+ipdb (0.10.1)
+ipykernel (4.5.2)
ipython (5.1.0)
ipython-genutils (0.1.0)
iso8601 (0.1.11)
+isort (4.2.5)
itsdangerous (0.24)
+jdcal (1.3)
+jedi (0.9.0)
Jinja2 (2.9.4)
jmespath (0.9.0)
+json2html (1.1.1)
jsonpatch (1.15)
jsonpointer (1.10)
jsonschema (2.5.1)
-keystoneauth1 (2.16.0)
+keyring (10.1)
+keystoneauth1 (2.18.0)
+lazr.uri (1.0.3)
lazy-object-proxy (1.2.2)
-ldap3 (2.1.1)
+ldap3 (2.2.0)
liac-arff (2.1.0)
+locket (0.2.0)
lockfile (0.12.2)
-lxml (3.7.2)
+logilab-common (1.3.0)
+lxml (3.7.1)
Mako (1.0.6)
Markdown (2.6.8)
MarkupSafe (0.23)
-matplotlib (2.0.0rc2)
-memoized-property (1.0.3)
+matplotlib (2.0.0)
+mccabe (0.5.3)
mincemeat (0.1.4)
-misopy (0.5.3)
+mistune (0.7.3)
mock (2.0.0)
monotonic (1.2)
+moves (0.1)
mpi4py (2.0.0)
+mpmath (0.19)
msgpack-python (0.4.8)
-natsort (3.5.6)
-ndg-httpsclient (0.4.2)
+multipledispatch (0.4.9)
+natsort (5.0.1)
netaddr (0.7.18)
netifaces (0.10.5)
+networkx (1.11)
+nltk (3.2.2)
nose (1.3.7)
+novaclient-auth-secretkey (0.1)
npyscreen (4.10.5)
Nuitka (0.5.25)
-numpy (1.12.0rc1)
+numexpr (2.6.1)
+numpy (1.12.0)
+numpydoc (0.6.0)
oauth (1.0.1)
-olefile (0.44)
-openstacksdk (0.9.11)
-os-client-config (1.24.0)
-osc-lib (1.2.0)
-oslo.concurrency (3.16.0)
-oslo.config (3.21.0)
-oslo.context (2.11.0)
-oslo.i18n (3.11.0)
+odo (0.5.0)
+olefile (0.43)
+openpyxl (2.4.1)
+openstacksdk (0.9.13)
+os-client-config (1.26.0)
+os-diskconfig-python-novaclient-ext (0.1.3)
+os-networksv2-python-novaclient-ext (0.26)
+os-virtual-interfacesv2-python-novaclient-ext (0.20)
+osc-lib (1.3.0)
+oslo.concurrency (3.18.0)
+oslo.config (3.22.0)
+oslo.context (2.12.0)
+oslo.i18n (3.12.0)
oslo.log (3.19.0)
-oslo.serialization (2.15.0)
-oslo.utils (3.21.0)
+oslo.serialization (2.16.0)
+oslo.utils (3.22.0)
packaging (16.8)
pandas (0.19.2)
+pandocfilters (1.4.1)
paramiko (2.1.1)
-parse (1.6.6)
-pathlib2 (2.2.1)
+path.py (10.1)
pathtools (0.1.2)
-paycheck (1.0.2)
+patsy (0.4.1)
pbr (1.10.0)
+pep8 (1.7.0)
pexpect (4.2.1)
pickleshare (0.7.4)
Pillow (4.0.0)
pip (9.0.1)
+ply (3.9)
positional (1.1.1)
+post-0 (0.0.0)
prettytable (0.7.2)
-progressbar33 (2.4)
prompt-toolkit (1.0.9)
+protobuf (3.2.0rc2)
psutil (5.0.1)
psycopg2 (2.6.2)
ptyprocess (0.5.1)
+public-0 (0.0.0)
+py (1.4.32)
+py2bit (0.2.0)
pyasn1 (0.1.9)
-pyastro (1.1)
pyBigWig (0.3.2)
PyClone (0.12.9)
+pycosat (0.6.1)
pycparser (2.17)
pycrypto (2.6.1)
pycurl (7.43.0)
PyDP (0.2.2)
-pyensembl (1.0.3)
-pygithub3 (0.5.1)
+pyflakes (1.4.0)
+PyGithub (1.29)
Pygments (2.2.0)
+PyGreSQL (5.0.3)
pyinotify (0.9.6)
+pylint (1.6.4)
pymongo (3.4.0)
pymssql (2.1.3)
PyMySQL (0.7.9)
-pynast (1.2.2)
+pyodbc (4.0.0)
pyOpenSSL (16.2.0)
pypandoc (1.3.3)
pyparsing (2.1.10)
-pyqi (0.3.2)
+PyQtX (0.1.2)
+pyrax (1.9.8)
pysam (0.9.1.4)
-python-cinderclient (1.9.0)
+pyserial (3.2.1)
+pytest (3.0.6)
+python-cinderclient (1.10.0)
python-dateutil (2.6.0)
+python-debian (0.1.28)
python-editor (1.0.3)
python-freezerclient (1.1.0)
-python-glanceclient (1.1.2)
-python-keystoneclient (3.8.0)
-python-memcached (1.58)
+python-glanceclient (2.6.0)
+python-hostlist (1.17)
+python-keystoneclient (3.10.0)
python-novaclient (7.0.0)
-python-openstackclient (3.6.0)
+python-openstackclient (3.7.0)
python-pam (1.8.2)
python-swiftclient (3.2.0)
pytz (2016.10)
-PyVCF (0.6.8)
+pyxdg (0.25)
PyYAML (3.12)
-qcli (0.1.1)
-qiime (1.9.1)
-qiime-default-reference (0.1.3)
+pyzmq (16.0.2)
qrcode (5.3)
+QtAwesome (0.4.3)
+qtconsole (4.2.1)
+QtPy (1.2.1)
+rackspace-auth-openstack (1.3)
+rackspace-novaclient (2.1)
+rax-default-network-flags-python-novaclient-ext (0.4.0)
+rax-scheduled-images-python-novaclient-ext (0.3.1)
+redis (2.10.5)
reportlab (3.3.0)
-requests (2.12.4)
+request-0 (0.0.0)
+requests (2.13.0)
requestsexceptions (1.1.3)
rfc3986 (0.4.1)
-rpy2 (2.8.5)
+roman (2.0.0)
rsa (3.4.2)
+ruffus (2.6.3)
+s3fs (0.0.8)
s3transfer (0.1.10)
scandir (1.4)
-scikit-bio (0.2.3)
-scikit-learn (0.18.1)
scipy (0.18.1)
-seqmagick (0.6.1)
-serializable (0.1.1)
-setuptools (32.2.0)
-shove (0.6.6)
+SecretStorage (2.3.1)
+setuptools (28.8.0)
simplegeneric (0.8.1)
simplejson (3.10.0)
singledispatch (3.4.0.3)
six (1.10.0)
-smmap2 (2.0.1)
+snowballstemmer (1.2.1)
+sockjs-tornado (1.0.3)
+sortedcollections (0.4.2)
+sortedcontainers (1.5.7)
+Sphinx (1.5.2.dev20170126)
+spyder (3.1.2)
SQLAlchemy (1.1.5)
+ssh-import-id (5.6)
+stdeb (0.8.5)
stevedore (1.19.1)
-stuf (0.9.16)
-subprocess32 (3.2.7)
-tinytimer (0.0.0)
+swiftly (2.6)
+sympy (1.0)
+tblib (1.3.0)
+terminado (0.6)
+toolz (0.8.2)
+tornado (4.4.2)
traitlets (4.3.1)
twobitreader (3.1.4)
-typechecks (0.0.2)
tzlocal (1.3)
+ujson (1.35)
unicodecsv (0.14.1)
uritemplate (3.0.0)
urwid (1.3.1)
+utils (0.9.0)
virtualenv (15.1.0)
visitor (0.1.3)
+wadllib (1.3.2)
warlock (1.3.0)
watchdog (0.8.3)
wcwidth (0.1.7)
@@ -215,4 +286,10 @@
wheel (0.30.0a0)
wrapt (1.10.8)
WTForms (2.1)
+xlrd (1.0.0)
+XlsxWriter (0.9.6)
xopen (0.1.1)
+ZConfig (3.1.0)
+zict (0.1.1)
+zmq (0.0.0)
+zope.interface (4.3.3)

please add these packages to our python easy builds

• dms_tools
• pydot
• pam.py
• mako
• markdown
• nuitka
• python-swiftclient
• python-keystoneclient
• python-novaclient
• flask
• flask_wtf
• flask_bootstrap
• flask_appconfig
• flask_debug
• flask_nav
• flask_script
• flask_sqalchemy
• flask_migrate
• psycopg2
• pypandoc
• pip
• darkslide
• watchdog
• requests
• pycurl
• ipython (removed from 2.7.12)
• pandas
• flask_sqlalchemy
• bokeh
• boto3
• debian (has issues)
• docutils
• drmaa
• easygui
• freezer
• github3
• google
• h5py
• html5lib
• ldap3
• matplotlib
• mincemeat.py
• mpi4py
• npyscreen
• Nuitka
• numpy
• oauth
• pandas
• paramiko
• pymongo
• PyMySQL
• pymssql
• urwid
• yaml

rslurm is in CRAN now (as of June)

Add misopy to python builds

https://github.com/etal/cnvkit

peddy
networkx
cyvcf2
python-snappy
geneimpacts

The ubuntu package provides libtiff 4 backwards compatibility in the libtiff5 library. Thus, many tools and libraries using libtiff (libpoppler is an example) are breaking when modules are loaded.

xpdf fails with:

$ xpdf
xpdf.real: /app/easybuild/software/LibTIFF/4.0.6-foss-2016b/lib/libtiff.so.5: version `LIBTIFF_4.0' not found (required by /usr/lib/x86_64-linux-gnu/libpoppler.so.44)

$ ldd /usr/lib/x86_64-linux-gnu/libpoppler.so.44
/usr/lib/x86_64-linux-gnu/libpoppler.so.44: /app/easybuild/software/LibTIFF/4.0.6-foss-2016b/lib/libtiff.so.5: version `LIBTIFF_4.0' not found (required by /usr/lib/x86_64-linux-gnu/libpoppler.so.44)
	linux-vdso.so.1 =>  (0x00007ffc5c94e000)
	liblcms2.so.2 => /usr/lib/x86_64-linux-gnu/liblcms2.so.2 (0x00007f45ea6ed000)
	libtiff.so.5 => /app/easybuild/software/LibTIFF/4.0.6-foss-2016b/lib/libtiff.so.5 (0x00007f45eaf28000)
..... snipped ...

lxc container eb-14.04-v1 has some permissions issues in /app. Specifically, some sub-directories do not have write permissions for group which causes easybuild to fail at times.

Workaround is to log into container as root and chmod/chown /app recursively.

ticket 94518
requester: Alli Black
R9.4 models available here:
mkdir models && cd models && wget http://s3.climb.ac.uk/nanopore/nanopolish_r94models.tar && tar xvf nanopolish_r94models.tar

• make eb put successfully built easyconfigs (including robo dependencies) into a separate folder
• link github repo to zenodo for auto archiving on release
• commit the separate easyconfigs, plus the 'sources' directory to repo

This gets us actual reproducibility as all the easyconfigs along with the sources will be in the repo (and then the archive with Zenodo).

kyotil is in CRAN and should be an easy addition

Import excel files into R. Supports '.xls' via the embedded 'libxls' C library <

messytables
toil
elasticluster (2.7 only)
pyinstaller

ticket number: 94518
requester: Alli Black

git clone --recursive https://github.com/jts/nanopolish.git && cd nanopolish && make -j8

easybuild_bootstrap.sh complained when I ran as root, so I made some changes and it now shows this error when I try to run eb after logout / login :

eb@ebtest3:$ module load EasyBuild
eb@ebtest3:$ eb
== temporary log file in case of crash /tmp/eb-r5mvnj/easybuild-dmum1p.log
ERROR: EnvironmentModulesC modules tool can not be used, 'modulecmd' command is not available.
eb@ebtest3:~$

HDC is getting more interested in microbiome and we already have some tools on our todo list

http://qiime.org/