cmbi / hommod Goto Github PK
View Code? Open in Web Editor NEWHomology Modeling Service
Homology Modeling Service
The update scripts produce models before they are needed so future requests are quick. When a model doesn't exist, it's produced on-the-fly which is time consuming.
The update script was being run on bamboo
but the produced files were never used by the frontend on chelonium
. The service has since been removed from bamboo because it makes more sense to run the script on chelonium.
The docker scripts need updating to add the updating as a separate process. Care needs to be taken to ensure it doesn't slow the machine down too much.
HOPE gets the following error from HOMMOD:
Exception: the following models have failed:
f1a9c8ce-7bb9-feda-7ad6-f527c0b3de8f_HUMAN_175-298_3NOG-D:
Traceback (most recent call last):
File "/usr/src/app/hommod_rest/services/model.py", line 967, in modelProc
main_domain_alignment, main_domain_range)
File "/usr/src/app/hommod_rest/services/model.py", line 465, in _collect_template
raise e
IOError: [Errno ftp error] 200 TYPE is now 8-bit binary
What is the cause?
I get the following error in HOPE from hommod:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hope/factory.py", line 171, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hope/tasks.py", line 206, in get_structure_features
model = modeler.from_sequence(sequence, position, template_id)
File "/usr/src/app/hope/domain/modeler.py", line 14, in from_sequence
template_id)
File "/usr/src/app/hope/services/hommod.py", line 68, in run
raise ServiceError(json.loads(r.text)['message'])
hope.services.types.ServiceError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 35, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 1003, in modelProc
_log.debug("ending lock on {}".format(lockfile_path))
UnboundLocalError: local variable 'lockfile_path' referenced before assignment
The get_metadata
endpoint has two related issues:
When a job is started using the frontend's submit method, followed by a call to the status method (see: https://github.com/cmbi/hommod-rest/blob/master/hommod_rest/frontend/api/endpoints.py#L38), it does not always return the right status. Status STARTED has been observed for jobs that had already finished or failed.
There are a few functions that have O(n^3)
or worse running time (based on a quick analysis, it's actually more complex than that). In any case, these functions have 4-6 nested loops.
HOPE is getting a lot of internal server errors from hommod. In the logs I see:
[2016-03-02 08:26:38 +0000] [2167] [ERROR] Error handling request
Traceback (most recent call last):
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/gunicorn/workers/async.py", line 52, in handle
self.handle_request(listener_name, req, client, addr)
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 159, in handle_request
super(GeventWorker, self).handle_request(*args)
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/gunicorn/workers/async.py", line 105, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/srv/www/hommod/hommod-rest/hommod_rest/middleware.py", line 44, in __call__
return self.app(environ, start_response)
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/flask/app.py", line 1478, in full_dispatch_request
response = self.make_response(rv)
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/flask/app.py", line 1577, in make_response
rv = self.response_class.force_type(rv, request.environ)
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/werkzeug/wrappers.py", line 841, in force_type
response = BaseResponse(*_run_wsgi_app(response, environ))
File "/srv/www/hommod/.virtualenvs/hommod-rest/local/lib/python2.7/site-packages/werkzeug/test.py", line 867, in run_wsgi_app
app_rv = app(environ, start_response)
TypeError: 'dict' object is not callable
For some models, yasara deletes molecules from the template while it's modeling. This leads to an error, because the number of molecules no longer matches the number of alignments.
Here's an example:
target sequence:
MGSKGVYQYHWQSHNVKHSGVDDMVLLSKITENSIVENLKKRYMDDYIFTYIGSVLISVNPFKQMPYFGEKEIEMYQGAAQYENPPHIYALADNMYRNMIIDRENQCVIISGESGAGKTVAAKYIMSYISRVSGGGTKVQHVKDIILQSNPLLEAFGNAKTVRNNNSSRFGKYFEIQFSPGGEPDGGKISNFLLEKSRVVMRNPGERSFHIFYQLIEGASAEQKHSLGITSMDYYYYLSLSGSYKVDDIDDRREFQETLHAMNVIGIFAEEQTLVLQIVAGILHLGNISFKEVGNYAAVESEEFLAFPAYLLGINQDRLKEKLTSRQMDSKWGGKSESIHVTLNVEQACYTRDALAKALHARVFDFLVDSINKAMEKDHEEYNIGVLDIYGFEIFQKNGFEQFCINFVNEKLQQIFIELTLKAEQEEYVQEGIRWTPIEYFNNKIVCDLIENKVNPPGIMSILDDVCATMHAVGEGADQTLLQKLQMQIGSHEHFNSWNQGFIIHHYAGKVSYDMDGFCERNRDVLFMDLIELMQSSELPFIKSLFPENLQADKKGRPTTAGSKIKKQANDLVSTLMKCTPHYIRCIKPNETKKPRDWEESRVKHQVEYLGLKENIRVRRAGYAYRRIFQKFLQRYAILTKATWPSWQGEEKQGVLHLLQSVNMDSDQFQLGRSKVFIKAPESLFLLEEMRERKYDGYARVIQKSWRKFVARKKYVQMREEASDLLLNKKERRRNSINRNFIGDYIGMEEHPELQQFVGKREKIDFADTVTKYDRRFKGVKRDLLLTPKCLYLIGREKVKQGPDKGLVKEVLKRKIEIERILSVSLSTMQDDIFILHEQEYDSLLESVFKTEFLSLLAKRYEEKTQKQLPLKFSNTLELKLKKENWGPWSAGGSRQVQFHQGFGDLAVLKPSNKVLQVSIGPGLPKNSRPTRRNTTQNTGYSSGTQNANYPVRAAPPPPGYHQNGVIRNQYVPYPHAPGSQRSNQKSLYTSMARPPLPRQQSTSSDRVSQTPESLDFLKVPDQGAAGVRRQTTSRPPPAGGRPKPQPKPKPQVPQCKALYAYDAQDTDELSFNANDIIDIIKEDPSGWWTGRLRGKQGLFPNNYVTKI
position: 346
species: HUMAN
template: 6C1D_P
Whenever there's an error parsing the cookie, the badly-formed cookie simply stays and parsing keeps failing. Hommod should delete bad cookies and start with an empty object again.
Message type: ERROR
Location: /usr/src/app/hope/services/hommod.py:68
Module: hommod
Function: run
Time: 2017-03-28 10:03:59,680
Message:
Model job failed:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 60, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 786, in modelProc
ranges = interpro.get_domain_locations(main_target_sequence)
File "/usr/src/app/hommod_rest/services/interpro.py", line 198, in get_domain_locations
filepath = self._create_data_file(sequence)
File "/usr/src/app/hommod_rest/services/interpro.py", line 172, in _create_data_file
status = _interpro_get_status(jobid)
File "/usr/src/app/hommod_rest/services/interpro.py", line 83, in _interpro_get_status
return _interpro_get(_interpro_base_url + '/status/' + jobid)
File "/usr/src/app/hommod_rest/services/interpro.py", line 63, in _interpro_get
reqH = urllib2.urlopen(req)
File "/usr/local/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/usr/local/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/local/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
The following error happened via a call from HOPE:
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 33, in create_model
paths = modeler.modelProc (sequence, species_id, residue_number, False)
File "/usr/src/app/hommod_rest/services/model.py", line 824, in modelProc
domainalign.getAlignments (ranges, mainTargetSeq)
File "/usr/src/app/hommod_rest/services/domainalign.py", line 709, in getAlignments
if not secstr.hasSecStr(template):
File "/usr/src/app/hommod_rest/services/secstr.py", line 62, in hasSecStr
obj = self.yasara.LoadPDB (pdbfile)[0]
File "/deps/yasara/yasara/pym/yasaramodule.py", line 4773, in LoadPDB
return(run(command[:-1]))
File "/deps/yasara/yasara/pym/yasaramodule.py", line 1012, in run
if (command.find("\n")==-1): return(process("EXECUTE",command))
File "/deps/yasara/yasara/pym/yasaramodule.py", line 1006, in process
result=com.receivemessage(com.RESULT)
File "/deps/yasara/yasara/pym/yasaramodule.py", line 812, in receivemessage
raise RuntimeError("YASARA raised error %d: %s"%(messagedata[0],messagedata[1]))
RuntimeError: YASARA raised error 11: The file '/data/tmp/run-yasara-HUMAN-edadbbc0-7d72-e6ed-53f0-ad578d6b661e-51/3B8C.pdb' could not be read, make sure that the file is present and that you have read permissions.
HOPE's input sequence:
MGRGAGREYSPAATTAENGGGKKKQKEKELDELKKEVAMDDHKLSLDELGRKYQVDLSKGLTNQRAQDVLARDGPNALTPPPTTPEWVKFCRQLFGGFSILLWIGAILCFLAYGIQAAMEDEPSNDNLYLGVVLAAVVIVTGCFSYYQEAKSSKIMDSFKNMVPQQALVIREGEKMQINAEEVVVGDLVEVKGGDRVPADLRIISSHGCKVDNSSLTGESEPQTRSPEFTHENPLETRNICFFSTNCVEGTARGIVIATGDRTVMGRIATLASGLEVGRTPIAMEIEHFIQLITGVAVFLGVSFFVLSLILGYSWLEAVIFLIGIIVANVPEGLLATVTVCLTLTAKRMARKNCLVKNLEAVETLGSTSTICSDKTGTLTQNRMTVAHMWFDNQIHEADTTEDQSGATFDKRSPTWTALSRIAGLCNRAVFKAGQENISVSKRDTAGDASESALLKCIELSCGSVRKMRDRNPKVAEIPFNSTNKYQLSIHEREDSPQSHVLVMKGAPERILDRCSTILVQGKEIPLDKEMQDAFQNAYMELGGLGERVLGFCQLNLPSGKFPRGFKFDTDELNFPTEKLCFVGLMSMIDPPRAAVPDAVGKCRSAGIKVIMVTGDHPITAKAIAKGVGIISEGNETVEDIAARLNIPMSQVNPREAKACVVHGSDLKDMTSEQLDEILKNHTEIVFARTSPQQKLIIVEGCQRQGAIVAVTGDGVNDSPALKKADIGIAMGISGSDVSKQAADMILLDDNFASIVTGVEEGRLIFDNLKKSIAYTLTSNIPEITPFLLFIIANIPLPLGTVTILCIDLGTDMVPAISLAYEAAESDIMKRQPRNSQTDKLVNERLISMAYGQIGMIQALGGFFTYFVILAENGFLPSRLLGIRLDWDDRTMNDLEDSYGQEWTYEQRKVVEFTCHTAFFASIVVVQWADLIICKTRRNSVFQQGMKNKILIFGLLEETALAAFLSYCPGMGVALRMYPLKVTWWFCAFPYSLLIFIYDEVRKLILRRYPGGWVEKETYY
Mutation: 383H
Update the README to reflect the changes in deployment because of docker.
There are errors in the integrations tests which need fixing. These tests haven't run for a long time, so they may be obsolete.
Right now, hommod only has a rest api for building models. It could use a web frontend.
HOPE gets this error back from hommod:
f551ff58-c277-6de1-376b-4f8830643b0f_HUMAN_40-263:
Traceback (most recent call last):
File "/usr/src/app/hommod_rest/services/model.py", line 929, in modelProc
mainTargetSeq, mainDomainRange)
File "/usr/src/app/hommod_rest/services/model.py", line 468, in _build_for_domain
self._set_template (mainTemplateID.pdbac)
File "/usr/src/app/hommod_rest/services/model.py", line 390, in _set_template
raise Exception ("chain %s occurs more than once after cleaning" % chain)
Exception: chain A occurs more than once after cleaning
The position is 227
and the sequence is:
MPKKKPTPIQLNPAPDGSAVNGTSSAETNLEALQKKLEELELDEQQRKRLEAFLTQKQKVGELKDDDFEK
ISELGAGNGGVVFKVSHKPSGLVMARKLIHLEIKPAIRNQIIRELQVLHECNSPYIVGFYGAFYSDGEIS
ICMEHMDGGSLDQVLKKAGRIPEQILGKVSIAVIKGLTYLREKHKIMHRDVKPSNILVNSRGEIKLCDFG
VSGQLIDSMANSFVGTRSYMSPERLQGTHYSVQSDIWSMGLSLVEMAVGRYPIPPPDAKELELMFGCQVE
GDAAETPPRPRTPGRPLSSYGMDSRPPMAIFELLDYIVNEPPPKLPSGVFSLEFQDFVNKCLIKNPAERA
DLKQLMVHAFIKRSDAEEVDFAGWLCSTIGLNQPSTPTHAAGV
What does this error mean? Is this a bug in hommod or can't hommod do anything about it?
The following error appeared in HOPE:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 60, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 1014, in modelProc
raise Exception("the following models have failed:\n" + s)
Exception: the following models have failed:
9a6424c4-4096-d62d-4150-e38ebec3799b_HUMAN_32-188_3UX9-A:
Traceback (most recent call last):
File "/usr/src/app/hommod_rest/services/model.py", line 944, in modelProc
main_target_sequence, main_domain_range)
File "/usr/src/app/hommod_rest/services/model.py", line 628, in _build_for_domain
targetsInterproRanges, picker)
File "/usr/src/app/hommod_rest/services/domainalign.py", line 439, in pickAlignments
if picker.accepts(targetID, alignment):
File "/usr/src/app/hommod_rest/services/interaction.py", line 80, in accepts
sstart, send = get_target_covered_range(subjectChainAlignment, sseq)
File "/usr/src/app/hommod_rest/services/modelutils.py", line 812, in get_target_covered_range
% (alignment['template'], template_seq))
Exception: mismatch between alignment and template sequence:
IVLTQPPSVSGAPGQRVTISCSGSSSNIGSNYVSWYQQLPGTAPKLLIYDNNQRPSGVPDRFSGSKSGTSASLAITGLQSEDEADYYCQVRDNNENEWVFGGGTKLTVLEVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKGLEWVSAISGSGGSTYYADSVKGRFTISRDNSKNTLYLQMNSLRAEDTAVYYCARYIDFGDHMDFWGQGTLVTVSSL
IVLTQPPSVSGAPGQRVTISCSGSSSNIGSNYVSWYQQLPGTAPKLLIYDNNQRPSGVPDRFSGSKSGTSASLAITGLQSEDEADYYCQVRDNNENEWVFGGGTKLTVLEVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKGLEWVSAISGSGGSTYYADSVKGRFTISRDNSKNTLYLQMNSLRAEDTAVYYCARYIDFGDHMDFWGQGTLVTVSSLE
9a6424c4-4096-d62d-4150-e38ebec3799b_HUMAN_32-189_3UX9-A:
Traceback (most recent call last):
File "/usr/src/app/hommod_rest/services/model.py", line 944, in modelProc
main_target_sequence, main_domain_range)
File "/usr/src/app/hommod_rest/services/model.py", line 628, in _build_for_domain
targetsInterproRanges, picker)
File "/usr/src/app/hommod_rest/services/domainalign.py", line 439, in pickAlignments
if picker.accepts(targetID, alignment):
File "/usr/src/app/hommod_rest/services/interaction.py", line 80, in accepts
sstart, send = get_target_covered_range(subjectChainAlignment, sseq)
File "/usr/src/app/hommod_rest/services/modelutils.py", line 812, in get_target_covered_range
% (alignment['template'], template_seq))
Exception: mismatch between alignment and template sequence:
IVLTQPPSVSGAPGQRVTISCSGSSSNIGSNYVSWYQQLPGTAPKLLIYDNNQRPSGVPDRFSGSKSGTSASLAITGLQSEDEADYYCQVRDNNENEWVFGGGTKLTVLEVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKGLEWVSAISGSGGSTYYADSVKGRFTISRDNSKNTLYLQMNSLRAEDTAVYYCARYIDFGDHMDFWGQGTLVTVSSL
IVLTQPPSVSGAPGQRVTISCSGSSSNIGSNYVSWYQQLPGTAPKLLIYDNNQRPSGVPDRFSGSKSGTSASLAITGLQSEDEADYYCQVRDNNENEWVFGGGTKLTVLEVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKGLEWVSAISGSGGSTYYADSVKGRFTISRDNSKNTLYLQMNSLRAEDTAVYYCARYIDFGDHMDFWGQGTLVTVSSLE
@cbaakman, can you explain what the cause is before fixing it?
Right now when hommod runs, it repeatedly prints the following message:
celery_1 | [2017-03-03 15:58:58,391: ERROR/Beat] beat: Connection error: [Errno -2] Name or service not known. Trying again in 20.0 seconds...
celery_1 | [2017-03-03 15:59:18,407: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@hommodrest_rabbitmq_1:5672//: [Errno -2] Name or service not known.
celery_1 | Trying again in 22.00 seconds...
celery_1 |
Also hommod returns 500 errors to all users
If the user submits the same data, a duplicate job will be started. How can we prevent this?
hope-flask has an implementation for this. Maybe it can be used here?
HOPE gets the following error from hommod:
BLAST Database error: Could not find volume or alias file (/data/blast/uniprot_trembl) referenced in alias file (/data/blast/uniprot)
/srv/hommod
on the host (chelonium in production) is mapped to /data
in the docker containers; however, /srv/hommod
doesn't contain the file shown in the error:
➜ ~ ls /srv/hommod
blacklisted_templates blast fasta interpro models tmp
➜ ~ ls /srv/hommod/blast
templates.phr uniprot_trembl.06.psq uniprot_trembl.15.psq
templates.pin uniprot_trembl.07.phr uniprot_trembl.16.phr
templates.psq uniprot_trembl.07.pin uniprot_trembl.16.pin
uniprot.pal uniprot_trembl.07.psq uniprot_trembl.16.psq
uniprot_sprot.phr uniprot_trembl.08.phr uniprot_trembl.17.phr
uniprot_sprot.pin uniprot_trembl.08.pin uniprot_trembl.17.pin
uniprot_sprot.psq uniprot_trembl.08.psq uniprot_trembl.17.psq
uniprot_trembl.00.phr uniprot_trembl.09.phr uniprot_trembl.18.phr
uniprot_trembl.00.pin uniprot_trembl.09.pin uniprot_trembl.18.pin
uniprot_trembl.00.psq uniprot_trembl.09.psq uniprot_trembl.18.psq
uniprot_trembl.01.phr uniprot_trembl.10.phr uniprot_trembl.19.phr
uniprot_trembl.01.pin uniprot_trembl.10.pin uniprot_trembl.19.pin
uniprot_trembl.01.psq uniprot_trembl.10.psq uniprot_trembl.19.psq
uniprot_trembl.02.phr uniprot_trembl.11.phr uniprot_trembl.20.phr
uniprot_trembl.02.pin uniprot_trembl.11.pin uniprot_trembl.20.pin
uniprot_trembl.02.psq uniprot_trembl.11.psq uniprot_trembl.20.psq
uniprot_trembl.03.phr uniprot_trembl.12.phr uniprot_trembl.21.phr
uniprot_trembl.03.pin uniprot_trembl.12.pin uniprot_trembl.21.pin
uniprot_trembl.03.psq uniprot_trembl.12.psq uniprot_trembl.21.psq
uniprot_trembl.04.phr uniprot_trembl.13.phr uniprot_trembl.22.phr
uniprot_trembl.04.pin uniprot_trembl.13.pin uniprot_trembl.22.pin
uniprot_trembl.04.psq uniprot_trembl.13.psq uniprot_trembl.22.psq
uniprot_trembl.05.phr uniprot_trembl.14.phr uniprot_trembl.23.phr
uniprot_trembl.05.pin uniprot_trembl.14.pin uniprot_trembl.23.pin
uniprot_trembl.05.psq uniprot_trembl.14.psq uniprot_trembl.23.psq
uniprot_trembl.06.phr uniprot_trembl.15.phr uniprot_trembl.pal
uniprot_trembl.06.pin uniprot_trembl.15.pin
Why isn't the file there?
The unit and integration tests are failing. Fix them.
I get this error in HOPE. What does it mean? Can anything be done about it?
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 60, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 1014, in modelProc
raise Exception("the following models have failed:\n" + s)
Exception: the following models have failed:
147b60cf-e161-6093-2b7f-390a5b952991_HUMAN_34-732_3LY6-A:
Traceback (most recent call last):
File "/usr/src/app/hommod_rest/services/model.py", line 944, in modelProc
main_target_sequence, main_domain_range)
File "/usr/src/app/hommod_rest/services/model.py", line 432, in _build_for_domain
len(main_target_sequence)))
Exception: main domain range: 33 - 732 exceeds sequence length(721)
The following job in HOPE fails because of the error below in hommod:
The sequence:
MERAESSSTEPAKAIKPIDRKSVHQICSGQVVLSLSTAVKELVENSLDAGATNIDLKLKDYGVDLIEVSDNGCGVEEENFEGLTLKHHTSKIQEFADLTQVETFGFRGEALSSLCALSDVTISTCHASAKVGTRLMFDHNGKIIQKTPYPRPRGTTVSVQQLFSTLPVRHKEFQRNIKKEYAKMVQVLHAYCIISAGIRVSCTNQLGQGKRQPVVCTGGSPSIKENIGSVFGQKQLQSLIPFVQLPPSDSVCEEYGLSCSDALHNLFYISGFISQCTHGVGRSSTDRQFFFINRRPCDPAKVCRLVNEVYHMYNRHQYPFVVLNISVDSECVDINVTPDKRQILLQEEKLLLAVLKTSLIGMFDSDVNKLNVSQQPLLDVEGNLIKMHAADLEKPMVEKQDQSPSLRTGEEKKDVSISRLREAFSLRHTTENKPHSPKTPEPRRSPLGQKRGMLSSSTSGAISDKGVLRPQKEAVSSSHGPSDPTDRAEVEKDSGHGSTSVDSEGFSIPDTGSHCSSEYAASSPGDRGSQEHVDSQEKAPETDDSFSDVDCHSNQEDTGCKFRVLPQPTNLATPNTKRFKKEEILSSSDICQKLVNTQDMSASQVDVAVKINKKVVPLDFSMSSLAKRIKQLHHEAQQSEGEQNYRKFRAKICPGENQAAEDELRKEISKTMFAEMEIIGQFNLGFIITKLNEDIFIVDQHATDEKYNFEMLQQHTVLQGQRLIAPQTLNLTAVNEAVLIENLEIFRKNGFDFVIDENAPVTERAKLISLPTSKNWTFGPQDVDELIFMLSDSPGVMCRPSRVKQMFASRACRKSVMIGTALNTSEMKKLITHMGEMDHPWNCPHGRPTMRHIANLGVISQN
Mutation: P470S
The template:
{'file_path': None, 'template': {'required_identity': 23.28984896457817, 'ac_code': '1R6F', 'chain': 'A', 'identity': 44}}
The exception is:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hope/factory.py", line 171, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hope/tasks.py", line 233, in get_structure_features
model = modeler.from_sequence(sequence, position, template_id)
File "/usr/src/app/hope/domain/modeler.py", line 12, in from_sequence
template_id)
File "/usr/src/app/hope/services/hommod.py", line 68, in run
raise ServiceError(json.loads(r.text)['message'])
hope.services.types.ServiceError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 38, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 849, in modelProc
self._set_template(chosenTemplateID.pdbac)
File "/usr/src/app/hommod_rest/services/model.py", line 412, in _set_template
raise Exception("chain %s in %s occurs more than once after cleaning" % (chain, tempac))
Exception: chain A in 1R6F occurs more than once after cleaning
The purpose of hommod-rest is to have models ready so that services don't need to wait a long time for them to be built. In order to do this, hommod-rest must trigger the building/updating of models by itself.
How can we do this? One idea is to use celery's periodic tasks.
Allow clients to specify which template should be used to create the model, bypassing the BLAST search done at the beginning.
This is required for issue https://github.com/cmbi/hope-flask/issues/303 in HOPE. Coos posted the following comment in that discussion:
Right now hommod's organizes models by naming them after their input sequence, species and covered range. If hommod is to take a template id as input, the model storage system would have to change, because there would be an extra parameter. There can then be two different models for the same piece of protein, differing by template only.
...and...
Hommod blasts for a template by itself, it forces yasara to use the found template. Yes, it's possible to tell yasara which template it should take!
We need to discuss what changes would need to be made to the storage system.
Currently the result is a json formatted response containing:
This response is quite large. Is it better to just return the zipped model result? Is there a another option?
It should be possible to install java 8 in the docker image using apt. Because it's version 14.04 this might be via another package repository (e.g on launchpad) or via backports. This needs looking into.
Another error from hommod when using HOPE:
Exception: the following models have failed:
6dd7080c-8fdb-6b4d-993d-75bfe3060c94_HUMAN_13-1488:
Traceback (most recent call last):
File "/usr/src/app/hommod_rest/services/model.py", line 927, in modelProc
mainTargetSeq, mainDomainRange)
File "/usr/src/app/hommod_rest/services/model.py", line 636, in _build_for_domain
yasaraChains, interactingChainAlignments)
File "/usr/src/app/hommod_rest/services/interaction.py", line 98, in __init__
"interacting chain %s" % chainID)
Exception: alignment not provided for interacting chain B
Docker image generation results in a 48MB log file. Commands like tar can be made less verbose to shrink this substantially.
The following error is returned when called from HOPE during get_structure_features
:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hope/factory.py", line 171, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hope/tasks.py", line 206, in get_structure_features
model = modeler.from_sequence(target_seq, position)
File "/usr/src/app/hope/domain/modeler.py", line 14, in from_sequence
model = modeling_service.run(target_sequence, position, 'HUMAN')
File "/usr/src/app/hope/services/hommod.py", line 66, in run
raise ServiceError(json.loads(r.text)['message'])
hope.services.types.ServiceError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 35, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 778, in modelProc
self._check_init()
File "/usr/src/app/hommod_rest/services/model.py", line 226, in _check_init
self.yasara_dir = flask_app.config ['YASARADIR']
File "/usr/src/app/hommod_rest/services/model.py", line 210, in yasara_dir
raise ValueError("{} not found".format (yasara_dir))
ValueError: /deps/yasara/yasara/ not found
This is possibly related to #11.
I get the following error from hommod in hope. The input to HOPE is:
MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGPDEAPRMPEAAPPVAPAPAAPTPAAPAPAPSWPLSSSVPSQKTYQGSYGFRLGFLHSGTAKSVTCTYSPALNKMFCQLAKTCPVQLWVDSTPPPGTRVRAMAIYKQSQHMTEVVRRCPHHERCSDSDGLAPPQHLIRVEGNLRVEYLDDRNTFRHSVVVPYEPPEVGSDCTTIHYNYMCNSSCMGGMNRRPILTIITLEDSSGNLLGRNSFEVRVCACPGRDRRTEEENLRKKGEPHHELPPGSTKRALPNNTSSSPQPKKKPLDGEYFTLQIRGRERFEMFRELNEALELKDAQAGKEPGGSRAHSSHLKSKKGQSTSRHKKLMFKTEGPDSD
The mutation is E349R
.
ServiceError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 33, in create_model
paths = modeler.modelProc (sequence, species_id, residue_number, False)
File "/usr/src/app/hommod_rest/services/model.py", line 815, in modelProc
domainalign.getAlignments (ranges, mainTargetSeq)
File "/usr/src/app/hommod_rest/services/domainalign.py", line 734, in getAlignments
alignment, pid, pcover)
File "/usr/src/app/hommod_rest/services/domainalign.py", line 560, in _get_range_from
m = getCoveredTargetRange(alignment)
File "/usr/src/app/hommod_rest/services/domainalign.py", line 435, in getCoveredTargetRange
while not alignment['target'][targetStart].isalpha():
TypeError: 'BlastAlignment' object has no attribute '__getitem__'
I got the following error from hommod when using HOPE.
ServiceError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 33, in create_model
paths = modeler.modelProc (sequence, species_id, residue_number, False)
File "/usr/src/app/hommod_rest/services/model.py", line 815, in modelProc
domainalign.getAlignments (ranges, mainTargetSeq)
File "/usr/src/app/hommod_rest/services/domainalign.py", line 701, in getAlignments
pdbid, pdbchain = getTemplatePDBIDandChain(hitID)
File "/usr/src/app/hommod_rest/services/modelutils.py", line 329, in getTemplatePDBIDandChain
raise Exception("No template blast hit syntax: \'%s\'" % blastHitID)
Exception: No template blast hit syntax: 'pdb|4KZZ|g'
When the template id is None, the following error happens:
TypeError: argument of type 'NoneType' is not iterable
Please write some unit tests for the new template parameter.
When proteins from multimer complexes, it's important to preserve the multimer interface when modeling. However, this domain is sometimes missing.
We must make hommod look at alternatives in such a case.
HOPE gets this error from HOMMOD:
hope.services.types.ServiceError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 38, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 849, in modelProc
self._set_template(chosenTemplateID.pdbac)
File "/usr/src/app/hommod_rest/services/model.py", line 416, in _set_template
return [tempobj, nMolsOligomerized / nMolsUnoligomerized]
ZeroDivisionError: integer division or modulo by zero
Create a test for this first. Whenever the denominator is a variable you should always check that it's not zero. I'm not sure if that should be possible but I suspect that it's normal for zero mols to be unoligomerized.
The input sequence is:
MMSFVQKGSWLLLALLHPTIILAQQEAVEGGCSHLGQSYADRDVWKPEPCQICVCDSGSVLCDDIICDDQELDCPNPEIPFGECCAVCPQPPTAPTRPPNGQGPQGPKGDPGPPGIPGRNGDPGIPGQPGSPGSPGPPGICESCPTGPQNYSPQYDSYDVKSGVAVGGLAGYPGPAGPPGPPGPPGTSGHPGSPGSPGYQGPPGEPGQAGPSGPPGPPGAIGPSGPAGKDGESGRPGRPGERGLPGPPGIKGPAGIPGFPGMKGHRGFDGRNGEKGETGAPGLKGENGLPGENGAPGPMGPRGAPGERGRPGLPGAAGARGNDGARGSDGQPGPPGPPGTAGFPGSPGAKGEVGPAGSPGSNGAPGQRGEPGPQGHAGAQGPPGPPGINGSPGGKGEMGPAGIPGAPGLMGARGPPGPAGANGAPGLRGGAGEPGKNGAKGEPGPRGERGEAGIPGVPGAKGEDGKDGSPGEPGANGLPGAAGERGAPGFRGPAGPNGIPGEKGPAGERGAPGPAGPRGAAGEPGRDGVPGGPGMRGMPGSPGGPGSDGKPGPPGSQGESGRPGPPGPSGPRGQPGVMGFPGPKGNDGAPGKNGERGGPGGPGPQGPPGKNGETGPQGPPGPTGPGGDKGDTGPPGPQGLQGLPGTGGPPGENGKPGEPGPKGDAGAPGAPGGKGDAGAPGERGPPGLAGAPGLRGGAGPPGPEGGKGAAGPPGPPGAAGTPGLQGMPGERGGLGSPGPKGDKGEPGGPGADGVPGKDGPRGPTGPIGPPGPAGQPGDKGEGGAPGLPGIAGPRGSPGERGETGPPGPAGFPGAPGQNGEPGGKGERGAPGEKGEGGPPGVAGPPGGSGPAGPPGPQGVKGERGSPGGPGAAGFPGARGLPGPPGSNGNPGPPGPSGSPGKDGPPGPAGNTGAPGSPGVSGPKGDAGQPGEKGSPGAQGPPGAPGPLGIAGITGARGLAGPPGMPGPRGSPGPQGVKGESGKPGANGLSGERGPPGPQGLPGLAGTAGEPGRDGNPGSDGLPGRDGSPGGKGDRGENGSPGAPGAPGHPGPPGPVGPAGKSGDRGESGPAGPAGAPGPAGSRGAPGPQGPRGDKGETGERGAAGIKGHRGFPGNPGAPGSPGPAGQQGAIGSPGPAGPRGPVGPSGPPGKDGTSGHPGPIGPPGPRGNRGERGSEGSPGHPGQPGPPGPPGAPGPCCGGVGAAAIAGIGGEKAGGFAPYYGDEPMDFKINTDEIMTSLKSVNGQIESLISPDGSRKNPARNCRDLKFCHPELKSGEYWVDPNQGCKLDAIKVFCNMETGETCISANPLNVPRKHWWTDSSAEKKHVWFGESMDGGFQFSYGNPELPEDVLDVQLAFLRLLSSRASQNITYHCKNSIAYMDQASGNVKKALKLMGSNEGEFKAEGNSKFTYTVLEDGCTKHTGEWSKTVFEYRTRKAVRLPIVDIAPYDIGGPDQEFGVDVGPVCF
And the template is: {'file_path': None, 'template': {'required_identity': 19.5, 'ac_code': '3HQV', 'chain': 'A', 'identity': 582}}
The functions in this file are very long and hard to test. Refactor.
To provide itself with interpro data, hommod runs interpro jobs during a modelling job. When two jobs need the same interpro file however, only one interpro job should run.
Right now this is solved by using lock files. Might there be a cleaner alternative?
I get the feeling that Hommod is less able to build models for HOPE than the old solution. Many more reports are "empty" because there's no structure. This is a feeling, so we need to first check if this is the case.
Once we answer these questions, we can decide what to do about it, if anything.
@hvenselaar Would you be able to look at point 1?
@cbaakman Would you be able to look at point 2?
Hommod needs to access template files (pdb structures) multiple times each modeling run. This can be with intervals of seconds.
Currently, hommod re-downloads the pdb structure with each new step that uses it. But I think it would be faster to have a pdb service in hommod that handles the request for a template file and downloads it only once per modeling run.
HOPE gets the following error when trying to create a model with hommod:
Exception: the following models have failed:
e8ccf04f-7409-2342-9f5d-9b80a1d99517_HUMAN_71-194_1UB1-A:
Traceback (most recent call last):
File "/usr/src/app/hommod_rest/services/model.py", line 951, in modelProc
mainTargetSeq, mainDomainRange)
File "/usr/src/app/hommod_rest/services/model.py", line 472, in _build_for_domain
mainTemplateID.chain))
AttributeError: 'TemplateID' object has no attribute 'chain'
Position is: 143
Template is: 1UB1_A
Sequence is:
MVAGMLGLREEKSEDQDLQGLKDKPLKFKKVKKDKKEEKEGKHEPVQPSAHHSAEPAEAGKAETSEGSGSAPAVPEASASPKQRRSIIRDRGPMYDDPTLPEGWTRKLKQRKSGRSAGKYDVYLINPQGKAFRSKVELIAYFEKVGDTSLDPNDFDFTVTGRGSPSRREQKPPKKPKSPKAPGTGRGRGRPKGSGTTRPKAATSEGVQVKRVLEKSPGKLLVKMPFQTSPGGKAEGGGATTSTQVMVIKRPGRKRKAEADPQAIPKKRGRKPGSVVAAAAAEAKKKAVKESSIRSVQETVLPIKKRKTRETVSIEVKEVVKPLLVSTLGEKSGKGLKTCKSPGRKSKESSPKGRSSSASSPPKKEHHHHHHHSESPKAPVPLLPPLPPPPPEPESSEDPTSPPEPQDLSSSVCKEEKMPRGGSLESDGCPKEPAKTQPAVATAATAAEKYKHRGEGERKDIVSSSMPRPNREEPVDSRTPVTERVS
HOPE received the following error from HOMMOD:
hope.services.types.ServiceError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 38, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 849, in modelProc
self._set_template(chosenTemplateID.pdbac)
File "/usr/src/app/hommod_rest/services/model.py", line 387, in _set_template
self.yasara.CleanObj(tempobj)
File "/deps/yasara/yasara/pym/yasaramodule.py", line 2338, in CleanObj
run(command[:-1])
File "/deps/yasara/yasara/pym/yasaramodule.py", line 1012, in run
if (command.find("\n")==-1): return(process("EXECUTE",command))
File "/deps/yasara/yasara/pym/yasaramodule.py", line 1006, in process
result=com.receivemessage(com.RESULT)
File "/deps/yasara/yasara/pym/yasaramodule.py", line 739, in receivemessage
(messagetype,datasize)=struct.unpack('ii',self.receive(8))
File "/deps/yasara/yasara/pym/yasaramodule.py", line 722, in receive
if (len(chunk)==0): self.reportbrokenconn()
File "/deps/yasara/yasara/pym/yasaramodule.py", line 692, in reportbrokenconn
raise RuntimeError("Connection to YASARA broken. Either you exited YASARA manually or it encountered a fatal error")
RuntimeError: Connection to YASARA broken. Either you exited YASARA manually or it encountered a fatal error
The following template was used: {'file_path': None, 'template': {'required_identity': 19.5, 'ac_code': '3JBH', 'chain': 'A', 'identity': 216}}
This is likely a bug in yasara but it would be nice to know the cause before we email Elmar.
Currently the databank dockerfile runs via a cron job. It would be better to make this a part of hommod as a periodic celery task and run its worker in a separate docker container. This means porting the scripts to python (most of it is already).
Fix pep8 non-compliance in the code. Use flake8
to identify most of the issues.
When a service requires settings, pass them in the constructor rather than importing the flask app where it's not needed.
SecStrProvider
is an example of a service with this problem. There may be others.
I noticed the following errors in the celery stdout log:
YASARA Version 15.3.8.L detected error 728: The connection to the Python module failed with error code 4.
and
Fatal error while trying to exit. It is possible that the scene could not be saved. Original error 728, subsequent error 39.
New error details: ysu_exit: Failed to get directory
Update the dockerfiles to match hope-flask. This allows a separate production and development environment, and supports running tests in docker.
I get the following error in HOPE from hommod.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/src/app/hommod_rest/factory.py", line 96, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/src/app/hommod_rest/tasks.py", line 60, in create_model
template_id)
File "/usr/src/app/hommod_rest/services/model.py", line 786, in modelProc
ranges = interpro.get_domain_locations(main_target_sequence)
File "/usr/src/app/hommod_rest/services/interpro.py", line 181, in get_domain_locations
filepath = self._create_data_file(sequence)
File "/usr/src/app/hommod_rest/services/interpro.py", line 164, in _create_data_file
raise Exception("inteproscan job timed out")
Exception: inteproscan job timed out
How long is the timeout and can we do anything to prevent this causing issue in HOPE?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.