googlegenomics / variant-annotation Goto Github PK

Currently, we only parallelize VEP as far as bringing up a VM per VCF file, running VEP on each file, and storing newly annotated files back on disk. An attractive option for parallelizing VEP is to host instances of a server listening for annotation requests. Several of these components are flexible, but the stack might look something like:

App Engine → Docker → Flask server → VEP

There are several attractive aspects of this parallelization option:

1. It's relatively straightforward; not much code needs to be written.
2. Since it processes each variant individually, it can be integrated into the Dataflow pipeline of VT.
3. In the long term, the docker container could contain multiple annotation programs, and users could dictate which of them to select as part of their request to the flask server.

We currently download the entire VEP cache of interest into every VM/instance, which takes a significant amount of time. When running smaller test sets, the downloading and unzipping likely accounts for most of the "annotation" time. gcsfuse might be able to alieviate this problem.

On the other hand, there may be some relative latency per variant, which could accumulate significantly on larger data sets. Performance testing would have to be done.