Giter VIP home page Giter VIP logo

firecloud-tools's Introduction

Tools for use with FireCloud

To run a given script using the run script:

  • ./run.sh scripts/directory/script_name.py <arguments>

To run a giving script using Docker:

  • docker run --rm -it -v "$HOME"/.config:/.config broadinstitute/firecloud-tools python /scripts/<script name.py> <arguments>

Prerequisites

When running without the run script or docker, check the packages that are pip installed in either run.sh or the Dockerfile.

firecloud-tools's People

Contributors

abaumann avatar amarotaylor avatar andy7i avatar dvoet avatar gordonwade avatar helgridly avatar jmthibault79 avatar katevoss avatar ruchim avatar wnojopra avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

firecloud-tools's Issues

register_service_account.py doesn't work when using the docker image

register_service_account.py fails with a "CERTIFICATE_VERIFY_FAILED" error when running in the latest docker image (broadinstitute/firecloud-tools:latest):

+ python scripts/register_service_account/register_service_account.py -j /gsa-key/key.json -e [email protected]
/usr/local/lib/python2.7/dist-packages/OpenSSL/crypto.py:14: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in the next release.
  from cryptography import utils, x509
Traceback (most recent call last):
  File "scripts/register_service_account/register_service_account.py", line 47, in <module>
    main()
  File "scripts/register_service_account/register_service_account.py", line 39, in main
    request = requests.post(uri, headers=headers, json=profile_json)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.firecloud.org', port=443): Max retries exceeded with url: /register/profile (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)'),))

To fix it, I replaced
requests.post(uri, headers=headers, json=profile_json)
with
requests.post(uri, headers=headers, json=profile_json, verify=False)
in my code @ https://github.com/bw2/hail-utils/blob/master/register_hail_batch_service_account_as_terra_user.py#L65

Docker error: mount path must be absolute.

docker run --rm -it -v "$HOME"/.config:/.config broadinstitute/firecloud-tools python /scripts/<script name.py>

docker: Error response from daemon: invalid volume specification: '"$HOME"/.config:.config': invalid mount config for type "bind": invalid mount path: '.config' mount path must be absolute.
See 'docker run --help'.

I think the problem may be the attempt to mount the my "$HOME"/.config to the image at .config - docker may now prefer to mount a directory instead of a file?

Keyerror in bigquery_billing_report.py

I talked with @ruchim during the firecloud office hours today about an error I get when running bigquery_billing_report.py.

.--- Submission: 840c08df-0ad6-4a14-8a5d-0aa2f004eceb (submitted by [email protected])
|	.--- Workflow 4 of 24: 7b47577c-7d53-4fd5-b06a-ca23fde0e76f (Succeeded)
Traceback (most recent call last):
  File "/scripts/bigquery_billing_report/bigquery_billing_report.py", line 361, in <module>
    main()
  File "/scripts/bigquery_billing_report/bigquery_billing_report.py", line 357, in main
    args.print_queries)
  File "/scripts/bigquery_billing_report/bigquery_billing_report.py", line 191, in print_submission_pricing
    call_name]
KeyError: u'oncotate-m2'

It seems to encounter this error when the Workflow it's about to check on failed, particularly, it shows

message: Unable to complete JES Api Request
causedBy: 
message: the local copy message must have path set.

When I use the Google Big Query website to check on this workflow, it says the Query returned zero records.

It seems this might be an edge case that needs to be caught.

Error when using using BigQuery billing script

  File "/scripts/bigquery_billing_report/bigquery_billing_report.py", line 358, in <module>
    main()
  File "/scripts/bigquery_billing_report/bigquery_billing_report.py", line 354, in main
    args.print_queries)
  File "/scripts/bigquery_billing_report/bigquery_billing_report.py", line 184, in print_submission_pricing
    elif call["callCaching"]["hit"]:
KeyError: 'hit'```

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.