hepdata / hepdata-converter-ws Goto Github PK

There's currently a circular dependency between hepdata-converter-ws-docker and hepdata-converter-ws: the docker image includes hepdata-converter-ws in its requirements, but the hepdata-converter-ws package runs its tests on the hepdata-converter-ws docker image.

We should instead run the hepdata-converter-ws tests on the hepdata-converter docker image. (It will need to install its requirements.) Then the hepdata-converter-ws docker image can install hepdata-converter-ws from pypi so it can run the service.

We'll need to move the requirements.txt from the docker repo. We should also see if we can install them for travis using the setup.py rather than a separate requirements.txt.

See https://docs.sentry.io/platforms/python/flask/ . The dsn variable can be omitted from the configuration, since the SENTRY_DSN environment variable will be passed at runtime in deployment on the Kubernetes cluster at CERN.

For testing, it would be good to add the /debug-sentry endpoint as described in the Sentry docs.

As a relatively simple package I would suggest creating pip package for installation, instead of documentation describing how to do it "by hand". Of course possible connection of this service to any "real" web server (ngnix / apache / etc) would require additional manual work, but it would still be much better then doing it "invenio style"

If we're doing this style of installation then where should configuration be placed? /etc/hepdata-converter-ws/ sounds like a resonable idea.

Any comments @eamonnmag @jstypka ?

Add unit tests to check.

Also update the docker version and produce a new pypi package.

Every time that script hepdata-converter-ws is run the following stack trace is produced:

Traceback (most recent call last):
  File "/usr/local/bin/hepdata-converter-ws", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/hepdata_converter_ws/__init__.py", line 33, in main
    app.run()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 772, in run
    run_simple(host, port, self, **options)
  File "/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 623, in run_simple
    reloader_type)
  File "/usr/local/lib/python2.7/site-packages/werkzeug/_reloader.py", line 229, in run_with_reloader
    reloader.run()
  File "/usr/local/lib/python2.7/site-packages/werkzeug/_reloader.py", line 120, in run
    for filename in chain(_iter_module_files(), self.extra_files):
  File "/usr/local/lib/python2.7/site-packages/werkzeug/_reloader.py", line 22, in _iter_module_files
    filename = getattr(module, '__file__', None)
  File "/usr/local/Cellar/root6/6.04.00/lib/root/cppyy.py", line 131, in __getattr__
    klass = _backend.CreateScopeProxy( name, cls )
TypeError: requested class '__file__' does not exist

This error happens on MAC OS X, investigation must be made to check if it also happens on other systems (CentOS, Ubuntu)

@eamonnmag - that may be the reason why it didn't work with docker... but it did not produce any stack trace, so those two may be unrelated

If the hepdata_converter.convert function fails, for example if the YAML does not validate, currently there is no error handling by the web service, so no useful info is passed back to the client.

We could modify the API to send back a 400 error response with an appropriate error message in the case of failure.

I originally was thinking that returned value in response to request should simply be a file, but how should multiple file formats be resolved then (mainly YAML)? Maybe a proper JSON response is more in order (with Base64 encoded binary?) though it would put additional strain on the service receiving data.

Another option may be to send tar.gz (or other compressed file format) with all files packed into it.

What do you think? @eamonnmag @jstypka

http://stackoverflow.com/questions/18280612/ioerror-errno-24-too-many-open-files

Happens for http://hepdata.net/record/60092 when downloading ROOT files.
See HTML output here (had to attach as txt file).
I restarted the docker container and now things work again (well the error is about the bin ordering for this record). But there are places in the code where the files are not being closed, this is most probably in the actual converter code though.

60092-1-root.txt