jbaiter / demetsiiify Goto Github PK

View Code? Open in Web Editor NEW

33.0 6.0 6.0 10.42 MB

Web service for creating and hosting IIIF manifests from METS/MODS documents

Home Page: https://demetsiiify.jbaiter.de

License: GNU Affero General Public License v3.0

Python 69.86% CSS 1.77% JavaScript 18.60% HTML 9.33% Mako 0.36% Dockerfile 0.08%

iiif mets mods iiif-presentation iiif-image iiif-annotations

demetsiiify's People

Contributors

Stargazers

Watchers

Forkers

zuphilip stabiberlin stweil stefan-it computervisionfans tuub

demetsiiify's Issues

Could not import METS

I just tried to create an IIIF manifest by pasting this URL
http://dfg-viewer.de/show/?set%5Bmets%5D=http%3A%2F%2Fdibiki.ub.uni-kiel.de%2Fviewer%2Fmetsresolver%3Fid%3DPPN725358769&set%5Bimage%5D=202
into the input field at https://zvdd-ng.de/

resulting in "Could not import METS", the traceback seems to be empty.

Setup demetsiiify behind Nginx

I setup the app on a server and have no problem to get the content of e.g. the root location with curl. But in the browser I always get a 404 and can't figure out the needed nginx configuration.

My nginx site config are:

server {
  listen [::]:80;
  listen 80;
  server_name foo.bar;

  location /socket.io {
    proxy_pass http://127.0.0.1:5000/socket.io;
    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
  }

  location / {
    proxy_pass http://127.0.0.1:5000;
    proxy_redirect off;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

I set up logging for gunicorn and can see that the calls are passed to the app in the docker container, but all there is something like:

[2017-06-29 15:04:03 +0000] [7] [DEBUG] GET /
[2017-06-29 15:04:03 +0000] [7] [DEBUG] Closing connection. 
[2017-06-29 15:04:10 +0000] [7] [DEBUG] GET /browse
[2017-06-29 15:04:10 +0000] [7] [DEBUG] Closing connection.

As I see that you are using also Nginx for https://zvdd-ng.de/ can you tell me whats going wrong?

Don't create IIIF Image Proxy-Service if there's only one image

Currently the app will always generate a proxy Image API info.json and reference that in the manifest for the page image.
In cases where there is only a single image available for a given page, this is not necessary and we should instead just directly put the image URL into the manifest.
In these cases we could also support non-JPEG formats like PNG, since the referenced image is no longer an IIIF Image (which mandates JPEG-support).

Use a Push API to notify about completion

The user should be notified by an (optional) push notification if the manifest has been generated.

Rosetta API?

The Getty Research Institute uses Ex Libris's Rosetta platform. We built a METS to IIIF conversion:
https://github.com/gri-is/mets2man/blob/master/mets-to-iiif.py

But yours seems much more robust! We could look at merging functionality?
(tag: @joshuago78 @alyxrossetti)

Could not import DFG-Viewer example (caused by image URL with redirect?)

While trying to process the BSB example, the METS import fails:

[...]
  File "/code/demetsiiify/mets.py", line 376, in image_info
    'mimetype': server_mime}) from exc
demetsiiify.mets.MetsImportError: Could not open image from http://daten.digitale-sammlungen.de/~db/0002/bsb00020619/images/150/bsb00020619_00015.jpg, likely the server sent corrupt data.
[...]

The image link works in a web browser and redirects to a different web page.

demetsiiify shows an error message with missing information:

 Unfortunately we were unable to generate a IIIF manifest from the METS located at

                                                           .

The error was logged in our backend and will be examined. If you wish to help with debugging, you can consult the traceback below and open an issue on GitHub.

Service unavailable

https://demetsiiify.jbaiter.de/ is currently not available (redirected to https://zvdd-ng.de/, then status 404 Not Found).

Import of ZEFYS METS fails

Currently, the digitised newspapers in ZEFYS rely on a somewhat sneaky way to publish METS files dynamically from Solr - which nevertheless seems to work fine with the venerable DFG-Viewer as in e.g. http://zefys.staatsbibliothek-berlin.de/dfg-viewer/?no_cache=1&set%5Bmets%5D=http%3A%2F%2Fzefys.staatsbibliothek-berlin.de%2Foai%2F%3Ftx_zefysoai_pi1%255Bidentifier%255D%3D6ab07f67-b3be-4309-9e5a-e8e97781048b

However, importing the above URL into https://zvdd-ng.de/ results in an error with below stracktrace

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/code/demetsiiify/blueprints/api.py", line 127, in api_import
    job_meta = mets.get_basic_info(mets_url)
  File "/code/demetsiiify/mets.py", line 380, in get_basic_info
    tree = etree.parse(mets_url)
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
  File "src/lxml/parser.pxi", line 1811, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:117848)
  File "src/lxml/parser.pxi", line 1837, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:118195)
  File "src/lxml/parser.pxi", line 1741, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:117107)
  File "src/lxml/parser.pxi", line 1138, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:111653)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105671)
  File "http://zefys.staatsbibliothek-berlin.de/dfg-viewer/?no_cache=1&set%5Bmets%5D=http%3A%2F%2Fzefys.staatsbibliothek-berlin.de%2Foai%2F%3Ftx_zefysoai_pi1%255Bidentifier%255D%3D4b94aaaa-9d1b-4afa-8211-b52720ffc22e", line 184
lxml.etree.XMLSyntaxError: Entity 'nbsp' not defined, line 184, column 27

When the actual METS URL http://zefys.staatsbibliothek-berlin.de/oai/?tx_zefysoai_pi1%5Bidentifier%5D=6ab07f67-b3be-4309-9e5a-e8e97781048b is extracted from the DFG-Viewer URL above and fed directly to https://zvdd-ng.de/ instead, a different error ensues

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/code/demetsiiify/blueprints/api.py", line 127, in api_import
    job_meta = mets.get_basic_info(mets_url)
  File "/code/demetsiiify/mets.py", line 380, in get_basic_info
    tree = etree.parse(mets_url)
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
  File "src/lxml/parser.pxi", line 1811, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:117848)
  File "src/lxml/parser.pxi", line 1837, in lxml.etree._parseDocumentFromURL (src/lxml/lxml.etree.c:118195)
  File "src/lxml/parser.pxi", line 1741, in lxml.etree._parseDocFromFile (src/lxml/lxml.etree.c:117107)
  File "src/lxml/parser.pxi", line 1138, in lxml.etree._BaseParser._parseDocFromFile (src/lxml/lxml.etree.c:111653)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105671)
  File "<string>", line 0
lxml.etree.XMLSyntaxError: Unknown encoding ""

Latest code not working (undefined variables, wrong function call)

The code uses undefined variables mets_doc and manifest_ident and calls _fill_manifest_metadata with only one instead of two arguments.

I had to fix those lines to get something which works partially:

diff --git a/demetsiiify/iiif.py b/demetsiiify/iiif.py
index 7b861db..1aeafc5 100644
--- a/demetsiiify/iiif.py
+++ b/demetsiiify/iiif.py
@@ -175,7 +175,7 @@ def _make_empty_manifest(ident, label):
         current_app.config['SERVER_NAME']))
     manifest_factory.set_iiif_image_info('2.0', 0)
     manifest = manifest_factory.manifest(ident=manifest_ident,
-                                         label=make_label(mets_doc.metadata))
+                                         label=label)
     return manifest
 
 
@@ -212,9 +212,12 @@ def make_manifest(ident, mets_doc, physical_map, thumbs_map):
     :returns:               Generated IIIF manifest
     :rtype:                 dict
     """
+    manifest_ident = '{}://{}/iiif/{}/manifest'.format(
+        current_app.config['PREFERRED_URL_SCHEME'],
+        current_app.config['SERVER_NAME'], ident)
     manifest = _make_empty_manifest(ident=manifest_ident,
                                     label=make_label(mets_doc.metadata))
-    _fill_manifest_metadata(manifest)
+    _fill_manifest_metadata(manifest, mets_doc.metadata)
 
     phys_to_canvas = {}
     seq = manifest.sequence(ident='default')

List index out of range.

I'm getting this error when I input mets into a locally hosted demetsiiify:

  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/code/demetsiiify/blueprints/api.py", line 127, in api_import
    job_meta = mets.get_basic_info(mets_url)
  File "/code/demetsiiify/mets.py", line 382, in get_basic_info
    doc.read_metadata()
  File "/code/demetsiiify/mets.py", line 162, in read_metadata
    self.metadata['title'] = self._read_titles()
  File "/code/demetsiiify/mets.py", line 148, in _read_titles
    ".//mods:relatedItem[@type='host']/mods:titleInfo")[0]]
IndexError: list index out of range

Wrong URL for generated manifest

With the patches from issue #7 applied, it was possible to create manifests for some of the DFG viewer examples, but demetsiiify shows wrong URLs:

http://localhost:5000/iiif/http://localhost:5000/iiif/urn:nbn:de:gbv:23-drucke/li-1876-12/manifest/manifest (should be http://localhost:5000/iiif/urn:nbn:de:gbv:23-drucke/li-1876-12/manifest), http://localhost:5000/iiif/http://localhost:5000/iiif/urn:nbn:de:bsz:14-db-id2635668113/manifest/manifest, http://localhost:5000/iiif/http://localhost:5000/iiif/urn:nbn:de:gbv:23-drucke/li-1876-12/manifest/manifest, ...

jbaiter / demetsiiify Goto Github PK

demetsiiify's People

Contributors

Stargazers

Watchers

Forkers

demetsiiify's Issues

Could not import METS

Setup demetsiiify behind Nginx

Don't create IIIF Image Proxy-Service if there's only one image

Use a Push API to notify about completion

Rosetta API?

Could not import DFG-Viewer example (caused by image URL with redirect?)

Service unavailable

Import of ZEFYS METS fails

Latest code not working (undefined variables, wrong function call)

List index out of range.

Wrong URL for generated manifest

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent