xlcnd / isbnlib Goto Github PK

The xID service used by isbnlib for WorldCat metadata lookup is being decommissioned.

One possible workaround for this is to use the Classify - ClassificationResource Read to convert an ISBN to an OCLC identifier, then use the WorldCat Metadata API - bib Read (which uses an OCLC number) to retrieve the metadata. It looks like this might require WorldCat credentials, though?

Hi,

there are quite a lot of services out there using a lower-case x instead of an upper-case one in ISBN10. Unfortunately, canonical(isbnlike) just removes the x, thus rendering the ISBN invalid. I'm not really good at programming, so I solved the problem by modifying the function as follows:

def canonical(isbnlike):
    """Keep only numbers and X."""
    numb = [c for c in isbnlike if c in '0123456789Xx']
    if numb[-1] == 'x':
        numb[-1] = 'X'
    return ''.join(numb)

...although there are probably more elegant or efficient ways of doing that.

The Bug:
When I ran a program I made to that finds the details of the ISBN, a description, and a cover of it. The cover and description functions didn't work and I got this big long error message which I screenshotted below.

My code was:

from isbnlib import meta, desc, cover
Service = 'openl'
ISBN = input("What is your ISBN number (no spaces) ")
book = meta(ISBN, Service)
print(book)
print(desc(ISBN))
bookCover = cover(ISBN)

Screenshot of Error:

Computer info:

• OS: [Mac]
• Python version [3.6]
• isbnlib version [3.9.6]

If this isn't a bug and an error on my part I would love to know what I did wrong.

I tried converting some ISBNs like this:

>>> import isbnlib
>>> isbnlib.to_isbn13("91-43-01019-9")
'91-43-01019-9'

Huh? That doesn't look like an ISBN13? After some thinking it seems I should have stripped the dashes before trying to convert:

>>> isbnlib.to_isbn13("9143010199")
'9789143010190'

But then again I lose the dashes, that makes ISBNs more readable.

I would have expected the first version to work, and give me a formatted ISBN13 back (like on ISBN.org converter).

Or am I simply doing it wrong?

Thanks a lot for this python package. I'm using it for a small project at the moment.

But I'm struggling with changing the metadata output.

meta(isbn, service='default', cache='default')

With isbnlib.registry you can change the metadata service to be used by default (setdefaultservice), 
add a new service (add_service), access bibliographic formatters for metadata (bibformatters), set 
the default formatter (setdefaultbibformatter), add new formatters (add_bibformatter) and set a new 
cache (set_cache) (e.g. to switch off the chache set_cache(None)). The cache only works for calls 
through isbnlib.meta. These changes only work for the ‘current session’, so should be done always 
before calling other methods.

I want to change the output to bibtex. But I'm not sure, what I have to do.

Do you can point me in the right direction, please?

Thanks in advance!

I'm experimenting with this library to clean up a large-ish (~100k) list of ISBNs that I have. I would like to find the EAN for each, extract a list of previous editions, the meta data, and a description. Doing the naive thing results in google books giving a 403 after ~10 ISBNs.

The docs mention it is possible to do things in a more batched way - is there an example of this? Also, I believe I have set my API key for google books, but am not sure. Is there a way to check I have done so correctly (I used isbnlib.config.set_apikey('goob', <MYKEY>))?

ISBNdb has no free tier anymore and version 2 of the API stopped to work (they have a new API, strangely called version 1!). So I will stop his support, starting with version isbnlib-3.8.1.

In [11]: isbnlib.meta(u"2266202022", service="merge")
In [11]: isbnlib.meta(u"2266202022", service="wcat")
#....
/home/jojo/.envs/spreads/local/lib/python2.7/site-packages/isbnlib/_wcat.pyc in _mapper(isbn, records)
     30     except:  # pragma: no cover
     31         LOGGER.debug("RecordMappingError for %s with data %s", isbn, records)
---> 32         raise RecordMappingError(isbn)
     33     # call stdmeta for extra cleanning and validation
     34     return stdmeta(canonical)

RecordMappingError: the mapping `canonical <- records` doesn't work (9782266202022)

When digging around in the stack trace a bit, it seems that the services actually did return the correct metadata, isbnlib just can't produce a proper mapping for it.
I am using the latest version from PyPi (3.5.3).

Update:
I did some more digging, and the problem only seems to occur when I pass the ISBN as an unicode instance. The underlying exception is this:

isbnlib._wcat: decoding Unicode is not supported
Traceback (most recent call last):
  File "/home/jojo/.envs/spreads/local/lib/python2.7/site-packages/isbnlib/_wcat.py", line 23, in _mapper
    canonical['ISBN-13'] = u(isbn)
  File "/home/jojo/.envs/spreads/local/lib/python2.7/site-packages/isbnlib/dev/bouth23.py", line 19, in u
    return unicode(x, "utf-8")
TypeError: decoding Unicode is not supported

Heads up, OCLC announced that the xID services (including xISBN) is going to be retired on 3/15/16. After that the WorldCat integration in isbnlib will be broken.

http://www.oclc.org/developer/news/2015/change-to-xid-services.en.html

As a consequence of #28 the field Year is in many cases null or wrong... a fix should be found for the next release.

Hi,

According to the doc, I can think of two ways to get a BibTeX metadata for a given ISBN:

• Either meta(isbn) and formatting it.
• Either doi2tex(doi(isbn)).

The second method seems to be more reliable, in particular as it relies on the DOI system, which is using a unique identifier and gives better results than meta in my experience.

However, taken from [https://www.doi.org/factsheets/ISBN-A.html](the ISBN-A page on doi.org), they say:

ISBN-As do not automatically exist for every ISBN; they exist only once the agency has registered them in the DOI System.

Whereas when looking at your code you always derive an ISBN-A from a given ISBN13.

For my use case, I would need a way to check whether the generated ISBN-A is indeed a valid ISBN-A linked to any DOI or not. And reading the doc and looking a bit at the code, I was not sure what is the expected behavior of doi2tex(doi(isbn)) when the ISBN-A does not exist.

Thanks a lot!

P.S.: As I do not have examples of ISBNs without associated ISBN-A, I cannot check this easily =( Sorry if this is already documented somewhere else.

Make an alias to _goom.query

See xlcnd/isbntools#103

I'm getting a crash for this specific ISBN number:

>>> isbnlib.mask("9786131796364")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/EmilStenstrom/Envs/boktraven/lib/python2.7/site-packages/isbnlib/_ext.py", line 16, in mask
    return msk(isbn, separator)
  File "/Users/EmilStenstrom/Envs/boktraven/lib/python2.7/site-packages/isbnlib/_msk.py", line 45, in msk
    if l[0] <= int(sevens) <= l[1]:
TypeError: 'int' object has no attribute '__getitem__'

I would expect None as a return value if it's an invalid ISBN.

Hi!

For some ISBNs Google Books returns completely wrong data. One of them is 9780306807435 – The Devil's Music: A History of the Blues.
The problem lies within the SERVICE_URL.
https://github.com/xlcnd/isbnlib/blob/master/isbnlib/_goob.py#L13
SERVICE_URL = ' ... q=isbn+{isbn}'\

According to API documentation (https://developers.google.com/books/docs/v1/using#PerformingSearch), there should be : instead of +.
After these changes returned data is correct; however some of the tests start to fail.

Maybe tests are inaccurate? Could You please take a look?
Thanks!

In some cases the parsing from openl is not well done (making loose records).

It would be nice to have a bibformatter to export ISBN metadata to Citation Styles Language (CSL) JSON. This would help us add support for ISBN citations in the Manubot: see manubot/manubot#14.

I'm envisioning being able to do the following:

import isbnlib
isbn = '9780262035613'
metadata = isbnlib.meta(isbn, cache=None)
csl = isbnlib.registry.bibformatters['csl'](metadata)

csl would presumably a dict or collections.OrderedDict. Alternatively, it could be already dumped as a JSON string (although I think that's less preferable).

CSL JSON is a way of storing bibliographic metadata that is a successor to formats like bibtex. It's used commonly in scholarly publishing. The documentation isn't great, but here's a schema definition. Here's also some written doc.

I'm happy to help as needed. Especially I can help convert the output of isbnlib.meta to CSL JSON. Is there documentation of all the possible keys returned in the output of isbnlib.meta?

Hi there, I have several suggestions/ideas. Since this is all related somehow, I am posting it here as one issue.

I am using isbnlib-dnb and created a PR and an issue for that plugin: https://github.com/arangb/isbnlib-dnb/, both are leading to suggestions/ideas.

The issue is: isbnlib-dnb returns None if no result is found with meta(), isbnlib raises an exception. The docs and the example template do not mention that plugins should raise an exception. So my suggestion would be to give plugin authors more guidelines, which behaviours are expected. This could also be tested by isbnlib.

Same goes for the data that is parsed from various sources, such as dnb: my PR includes a small change to parse the language attribute too, otherwise it is always empty, if i use isbnlib-dnb. My suggestion here would be to make guidelines for the expected values and maybe to test this.

Thirdly, I spent most of the time doing boilerplate stuff. Should I write a small howto what I did, so this might work faster for other devs? I really liked your docs and contribution guidelines and think such a guide would fit in well.

cover returns None instead of (None, None) in some situations.

Hi,

I have some strange behaviour of the "default" option for meta (same behaviour in isbnlib and isbntools), maybe I did not get what it should do correctly.

Here is an example:

>>> isbnlib.meta("0198507194", "wcat")
{'Publisher': u'Clarendon Press', 'Language': u'eng', 'Title': u'Bose-Einstein Condensation', 'Authors': [u'Lev Pitaevskii ; Sandro Stringari'], 'ISBN-13': u'0198507194', 'Year': u'2004'}

>>> isbnlib.meta("0198507194", "goob")
{'Publisher': u'', 'Language': u'en', 'Title': u'Bose-Einstein Condensation', 'Authors': [u''], 'ISBN-13': u'0198507194', 'Year': u''}

>>> isbnlib.meta("0198507194", "default")
{'Publisher': u'Clarendon Press', 'Language': u'eng', 'Title': u'Bose-Einstein Condensation', 'Authors': [u''], 'ISBN-13': u'0198507194', 'Year': u'2004'}

If I understand the doc correctly, default option should be a merge of the two first outputs. That seems correct excepted for the authors field, which should not be overwritten by the `goob value, no ?

Thanks

Here profile of import isbnlib

   1295081 function calls (1285957 primitive calls) in 0.891 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    149/1    0.001    0.000    0.894    0.894 {built-in method builtins.exec}
        1    0.000    0.000    0.894    0.894 devel/profile-import.py:1(<module>)
    166/1    0.001    0.000    0.894    0.894 <frozen importlib._bootstrap>:978(_find_and_load)
    166/1    0.001    0.000    0.894    0.894 <frozen importlib._bootstrap>:948(_find_and_load_unlocked)
    159/1    0.001    0.000    0.894    0.894 <frozen importlib._bootstrap>:663(_load_unlocked)
    126/1    0.000    0.000    0.894    0.894 <frozen importlib._bootstrap_external>:722(exec_module)
    200/1    0.000    0.000    0.894    0.894 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
        1    0.000    0.000    0.894    0.894 .../site-packages/isbnlib/__init__.py:4(<module>)
        1    0.000    0.000    0.884    0.884 .../site-packages/isbnlib/_ext.py:2(<module>)
        1    0.000    0.000    0.837    0.837 .../site-packages/isbnlib/_metadata.py:2(<module>)
        1    0.000    0.000    0.836    0.836 .../site-packages/isbnlib/registry.py:2(<module>)
        1    0.000    0.000    0.797    0.797 .../site-packages/pkg_resources/__init__.py:16(<module>)
        2    0.000    0.000    0.729    0.364 .../site-packages/pkg_resources/__init__.py:3174(_call_aside)
        1    0.000    0.000    0.729    0.729 .../site-packages/pkg_resources/__init__.py:3191(_initialize_master_working_set)
       36    0.003    0.000    0.466    0.013 .../site-packages/pkg_resources/__init__.py:611(add_entry)
     2760    0.006    0.000    0.442    0.000 .../site-packages/pkg_resources/__init__.py:2005(find_on_path)
      454    0.001    0.000    0.262    0.001 .../site-packages/pkg_resources/__init__.py:3218(<genexpr>)
      453    0.001    0.000    0.261    0.001 .../site-packages/pkg_resources/__init__.py:2717(activate)
        1    0.000    0.000    0.238    0.238 .../site-packages/pkg_resources/__init__.py:569(_build_master)
        1    0.000    0.000    0.238    0.238 .../site-packages/pkg_resources/__init__.py:556(__init__)
     3297    0.010    0.000    0.238    0.000 .../site-packages/pkg_resources/__init__.py:2159(_handle_ns)
      453    0.002    0.000    0.233    0.001 .../site-packages/pkg_resources/__init__.py:2258(fixup_namespace_packages)
     5420    0.014    0.000    0.212    0.000 .../site-packages/pkg_resources/__init__.py:2092(distributions_from_metadata)
     3287    0.006    0.000    0.204    0.000 <frozen importlib._bootstrap_external>:421(_find_module_shim)
       18    0.000    0.000    0.203    0.011 .../site-packages/pkg_resources/__init__.py:1979(_by_version_descending)
       20    0.011    0.001    0.203    0.010 {built-in method builtins.sorted}
     3803    0.043    0.000    0.199    0.000 <frozen importlib._bootstrap_external>:1356(find_spec)
     3287    0.003    0.000    0.189    0.000 <frozen importlib._bootstrap_external>:1339(find_loader)
     2710    0.005    0.000    0.149    0.000 .../site-packages/pkg_resources/__init__.py:1994(_by_version)
     2710    0.005    0.000    0.133    0.000 .../site-packages/pkg_resources/__init__.py:2000(<listcomp>)
     2726    0.014    0.000    0.132    0.000 .../site-packages/pkg_resources/__init__.py:2546(from_location)
     8240    0.010    0.000    0.128    0.000 .../site-packages/pkg_resources/_vendor/packaging/version.py:24(parse)
    11080    0.047    0.000    0.116    0.000 .../site-packages/pkg_resources/_vendor/packaging/version.py:198(__init__)

v3.4.8 uses in-memory caching for 'desc' and 'gid'.

Allow the use of user defined caches for v3.4.9.

Hi,

It would be really nice to expose a main exception in dev, so that a dev could catch any isbnlib related exception easily.

What do you think about it ?

Google Books sometimes returns info about other book than the requested one! After all, it is a sales service not a bibliographic info provider... 😄

In order to keep high standards of data quality, isbnlib checks if the identifier of the returned info coincides with the requested ISBN. However, Google doesn't return the ISBN in many valid cases, causing isbnlib to drop some precious info!

FIX: Accept the data as valid if the ISBN is not returned.

Amongst other data wcat (probably others too but I've only ever used worldcat) returns city of publication, this should be passed back to the caller too.

The name of the index is changed by PY3 on Windows.

Please, leave your comments on the code.

Give examples and propose alternatives :)

Some services, like 'goob', allows you to retrieve the summary or description of a book. It would be great to return them when available.

Same happens with the number of pages.

Returning a tumbnail too would be great.

I can do all of them if you agree.

>>> import isbnlib
>>> isbn = "9786610326266"
>>> isbnlib.is_isbn13(isbn)
True
>>> isbnlib.mask(isbn)
None

This is an invalid ISBN according to http://isbn.org/ISBN_converter

Got the following warning on our CI build:

  /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/isbnlib/_imcache.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping

The issue is the following line:

Haven't checked whether any other imports in the package need to be moved to collections.abc, but with Python 3.8 just released, these imports will start failing.

Check if the isbn from return data is the same of the request.

Hi,

In this lib, there is a doi(isbn) function. It would be useful to have the reverse function, that is isbn(doi).

Not sure if it can be done reliably and easily?

Thanks

isbntools https://github.com/xlcnd/isbntools

Spreads https://github.com/DIYBookScanner/spreads

BiblioManager https://github.com/Phyks/BMC/

libBMC https://github.com/Phyks/libbmc/

Alessandria https://gitlab.com/openlabmatera/alessandria

Comic Collector https://github.com/wengole/comiccollector

Abelujo http://www.abelujo.cc/

BibLib https://pypi.python.org/pypi/biblib

One bug found but caching 'editions' is causing problems with some users... I hope to have a fix soon.

I noticed that the mask function returns None if for some reason (e.g. ISBN is not valid in the first place) the grouping is unsuccessful.
I think it's better to raise an exception (so the programmer explicitly handles it) or at least return the original string back (rather than omitting the data).
In any case, the behavior needs to be documented.
Thanks for the nice library.

I'm trying to use isbnlib.meta() and it raises an ISBNLibURLError exception, caused by

"isbnlib/_imcache.py", line 26, in __getitem__
    return self.d[k]
KeyError: '9781585109043default' 

Should be simple to reproduce:

import isbnlib

isbn = '9781491946008'
isbnlib.meta(isbn) 

The same error happens if I use a different service (e.g: 'worldcat'), except for 'loc'. The last just returns an empty dict.

I'm running OSX 10.12, python 3.7.3 and installed isbnlib inside a venv enviroment using pip install isbnlib.

Thank you.

Describe the bug
When attempting to use the ISBN_META tool it errors out and will not return any data

To Reproduce
Steps to reproduce the behavior:

1. Open a CLI and run python.
2. import isbnlib
3. isbn = 'ANY ISBN NUMBER'
4. book = isbnlib.meta(isbn)
5. Traceback (most recent call last):
   File "C:\ProgramData\Anaconda3\lib\site-packages\isbnlib_metadata.py", line 26, in query
   if cache[key]:
   File "C:\ProgramData\Anaconda3\lib\site-packages\isbnlib_imcache.py", line 26, in getitem
   return self.d[k]
   KeyError: '9784286088174default'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\lib\site-packages\isbnlib_goob.py", line 45, in _records
recs = data['items'][0]['volumeInfo']
KeyError: 'items'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "C:\ProgramData\Anaconda3\lib\site-packages\isbnlib_ext.py", line 26, in meta
return query(isbn, service, cache) if isbn else None
File "C:\ProgramData\Anaconda3\lib\site-packages\isbnlib_metadata.py", line 31, in query
meta = servicesservice
File "C:\ProgramData\Anaconda3\lib\site-packages\isbnlib_goob.py", line 64, in query
return _records(isbn, data)
File "C:\ProgramData\Anaconda3\lib\site-packages\isbnlib_goob.py", line 48, in _records
raise NoDataForSelectorError(isbn)
isbnlib.dev._exceptions.NoDataForSelectorError: no data for this selector (9784286088174)

The same issue occurs when running the tool isbn_meta from a command line without using python******

Expected behavior
I expect to receive meta data from google books

Screenshots
If applicable, add screenshots to help explain your problem.

Your computer (please complete the following information):

• OS: Windows 10
• Python version 3.7.1
• isbnlib version 4.3.18

Additional context
Add any other context about the problem here.

Hi, this is more a feature request than an issue. It could be useful to be able to get the expected key rather than the True or False that we can access right now through is_isbn10 and 13.

Hi,

>>> isbnlib.notisbn("9783161484100aa")
False

Is it expected? Reading the doc, I would have said that such an ISBN would be rejected.

Thanks

Hi, I have a problem similar to EvansMike.
When submitting a request, I would like to get info on the edition and the format, together with the other metadata.

I was trying to create, as it was suggested, a second provider ('wcat2') and to adapt it to the extra data I'd like to get.

I therefore created wcat2.py addig two lines

canonical['Edition'] = records.get('ed', u(''))
canonical['Format'] = records.get('form',u(''))

Then I ran:

isbnlib.registry.ass_service('wcat2', 'isbnlib_wcat2.py')

The issue I'm having now is to make it work. I also modified the file _data.py to include the two extra fields I need.

Still, it's not working. At the moment I'm getting NotValidMetadataError.

I have two questions:
first, what would be the procedure? Do you have any suggestion on how to proceed?
second: why aren't these fields included? Is it due to problems with quality/availability?

Thank you in advance!

Most of the books, prior to 1990, don't have an ISBN (at least in EU, where country's National Library codes were used!). So it would be useful if we can have an 'ISBN' for these books.

The function produces an 'ISBN' based in a key formed by title, author (optional) and publisher (optional) of the book.

Consider the ISBN string '978-0-07-879984-6'. I know thats technically not a valid isbn, but I have encountered strings like this many times.

If I run get_isbnlike in normal mode:

isbnlib.get_isbnlike('978-0-07-879984-6', 'normal')

I get ['978-0-07-879984-6']. That's good. But when I run in loose mode:

isbnlib.get_isbnlike('978-0-07-879984-6', 'loose')

I get ['978-0-07-879984-']. Notice the missing 6 on the end. You would think that loose mode would catch just as many valid isbn's as normal mode would, or more. But it doesn't.

I'm not quite sure what the best solution is, but the source of the problem is the RE_LOOSE regex. It checks only for numbers and dashes 10-16 characters long. A simple solution would be to just increase that to 18 or something like that.

isbnlib 3.6.1+ supports plugins, making it very easy to add your own metadata provider or bibliographic format.

So I ask for your contribution to add metadata plugins for books in your language.

Hi,

I have the same issue as discussed here with this library. Maybe it is not caching, but that's what it sounds like to me.

Here is a full output of what I'm running, using isbnlib downloaded via pip2:

20:24 phyks@Phyks-laptop ~ % python2
Python 2.7.6 (default, Feb 26 2014, 12:07:17) 
[GCC 4.8.2 20140206 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import isbnlib
>>> isbnlib.meta("foo", "default")
>>> isbnlib.meta("0198507194", "default")
{'Publisher': u'Clarendon Press', 'Language': u'eng', 'Title': u'Bose-Einstein Condensation', 'Authors': [u''], 'ISBN-13': u'0198507194', 'Year': u'2004'}
>>> isbnlib.meta("foo", "default")
{'Publisher': u'Clarendon Press', 'Language': u'eng', 'Title': u'Bose-Einstein Condensation', 'Authors': [u''], 'ISBN-13': u'0198507194', 'Year': u'2004'}

Thanks

EDIT : Same thing happens using the master and the dev branch of this repo.

Hi,

I use isbnlib in my [BMC][https://github.com/Phyks/BMC) script. I recently updated it to be python 2 and python 3 compatible. However, I am experiencing some problems with Python 3.2 and isbnlib (but I don't have any problem in python 3.4).

Here are the relevant parts:

File "/home/travis/virtualenv/python3.2.5/lib/python3.2/site-packages/isbnlib/dev/bouth23.py", line 25
return type(u'')

SyntaxError: invalid syntax

File "/home/travis/virtualenv/python3.2.5/lib/python3.2/site-packages/isbnlib/__init__.py", line 33, in <module>
from . import config # <-- first import

ImportError: cannot import name config

You can see the full output of Travis here : https://travis-ci.org/Phyks/BMC/jobs/31568927

Do you have more infos about these errors ? Thanks !

>>> import isbnlib
>>> isbnlib.__version__
'3.4.7'
>>> isbnlib.meta('9788806219345') # No results
>>> isbnlib.meta('9782754003254') # Ok
{'ISBN-13': '9782754003254', 'Language': 'ita', 'Publisher': 'First Editions', 'Year': '2007', 'Authors': ['Francesca Romana Onofri', 'Karen Antje Möller'], 'Title': "L'italien pour les nuls"}
>>> isbnlib.meta('9788806219345') # Wrong we get the same result as for ISBN 9782754003254
{'ISBN-13': '9782754003254', 'Language': 'ita', 'Publisher': 'First Editions', 'Year': '2007', 'Authors': ['Francesca Romana Onofri', 'Karen Antje Möller'], 'Title': "L'italien pour les nuls"}

Is it a cache problem?

The commit 85b034d raise -> pass in metadata.py create this problem!

This is not a problem for the built-in (in-memory) cache!

It seems that the setdefaulttimeout call in config.py has ugly side effects when used in a Flask application with DEBUG = True.

Is there any reason for using this instead of the urlopen timeout parameter?

This doesn't allow merge to fully replace wcat data with goob data when wcat is not available!

The links to images of book covers are not accessible to direct download anymore (BAD BAD Google!).

Solution

Since a general alternative is not available, cover will provide links to images!