edsu / pymarc Goto Github PK
View Code? Open in Web Editor NEWprocess MARC records from Python
Home Page: http://python.org/pypi/pymarc
License: Other
process MARC records from Python
Home Page: http://python.org/pypi/pymarc
License: Other
If I have a MARC record where leader[9] == 'a' but that contains ANSEL-encoded data, which is read with a MarcReader where to_unicode=False, then pymarc fails to write the record because the presence of leader[9] forces UTF8 encoding, which blows up for ANSEL strings. See mbklein@ff31286
(Of course, if the record is read with 'to_unicode=True', it'll blow up during reader because the record can't be utf8-decoded.)
It seems to me a better strategy might be to remember in the record whether it should be utf8-encoded on write, rather than relying on the (possibly erronenous) leader[9] field. In short, code like this shouldn't fail even with defective/misencoded records:
for record in marcreader:
if (somecondition):
marcwriter.write(record)
ps:
On a related, but separate note, it seems to me that pymarc would double-encode a record that has leader[9] == 'a' on output unless the record was read with to_unicode=True.
I'm exporting records from our library system and I need to replicate the leader field where it makes sense to do so. Record's API does not allow me to do that.
Can I propose the addition of a leader keyword argument to init of Record? This argument would default to ' ' * LEADER_LEN
.
Leader positions 5 through 8 and 17 through 19 would be taken from the leader argument. The other leader positions would be set or calculated, as is currently done.
Happy to submit a pull request if this proposal is acceptable.
It would be really convenient to be able to mutate the leader somehow.
I find myself writing record.leader = record.leader[0:9] + 'a' + record.leader[10:]
when converting MARC-8 to UTF-8 to work around the leader being an immutable string, which is pretty ugly.
(On the other hand, I'm not all that sure this could be done in a way that's backwards-compatible with things that expect record.leader to be a normal string. record.leader[9] = 'a'
might be unrealistic to support, but perhaps we could add a method that flips a specific part of the leader?)
If you try to load a MARC record in MARCMaker format you get an obscure exception:
Traceback (most recent call last):
File "test.py", line 4, in <module>
for record in reader:
File "/usr/local/lib/python2.7/site-packages/six.py", line 558, in next
return type(self).__next__(self)
File "/Users/ed/Projects/pymarc/pymarc/reader.py", line 90, in __next__
length = int(first5)
ValueError: invalid literal for int() with base 10: '=LDR '
Something a bit more descriptive would be useful! pymarc does not support reading records in this format.
for whatever reason, the following statement fails inside a django 1.4.5 project (python 2.7.5)
from six.moves import zip_longest as izip_longest
(in record.py)
I have a format that needs to be parsed, like the structure test/bad.bat, but I didn't find the demo in your code.
I am trying to use pymarc to read the Harvard Open Metadata MARC files.
Most of the files process ok but some (for example ab.bib.14.20160401.full.mrc) produce errors when processing. The error I am getting is:
Traceback (most recent call last):
File "domark.py", line 21, in <module>
for record in reader:
File "/Library/Python/2.7/site-packages/six.py", line 535, in next
return type(self).__next__(self)
File "/Users/markwatkins/Sites/pharvard/pymarc/reader.py", line 97, in __next__
utf8_handling=self.utf8_handling)
File "/Users/markwatkins/Sites/pharvard/pymarc/record.py", line 74, in __init__
utf8_handling=utf8_handling)
File "/Users/markwatkins/Sites/pharvard/pymarc/record.py", line 307, in decode_marc
code = subfield[0:1].decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128)
The driver code I am using is:
#!/usr/bin/python -tt
# -*- coding: utf-8 -*-
import codecs
import sys
from pymarc import MARCReader
UTF8Writer = codecs.getwriter('utf8')
sys.stdout = UTF8Writer(sys.stdout)
if len(sys.argv) >= 2:
files = [sys.argv[1]]
for file in files:
with open(file, 'rb') as fh:
reader = MARCReader(fh, utf8_handling='ignore')
for record in reader:
# print "%s by %s" % (record.title(), record.author())
print(record.as_json())
Other MARC processing tools (e.g. MarcEdit seem to process the file with no issues so I think the file is legitimate).
Am I doing something wrong? Is there an issue with pymarc, possibly UTF-8 processing related?
pymarc can handles the standard MARC21 in ISO2709(raw MARC).
However I got a strange format of MARC, which is called line-format by the utility yaz-marcdump
.
How could I parse this marc with pymarc??
Following is the marc(with some char in Chinese), or on https://gist.github.com/mail6543210/6095688
File can be seen as UTF-8.
000 cam
010 0 $a978-986-157-529-2$b平裝$dNT$380
091 $acw$bCIP97006132
100 $a20080401d2008 k y0chiy50 e
101 0 $achi
102 $acw
200 1 $a危機OFF$e企業危機管理指南$f羅倫斯.巴頓(Laurence Barton)原著$g許瀞予譯
205 $a初版
210 $a臺北市$c麥格羅希爾$d2008.05
211 $a0805
215 0 $a 面$d 公分
225 1 $a經營管理叢書$vBM173
330 $a本書提及企業如何因應威脅、災難、人為破壞以及醜聞,如何控管各種風險。
454 1 $12001 $aCrisis leadership now$ea real-world guide to preparing for threats, disaster, sabotage, and scandal
606 $2cst$a危機管理
606 $2cst$a企業領導
606 $2lc$aLeadership.
606 $2lc$aCrisis management.
676 $a658.4/092
680 $aHD57.7
681 $a494
Not all, but a good portion of the records in the associated mrc file, when read, produce the warning "couldn't find 0xa0 in g0=66 g1=69". Is this expected?
>>> record.as_marc()
'01182nam0 22003253i 450 001001100000005001700011010001800028100004100046101000800087102000700095181002000102182001100122200008100133205001700214210003400231215001800265225001000283300003100293300004800324410003200372500004800404676004100452700003700493702004000530790004800570801002800618850001900646950017800665977001300843\x1eMIL0864540\x1e20180302002150.0\x1e \x1fa9788804642091\x1e \x1fa20140730d2014 ||||0itac50 ba\x1e| \x1faita\x1e \x1fait\x1e 1\x1f6z01\x1fai \x1fbxxxe \x1e 1\x1f6z01\x1fan\x1e1 \x1faRaccolto di sangue\x1fe[thriller]\x1ffSharon Bolton\x1fgtraduzione di Manuela Faimali\x1e \x1faEd. speciale\x1e \x1faMilano\x1fcOscar Mondadori\x1fd2014\x1e \x1fa453 p.\x1fd20 cm\x1e| \x1faOscar\x1e \x1faIn copertina: Oscar estate\x1e \x1faA pagina IV di copertina: ebook disponibile\x1e 0\x1f1001CFI0000102\x1f12001 \x1faOscar\x1e10\x1faHThe Iblood harvest\x1f3UBO3836087\x1f9RAVV580629\x1e \x1fa823.92\x1f9Narrativa inglese. 2000-\x1fv22\x1e 1\x1faBolton\x1fb, S. J.\x1f3RAVV580629\x1f4070\x1e 1\x1faFaimali\x1fb, Manuela\x1f3LO1V356745\x1f4070\x1e 1\x1faBolton\x1fb, Sharon\x1f3CFIV315469\x1fzBolton, S. J.\x1e 3\x1faIT\x1fbIT-000000\x1fc20140730\x1e \x1faIT-\x1faIT-MI0185\x1e 0\x1faArch. della Produzione Editoriale della Lombardia\x1fc1 v.\x1fd ELAPE-M F18 4698\x1fe ELAPE0001648725 VMN 1 v.\x1ffB \x1fh20141126\x1fi20141126\x1e \x1fa EL\x1fa NB\x1e\x1d'
>>> record = reader.next()
>>> record = reader.next()
>>> record = reader.next()
>>> record = reader.next()
>>> record = reader.next()
couldn't find 0xa0 in g0=66 g1=69
couldn't find 0xa0 in g0=66 g1=69
>>> record.as_marc()
'01030nam0 22003013i 450 001001100000005001700011010001800028010001800046100004100064101001300105102000700118181002000125182001100145200004200156210002500198215001800223225001600241300003300257410003800290500004800328517003100376676004200407700004100449801002800490850001900518950017800537977001300715\x1eMIL0864555\x1e20180302002150.0\x1e \x1fa9788856639339\x1e \x1fa9788856646948\x1e \x1fa20140730d2014 ||||0itac50 ba\x1e| \x1faita\x1fcita\x1e \x1fait\x1e 1\x1f6z01\x1fai \x1fbxxxe \x1e 1\x1f6z01\x1fan\x1e1 \x1faTutta mia la citt\xa9 \x1ffCarlotta Pistone\x1e \x1faMilano\x1fcPiemme\x1fd2014\x1e \x1fa306 p.\x1fd22 cm\x1e| \x1faPiemme voci\x1e \x1faIn copertina: Milano in love\x1e 0\x1f1001CAG1804037\x1f12001 \x1faPiemme voci\x1e10\x1faTutta mia la citt\xa9 \x1f3LO11530364\x1f9RMLV077939\x1e1 \x1faMilano in love\x1f9BVE0684571\x1e \x1fa853.92\x1f9Narrativa italiana. 2000-\x1fv22\x1e 1\x1faPistone\x1fb, Carlotta\x1f3RMLV077939\x1f4070\x1e 3\x1faIT\x1fbIT-000000\x1fc20140730\x1e \x1faIT-\x1faIT-MI0185\x1e 0\x1faArch. della Produzione Editoriale della Lombardia\x1fc1 v.\x1fd ELAPE-M F18 4725\x1fe ELAPE0001649025 VMN 1 v.\x1ffB \x1fh20141126\x1fi20141126\x1e \x1fa EL\x1fa NB\x1e\x1d'
>>> record.as_dict()
{'fields': [{'001': u'MIL0864555'}, {'005': u'20180302002150.0'}, {'010': {'ind1': u' ', 'subfields': [{u'a': u'9788856639339'}], 'ind2': u' '}}, {'010': {'ind1': u' ', 'subfields': [{u'a': u'9788856646948'}], 'ind2': u' '}}, {'100': {'ind1': u' ', 'subfields': [{u'a': u'20140730d2014 ||||0itac50 ba'}], 'ind2': u' '}}, {'101': {'ind1': u'|', 'subfields': [{u'a': u'ita'}, {u'c': u'ita'}], 'ind2': u' '}}, {'102': {'ind1': u' ', 'subfields': [{u'a': u'it'}], 'ind2': u' '}}, {'181': {'ind1': u' ', 'subfields': [{u'6': u'z01'}, {u'a': u'i '}, {u'b': u'xxxe '}], 'ind2': u'1'}}, {'182': {'ind1': u' ', 'subfields': [{u'6': u'z01'}, {u'a': u'n'}], 'ind2': u'1'}}, {'200': {'ind1': u'1', 'subfields': [{u'a': u'Tutta mia la citt\xa9 '}, {u'f': u'Carlotta Pistone'}], 'ind2': u' '}}, {'210': {'ind1': u' ', 'subfields': [{u'a': u'Milano'}, {u'c': u'Piemme'}, {u'd': u'2014'}], 'ind2': u' '}}, {'215': {'ind1': u' ', 'subfields': [{u'a': u'306 p.'}, {u'd': u'22 cm'}], 'ind2': u' '}}, {'225': {'ind1': u'|', 'subfields': [{u'a': u'Piemme voci'}], 'ind2': u' '}}, {'300': {'ind1': u' ', 'subfields': [{u'a': u'In copertina: Milano in love'}], 'ind2': u' '}}, {'410': {'ind1': u' ', 'subfields': [{u'1': u'001CAG1804037'}, {u'1': u'2001 '}, {u'a': u'Piemme voci'}], 'ind2': u'0'}}, {'500': {'ind1': u'1', 'subfields': [{u'a': u'Tutta mia la citt\xa9 '}, {u'3': u'LO11530364'}, {u'9': u'RMLV077939'}], 'ind2': u'0'}}, {'517': {'ind1': u'1', 'subfields': [{u'a': u'Milano in love'}, {u'9': u'BVE0684571'}], 'ind2': u' '}}, {'676': {'ind1': u' ', 'subfields': [{u'a': u'853.92'}, {u'9': u'Narrativa italiana. 2000-'}, {u'v': u'22'}], 'ind2': u' '}}, {'700': {'ind1': u' ', 'subfields': [{u'a': u'Pistone'}, {u'b': u', Carlotta'}, {u'3': u'RMLV077939'}, {u'4': u'070'}], 'ind2': u'1'}}, {'801': {'ind1': u' ', 'subfields': [{u'a': u'IT'}, {u'b': u'IT-000000'}, {u'c': u'20140730'}], 'ind2': u'3'}}, {'850': {'ind1': u' ', 'subfields': [{u'a': u'IT-'}, {u'a': u'IT-MI0185'}], 'ind2': u' '}}, {'950': {'ind1': u' ', 'subfields': [{u'a': u'Arch. della Produzione Editoriale della Lombardia'}, {u'c': u'1 v.'}, {u'd': u' ELAPE-M F18 4725'}, {u'e': u' ELAPE0001649025 VMN 1 v.'}, {u'f': u'B '}, {u'h': u'20141126'}, {u'i': u'20141126'}], 'ind2': u'0'}}, {'977': {'ind1': u' ', 'subfields': [{u'a': u' EL'}, {u'a': u' NB'}], 'ind2': u' '}}], 'leader': u'01030nam0 22003013i 450 '}
I’ve run into a problem that appears to be related to upgrading pymarc from version 2.9.2 to version 3.0.4. I have Python version 2.7.5 and version 2.0.4 of PyZ3950 installed on CentOS Linux 7. The error message is:
Traceback (most recent call last):
File "./recordtest.py", line 15, in
print "marc_record:[%s]" % marc_record
File “/path_to/lib/python/site-packages/pymarc/record.py", line 84, in str
text_list.extend([str(field) for field in self.fields])
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1024: ordinal not in range(128)
Below is a short test script I used to replicate the problem (works with pymarc 2.9.2 but not 3.0.4):
from PyZ3950 import zoom
from pymarc import Record
connection = zoom.Connection ('z3950.loc.gov', 7090)
connection.databaseName = 'VOYAGER'
connection.preferredRecordSyntax = 'USMARC'
query = zoom.Query ('CCL', 'isbn=9780415782654')
results = connection.search (query)
for result in results:
print "result:[%s]" % result
print
marc_record = Record(data=result.data)
print "marc_record:[%s]" % marc_record
connection.close ()
The record I am using to test can be found here:
http://lccn.loc.gov/2011052495
It could very well be that I missed flag or parameter that is required in pymarc 3.#, but I did not see anything in the documentation.
Thoughts or suggestions?
an email from [email protected]
I've just started to work with python/pymarc on a new computer and used pip to install pymarc (using Python 2.7.10), but when I've just tried to run a python script, I get the following error for the "import pymarc" line:
172-16-245-249:bin hf36$ python aco-1-xml2mrc-oclc-nums.py
Traceback (most recent call last):
File "aco-1-xml2mrc-oclc-nums.py", line 9, in <module>
import pymarc
File "/Library/Python/2.7/site-packages/pymarc/__init__.py", line 62, in <module>
from .record import *
File "/Library/Python/2.7/site-packages/pymarc/record.py", line 31, in <module>
@six.python_2_unicode_compatible
AttributeError: 'module' object has no attribute 'python_2_unicode_compatible'
I assume I've something out of date or such, but have no idea how to troubleshoot this error... (I'm not that versed in installation issues)
The title() method, if called on an item where the main title is in 245a and the subtitle in 245b, jams title and subtitle together. They need a colon and space separator.
[2010-10-25 14:17:46,099][test_db5] ERROR:web-services:[24]: 'raw_data': record.as_marc21(),
[2010-10-25 14:17:46,100][test_db5] ERROR:web-services:[25]: File "build/bdist.linux-i686/egg/pymarc/record.py", line 221, in as_marc
[2010-10-25 14:17:46,100][test_db5] ERROR:web-services:[26]: field_data = field_data.encode('utf-8')
I would attach a file with the real MARC packet that triggers the error, but I do not know how to do that.
In basic the problem is that, the subfields are in utf-8 already, still the method "as_marc" from class named "Field" returns strings with utf-8 inside them.
I have discovered that if the field object detect the indicator is a space, it will be converted as '\\'
for indicator in self.indicators:
if indicator in (' ','\\'):
text += '\\'
else:
text += '%s' % indicator
The "\" will be printed in the MARC file, another library such as Perl's MARC::Record will trigger it as an invalid indicator.
I take a file exported from NextGenLib (http://verussolutions.biz/web/content/download) software which contains around 1500 records in MARC21 format. When i try to open it using pymarc 3.0.3 following error gets thrown:
Traceback (most recent call last):
File "convert_marc21_records.py", line 18, in
for record in reader:
File "/Users/alexcorbi/Developing/ODC/venv/lib/python2.7/site-packages/six.py", line 530, in next
return type(self).**next**(self)
File "/Users/alexcorbi/Developing/ODC/venv/lib/python2.7/site-packages/pymarc/reader.py", line 97, in **next**
utf8_handling=self.utf8_handling)
File "/Users/alexcorbi/Developing/ODC/venv/lib/python2.7/site-packages/pymarc/record.py", line 72, in **init**
utf8_handling=utf8_handling)
File "/Users/alexcorbi/Developing/ODC/venv/lib/python2.7/site-packages/pymarc/record.py", line 321, in decode_marc
raise NoFieldsFound
pymarc.exceptions.NoFieldsFound: Unable to locate fields in record data
However if i try the sample files provided with the library, they get read properly. Is there any difference between MARC21 files?
Hello. I have simple data collected from YAZ commands.
yaz-client -m catalogue.dat
I am connecting to library which has MARC21 format and UTF-8 encoding.
I am saving records to catalogue.dat file. It's CZECH library so titles are with special characters for example Ř or Ě etc. when i will run this code:
def get_books(request):
with open('catalogue.dat', 'rb') as fh:
reader = MARCReader(fh)
for record in reader:
print(str(record.title()))
return HttpResponseRedirect('/')
Console will print this:
couldn't find 0xbe in g0=66 g1=69
Zelen©Ł kniha /
couldn't find 0xbe in g0=66 g1=69
Kniha p¿©Łtel /
Kniha ¿©Ưkadel /
Kniha poezie /
Kniha dn©Ư /
Kniha ¿©Ưkadel /
Kniha definic /
Kniha cest /
Kniha Frenesis /
Smoln©Ł kniha /
couldn't find 0xbe in g0=66 g1=69
couldn't find 0xbe in g0=66 g1=69
couldn't find 0xbe in g0=66 g1=69
couldn't find 0xaf in g0=66 g1=69
So basicly there are two issues. First why it prints couldn't find errors and why it prints data without that special characters? Thank you so much.
Having some xml data,
data = open('test.xml', 'rb')
, I expected from the README example to be able to do something like
from pymarc import MARCReader
for record in MARCReader(data):
...
but instead I had to do
import pymarc
for record in pymarc.parse_xml_to_array(data):
...
Determining the file type should be quite easy from reading the first characters of the file stream: xml if "<?xml", json if "{", plain marc otherwise.
Next I wanted to try to serialize a record to XML. The Record
object has methods like as_marc()
, as_marc21()
and hm, even as_json()
, but no as_xml()
! Instead:
pymarc.record_to_xml(record)
I ran into an issue with serializing from marc_xml into marc21 transmission. I get a UnicodeDecode error if any of my data is beyond the default encoding (ASCII) because there is a mix of types. This could also be an issue if someone modifies the leader through record.leader and somehow puts in a unicode object instead of an ordinary string object. It doesn't show up in the other as_marc tests because there isn't any writing out of both unicode data and leaders.
I've got a test at: https://gist.github.com/1322248
def test_writing_unicode_leader(self):
"""Test serializing a record with a unicode leader"""
record = Record()
record.add_field(Field(245, ['1', '0'], ['a', unichr(0x1234)]))
record.leader = u' a ' # unicode leader here.
writer = MARCWriter(open('test/foo', 'w'))
writer.write(record)
writer.close()
reader = MARCReader(open('test/foo'), to_unicode=True)
record = reader.next()
self.assertEqual(record['245']['a'], unichr(0x1234))
os.remove('test/foo')
Here is my traceback:
Traceback (most recent call last):
File "/.../marc8.py", line 69, in test_writing_unicode_leader
writer.write(record)
File "/...", line 40, in write
self.file_handle.write(record.as_marc())
File "/...", line 249, in as_marc
return self.leader + directory + fields
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 4: ordinal not in range(128)
I can submit a patch, but my problem is I am kind of mystified by where I should be dealing with this:
Over in #89 there has been a long discussion about working with invalid MARC data in the wild. I must admit I don't work with MARC much these days, so I had no idea people were running into so many problems processing large batches of MARC records.
Currently when pymarc runs into a structural problem in a MARC record it will throw an exception (RecordLeaderInvalid, BaseAddressNotFound, BaseAddressInvalid, RecordDirectoryInvalid, NoFieldsFound) which will also cause record iteration to stop.
@anarchivist offered up a PermissiveMARCReader which he has used to process large amounts of MARC data. PermissiveMARCReader catches all the exceptions thrown by structural problems with the MARC record and moves on to the next record.
Rather than introducing a new class I suggest that a new parameter named strict be added to the MARC.MARCReader constructor. When set to True it will continue to throw these exceptions. When set to False it will catch the exceptions, log them, and move on to the next record. It may be that some of these exceptions need to be relaxed, and the invalid data interpreted in some way. But let's open new issue tickets for those situations as they come up.
Based on the conversation we've been seeing lately I think the default for strict should be set to False. The MARCReader API will be backwards compatible (code that uses pymarc won't need to change). However this will be a significant change in behavior so I think a new minor version release will be needed, v3.2.
What do folks think?
Then, there will not be a test
folder in the site-packages
, it will appears in site-packages/pymarc/test
What you need to do are:
git mv test pymarc/
sed -i "s/'test\//'pymarc\/test\//g" pymarc/test/*.py
sed -i "s/'test'/'pymarc\.test'/" setup.py
I need to output records encoded in MARC-8. I am willing to code this, but I would like to know how you would advice I should do it ?
For backwards compatibility it would be better to give a default encoding for as_marc in field.py like:
def as_marc(self, encoding='utf8'):
I noticed some tests failed with a very simple test case between PyMARC 2.8.9 and 2.9.0:
with open(sys.argv[1], 'rb') as marc_file:
reader = MARCReader(marc_file)
for record in reader:
print record['752']['a'], record['752']['b'], record['752']['d']
Using this data file: https://github.com/loc-rdc/wdl/blob/master/importers/tests/data/LOT10340.mrc the output looks like this:
2.9+:
None None None
None None None
None None None
None None None
None None None
None None None
None None None
None None None
None None None
None None None
None None None
Russian Federation None None
2.8.9:
Russian Federation Kostroma Oblast Kostroma
Russian Federation Kostroma Oblast Kostroma
Russian Federation Kostroma Oblast Kostroma
Russian Federation None None
Russian Federation Kostroma Oblast Kostroma
Russian Federation Kostroma Oblast Kostroma
Russian Federation Kostroma Oblast Kostroma
Russian Federation Kostroma Oblast Kostroma
Russian Federation Kostroma Oblast Kostroma
Russian Federation Kostroma Oblast Kostroma
Russian Federation Kostroma Oblast Kostroma
Russian Federation None None
With version 2.8.2, the setup.py indicates the version number correctly (2.8.2) but the pymarc/init.py file specifies it as 2.8.1 which is wrong. You've mentioned this in your comment in the setup.py file as well # remember to update pymarc/__init__.py on release!
RDA has brought with it three new fields which are mandatory according to the standard. They are
336 - Content Type
337 - Media Type
338 - Carrier Type
As they will (or should) appear in every RDA record, would helper methods for record.py
for these be of interest to pymarc? Something along the lines of content_type(self)
, etc.
Let me know and I'd be happy to submit a pull.
The JSON serializer [written by me :|] contains a bug, that will throw away multiple subfields and only keep one.
I'll shortly submit a pull request that fixes this issue.
I've been dealing with a problem related to a string vs. byte side-effect in a MARC record leader when using the as_marc() method when processing both new and existing MARC records in Python 3. The bug can be replicated as follows:
>> new_rec = pymarc.Record()
>> type(new_rec.leader)
<class 'str'>
>> new_rec.as_marc()
b'00026 2200025 4500\x1e\x1d'
>> type(new_rec.leader)
<class 'bytes'>
Any subsequent processing on the record will cause errors, most significantly when trying to convert the record to MARC XML:
>> pymarc.record_to_xml(new_rec) # Causes a TypeError exception trying to serialize a byte string
The problem occurs in line 367 in record.py when the strleader is being encoding to bytes and assigned to the Record instance's leader. Before suggesting a fix (I can think of a couple and I can submit a pull request) are there any reasons why this behavior is preferred in the as_marc() method? (Personal bias would be that a serialization method like as_marc() not alter the underlying MARC data structure)
Thanks,
Jeremy
I ran setup.py install but it doesn't seem to be able to fetch the required file - setuptools version 0.6c5 . The designated link gives me 404: http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c5-py2.7.egg
Help? Thx!
Documentation doesn't match code, creating confusion for programmers:
def delete_subfield(self, code):
"""
Deletes the first subfield with the specified 'code' and returns
its value:
field.del_subfield('a')
If no subfield is found with the specified code None is returned.
"""
I came across some records in the wild which had the eszett in them and noted that the existing marc8_mapping.py
doesn't have a mapping for that character (UTF-8: U+00DF).
It looks like the LC Code Tables for MARC-8 mappings were updated in 2004: see https://memory.loc.gov/diglib/codetables/45.html which might explain how the character (and the Euro symbol) are overlooked.
I can provide an updated file in a pull request.
But there are a a couple of other changes listed that aren't reflected in the mapping:
See:
Revised June 2004 to add the Eszett (M+C7) and the Euro Sign (M+C8) to the
MARC-8 set.
Revised September 2004 to change the mapping from MARC-8 to Unicode for
the Ligature (M+EB and M+EC) from U+FE20 and U+FE21 to U+0361.
Revised September 2004 to change the mapping from MARC-8 to Unicode for
the Double Tilde (M+FA and M+FB) from U+FE22 and U+FE23 to U+0360.
Revised March 2005 to change the mapping from MARC-8 to Unicode for the
Alif (M+2E) from U+02BE to U+02BC.
So the question is how to handle the revised mappings? Just do the right thing right now? Keep doing the old behavior? Its easy enough with the new characters but the changes might be problematic for some?
It would be nice to get pymarc up on readthedocs.
Below is a failing unit test. As I understand it subscript 2 can be represented in MARC-8 like this: ESC b 2 ESC s.
pymarc doesn't handle this correctly. Is it a bug, or am I doing something wrong?
from pymarc import marc8_to_unicode
assert marc8_to_unicode('CO\x1bb2\x1bs is a gas') == u'CO\u2082 is a gas'
a pymarc.field.Field object has both a .indicators attribute (which is a list of 2 indicators), and .indicator1 and .indicator2 attributes, which are strings that are set initially to the same values as in indicators.
Field.str uses the indicators list to determine what indicators to display in the textual format, while as_marc uses the individual attributes.
The upshot of this is that you if you want to edit indicators in the MARC you'll output, you need to change the individual attributes, but printing the record as text will make it appear you didn't change anything, while changing .indicators[1], for example, will appear to edit the indicator when printing for debugging, but when the MARC is written out it will still have the old value.
pymarc.field.Field.__str__()
uses native str as the arguments to self.data.replace
, but in a RawField, data
is a bytes object, so we get TypeError: 'str' does not support the buffer interface
RawField
should probably have its own implementation of __str__
, although it's unclear to me how we should represent a field with an unknown encoding as text...
https://pypi.org/project/pymarc/#description says: The author of this package has not provided a project description
. Can this be prevented by displaying README.md
's content there? Or some of it? Maybe even in a PyPI-standard way?
test_url in test/reader.py is failing because the resource http://inkdroid.org/data/marc.dat doesn't exist anymore.
When I am Iterating through a marc file containing multiple MARC records, After the last record the following exception is raised.
Traceback (most recent call last):
File "compare_marcs.py", line 320, in
pprint(compare_marc_files(exp, obs))
File "compare_marcs.py", line 299, in compare_marc_files
for exp_marc, obs_marc in izip(expected, observed):
File "/home/shinto/shinto/virtualenvs/marc/local/lib/python2.7/site-packages/six.py", line 558, in next
return type(self).next(self)
File "/home/shinto/shinto/virtualenvs/marc/local/lib/python2.7/site-packages/pymarc/reader.py", line 97, in next
utf8_handling=self.utf8_handling)
File "/home/shinto/shinto/virtualenvs/marc/local/lib/python2.7/site-packages/pymarc/record.py", line 74, in init
utf8_handling=utf8_handling)
File "/home/shinto/shinto/virtualenvs/marc/local/lib/python2.7/site-packages/pymarc/record.py", line 245, in decode_marc
raise BaseAddressInvalid
pymarc.exceptions.BaseAddressInvalid: Base address exceeds size of record
I've hit a problem that PyMARC doesn't handle the alternative graphic character set properly if they it appears at the end of a field. The example code below gives "IndexError: string index out of range"
from pymarc import marc8_to_unicode
assert marc8_to_unicode('CO\x1bb2\x1bs') == u'CO\u2082'
Here is a failing test and a sample MARC record
Looks like this was introduced in 022172c and e1b1f9a by me. :(
Really, Record.as_marc() should do some checking to see if leader/09 is set to "a" or perhaps have as_marc() passed a boolean argument. Currently, the Record constructor has an optional a "force_utf8" boolean argument which is thus far only used by decode_marc(). We could use it to set an attribute for the object.
Either way, currently, pymarc fails when trying to write MARC-8 encoded records. Presumably there are going to be some cases where people want to script a minor change and still output the record in MARC-8.
I use pymarc to generate iso2709 of records from internals datas.
When I read my records generated by pymarc (even with pymarc!) I have directory offset problem.
With yaz-marcdump the error is:
(Directory offset 204: Bad value for data length and/or length starting (394\x1E##\x1Fa9782))
(Base address not at end of directory, base 194, end 205)
(Directory offset 132: Data out of bounds 51665 >= 15063)
What's wrong ?
For a 001 field, which doesn't have any subfields, trying to fetch them generates an exception rather than returning empty list.
f.get_subfields()
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/site-packages/pymarc-3.0.1-py2.6.egg/pymarc/field.py", line 155, in get_subfields
for subfield in self:
File "/usr/lib/python2.6/site-packages/six-1.7.3-py2.6.egg/six.py", line 518, in next
return type(self).next(self)
File "/usr/lib/python2.6/site-packages/pymarc-3.0.1-py2.6.egg/pymarc/field.py", line 126, in next
while self.__pos < len(self.subfields):
AttributeError: 'Field' object has no attribute 'subfields'
I was getting the error: couldn't find 0x20 in g0=83 g1=69
That error eventually, led me down the right path, to the fact that the data was corrupt / was in the wrong encoding type. It should have shown Pi Squared, but instead it showed some weird characters.
An error that says something about unable to parse characters would have been helpful. I get that may be what it's saying, just in a more esoteric form, but it would be nice if it said something more like:
Unable to parse character 0x20 in g0=83 g1=69.
https://github.com/edsu/pymarc/blob/master/setup.py#L24 should cause PyPI to register that both Python 2 and Python 3 are supported.
The only subclass of Writer we have is MARCWriter.
I would like to create a few more subclasses of Writer.
My motivation here is so I can code to the generic Writer interface and plug in different writers as need be. Particularly I want to plug in a TextWriter during development, but a MARCWriter in production.
We are still trying to find out how to parse and serialize MARCXML. I then stumbled upon #73 (make it simpler) but one or two examples in the documentation may be enough to start with:
P.S: See also this help request.
from the mailing list:
But maybe it would be useful if the Field.add_subfield method took an optional 0 based position that defaults to the end of the field to preserve existing behavior? For example:
field = record.title() field.add_subfield('a', 'Middlemarch :', pos=0)
This shouldn't be difficult to implement, and seems a lot nicer than manipulating the internal heterogeneous list of code, data, code, data manually.
Thank you for pymarc!
But pymarc can't handle embedded fields in UNIMARC scheme. Eg:
EX 1 Embedded fields technique
200 1#$aKesteven chronicle ...
205 ##$aFosse Way ed.
430 #1$12001#$aLincolnshire chronicle$1205##$aNorth Kesteven ed.
More info: http://archive.ifla.org/VI/3/p1996-1/uni4.htm#410
Unless I am missing something with respect to the marcxml functionality, the record_to_xml function seems to return text encoded in us-ascii, which causes problems when systems are expecting utf-8 encoding. Tracing this issue to its source revealed that xml.etree.ElementTree.tostring takes an optional encoding parameter, which defaults to us-ascii. I am proposing to be able to pass an optional encoding parameter from marcxml.record_to_xml's invocation of ET.tostring.
In my local fork, I have made the following change:
def record_to_xml(record, quiet=False, namespace=False, encoding='us-ascii'):
node = record_to_xml_node(record, quiet, namespace)
return ET.tostring(node, encoding=encoding)
Without the change, my output for record_to_xml on UTF-8 strings that contain diacritics looks like this:
<record>
<leader> 22 4500</leader>
<datafield ind1=" " ind2=" " tag="246">
<subfield code="a">Nouvelles-Hébrides, communiqués par le gouvernement de la France et par le gouvernement du Royaume-Uni de Grande-Bretagne et d'Irlande du Nord :</subfield>
<subfield code="b">Lois et règlements promulgués pour donner effet aux dispositions de la Convention du 13 juillet 1931 pour limiter la fabrication et réglementer la distribution des stupéfiants, amendée par le Protocole du 11 décembre 1946</subfield>
</datafield>
</record>
And the resulting file ends up with a us-ascii encoding, which causes import of the record to fail on the MARC based system we are using.
With the change, I get output that looks like this when I pass the optional encoding:
<record>
<leader> 22 4500</leader>
<datafield ind1=" " ind2=" " tag="246">
<subfield code="a">Nouvelles-Hébrides, communiqués par le gouvernement de la France et par le gouvernement du Royaume-Uni de Grande-Bretagne et d'Irlande du Nord :</subfield>
<subfield code="b">Lois et règlements promulgués pour donner effet aux dispositions de la Convention du 13 juillet 1931 pour limiter la fabrication et réglementer la distribution des stupéfiants, amendée par le Protocole du 11 décembre 1946</subfield>
</datafield>
</record>
I invoke as follows:
out_file.write(marcxml.record_to_xml(record,encoding='utf-8'))
And the resulting file ends up with a utf-8 encoding.
Note that I tried forcing encoding to utf-8 at each successive level beginning with the open() function and working backward to the record itself. The only thing I found that actually works is to pass an encoding parameter in this particular function. If I am missing something (obvious or not), I'd be interested in correcting my oversight.
The change looks trivial to me and preserves the default functionality, but I don't know if there are tests that depend on it.
In the interest of collecting relevant projects to code4lib, could that be added to the repository as a topic? It'll help people discover software during their GitHub searches.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.