curationexperts-deprecated / darlingtonia Goto Github PK

ACCEPTANCE

• Default to a standard logfile location

ACCEPTANCE

• I can quickly check the build status by seeing the build badge on the repo's main readme.

ACCEPTANCE

• Allow me to configure the log location & name
• Allow me to display output on STDOUT optionally

Requires #36

ACCEPTANCE
Review https://github.com/curationexperts/darlingtonia/branches/all and

• Remove, rebase, or merge any branches significantly behind master

• CI should be green

ACCEPTANCE
Given a unique field that I can match records on.

• I can configure the importer to update the metadata (completely overwrite) for the matching record
• Files are not reimported

TODO - add import table truth table

This mapper should support the Hyrax BasicMetadata and CoreMetadata from a hash-like (CSV) import format.

Every Import

Should display the following info for each record being processed:

• Source record unique identifier
  • Filename OR record number in stream OR unique ID in record
# i.e. if there’s an error reported, how do I find the thing I need to go back and fix?
• Any record validation (input format) errors
• Any work validation (model level) errors
• File(s) being attached
  • Any file-level errors (missing files, etc.)

Should display a summary of the overall import run

• Number of input records read
• Number of new objects created
• Number of existing objects
  • skipped (if the rule is to skip existing)
OR
  • updated (if the rule is to update existing)
• Number of objects not created due to errors

Timestamps would be nice for profiling

• Start and end time for each record
  • OR (even better) start and duration for each record
# this gives us important performance info

Logs should be easily machine parsable for easy integration into Splunk and other log aggregation systems.

BONUS: Configurable verbosity

As a repository owner, I'd like a quick way to get my application setup for batch ingest.

ACCEPTANCE
Given a CSV with headings matching the Hyrax Basic Metadata Fields and corresponding content (single files per row)

• There is a guide that walks me through setting up a vanilla Hyrax app to import my content

• There is a guide that walks me through custom field mappings for

  • additional fields
  • non-string fields types
  • fields with some kind of transformation

ACCEPTANCE
Provide a summary of as many of the following stats as feasible (in the scoped time)

• Number of input records read
• Number of new objects created
• Number of objects not created due to errors

ACCEPTANCE

• Provide 2+ options for performance profiling
• Provide a recommendation from the options (with team discussion if needed)
• Write up an implementation ticket

"Parser" is already proving to be a slightly unwieldy name. We chose this name to differentiate from an Importer and "parser" seemed viable given that examples read (parsed) records from a manifest file. However, the laevigata importer will not parse records from a manifest, but instead draw them from a live Fedora 3 instance.

In this case, the job of the so-called Parser is to provide a stream of mapped and validated InputRecord objects to the Importer. It turns out that "parsing" is an implementation detail.

Perhaps RecordStream?

I don't expect major Parser API changes to accompany any name change. We'll need to deprecate the old class names for removal in 2.0.0, however.

• Publish Darlingtonia YARD docs either on github.io or on a hosted service

mahonia formats its error and info streams. We should provide this feature here as a class that can be composed with other streams.