catmandu.yml:

importer:
  default:
    package: YAML
    options:
       file: data.yml

code:

use Catmandu -load;

Catmandu->importer; # Doesn't work
Catmandu->importer('default'); # works

Would like to see a way to e.g.

Support ways to plugin your own hash implementation (e.g Tie::IxHash, or any favorite implementation)

Support ways to add side-effects when executing fixes (e.g. logging, error handling, exports if individual records instead of the complete iterator).

(read monads)

It would be cleaner for Catmandu::Fix to put the Fix Path parser ("foo.bar.1.*.name") in a separate module. This would ease the documentation and testability of the Path language and provides maybe a way to plugin other Path parsing modules.

Test

Delete every key from foo except bar and doz:

retain_field('foo.bar','foo.doz');

One might also support this syntax:

retain_field('foo.(bar|doz)');
retain_field('bar|doz'); # remove all keys but bar and doz

My current workaround to remove all keys but bar and doz:

move_field("foo","tmp.foo");
move_field("bar","tmp.bar");
retain_field("tmp");
move_field("tmp.foo","foo");
move_field("tmp.bar","bar");
remove_field("tmp");

shows help on Catmandu::Importer

catmandu convert --help

should show help/documentation on Catmandu::Importer::CSV:

catmandu convert CSV --help

should show help/documentation on Catmandu::Exporter::CSV:

catmandu convert to CSV --help

The help command may read more information from the module's POD. Right now I am regularly switching between command line and metacpan to find the right command line arguments.

In version 0.8007 of Catmandu, the command "catmandu" shows this error:

Can't use an undefined value as an ARRAY reference at /home/njfranck/git/imaging/local/lib/perl5/Catmandu/CLI.pm line 33

Probably due to $lib_path being undefined.

#!/usr/bin/env perl
use Catmandu::Sane;
use Catmandu::Fix;
use Data::Dumper;

my $hash = {
  a => [1..4],
  b => []
};

my $fixer = Catmandu::Fix->new(fixes => ["copy_field('a.*','b.\$append')"]);
print Dumper($fixer->fix($hash));

output:

$VAR1 = {
          'a' => [
                   1,
                   2,
                   3,
                   4
                 ],
          'b' => [
                   1
                 ]
        };

Only the first value from "a" is copied

With YAML::XS, which is preferred by YAML::Any, the YAML importer fails when importing UTF-8 YAML files:

$ echo "umlaut: Ü" | catmandu convert YAML
YAML::XS::Load Error: The problem:

    invalid trailing UTF-8 octet

was found at document: 0

This behaviour of YAML::XS is documented (see https://rt.cpan.org/Public/Bug/Display.html?id=54683) and won't be changed. I suppose it can be fixed by adding an encoding parameter, but this is not documented and by a bug in Catmandu::Importer or Catmandu::App::convert the parameter is not passed to the file handle anyway. This should work (it does when hard-coding "raw" in Catmandu::Importer!), but it does not:

$ echo "umlaut: Ü" | catmandu convert YAML --encoding :raw

Even when it worked, the defaul!t encoding setting (:utf8) is annoying at least for YAML. I have not tested with JSON and I won't invest more work in fixes that don't get released anyway :-(.
.

I need something that moves a deep nested leaf to the root.

E.g.

{ "my": { "field": 123 }}

move_field("my.*",""); # { "field": 123 }

nichtich@033d096 (proposed as pull request #40) allows this:

catmandu -I lib convert MyFormat to JSON < file.myformat

where Catmandu/Importer/MyFormat.pm is located in lib. My current nasty workaround is

perl -Ilib `which catmandu` convert MyFormat to JSON < file.myformat

It's covered by a unit test. Looks like Catmandu::CLI was not being tested before.

There are Git merge markers (I think they are called) in https://github.com/LibreCat/Catmandu/blob/master/lib/Catmandu/Introduction.pod#importer

The current JSON importer can only handle line-based JSON as emitted by the JSON exporter. One should also be able to parse arbitrary JSON documents, including multiple objects in one file. For instance the following file includes three JSON objects:

{ "id": 1 }
{ 
  "id:" 2
}

{ "id": 
3 }

In addition the importer should support a path option as introduced in Catmandu-Importer-XML. For instance the following file with option path=/record.* could import the same three objects:

{
  "records": [ { "id": 1 }, { "id": 2 }, { "id": 3 } ]
}

This Perl Module may help implementing.

Maybe we can provide vagrant or something images to easily create a virtual box with Catmandu + ElasticSearch + MongoDb preinstalled.

The Programmers Guide tutorial contains a section showing basic usage of the TT2-based exporter. When trying out the following code snippet, it took me a long time to figure out why no output was produced:

use Catmandu::Exporter::Template;
my $data     = [
 { name => { first => 'James' , last => 'Bond' } , occupation => 'Secret Agent' } ,
 { name => { first => 'Ernst' , last => 'Blofeld' } , occupation => 'Supervillain' } ,
];
my $exporter = Catmandu::Exporter::Template->new(template => '/home/phochste/example.tt');
$exporter->add_many($data);

Of course, I am not phochste, so I changed the path to the template file into "example.tt", expecting the exporter to read the template from the current directory. That was a mistake. This value must be a full, absolute path, and no tilde (~) expansion is performed.

Perhaps this module should make more of an effort to find the template file. In any case, some kind of error message, warning or exception should be produced when the template file cannot be found.

Using Catmandu::Exporter::JSON gives you something like that:

{name: "Catmandu", creator: "Nicolas"}
{name: "Perl", creator: "Larry"}

But for a valid JSON document it should be:
[ {name: "Catmandu", creator: "Nicolas"},
{name: "Perl", creator: "Larry"} ]

e.g

if_exists('some_field');
skip_record();
end();

This would make it easier to create a UNIX grep in catmandu style.

Catmandu.pm fails on Win. Probably just because of the different file system structure.

This shell command line works as expected, downcasing the value of the job feature:

$echo '{"job":"Artist"}' | catmandu data --fix 'downcase("job")'
{"job":"artist"}

When trying to fix the value of a non-existing feature, a new feature with this key and a null value is created:

$ echo '{"job":"Artist"}' | catmandu data --fix 'downcase("occupation")'
{"occupation":null,"job":"Artist"}

This is should not happen.

We use emit functions in Fix modules to parse JSON paths and make the Fix code run very efficient over deeply nested hashes/arrays. But the emit logic is quite verbose..and something we might change in the future.

However, there are some patterns in when creating Fixes that are repeated over and over and could easily be factored out in a separate module.

e.g.

Fix: on a path change its value to another value could be implemented as

package SomeFix;

with 'EasyEmit';

sub on_path {
my $self = shift;
my $value = shift;
$value = somefunc($value);
}

1;

Catmandu::SRU is not usable for data such as MARCXML and PICAXML because it corrupts the order of XML elements. The module should not use XMLIn from XML::LibXML::Simple but XML::LibXML::Reader and emit a different object format for mapping XML.

Mapping XML to record structures with irreversible order is common and useful for specific formats. See this summary for most common mapping rules. Whether this mapping as done by XML::Simple is suitable, however, depends on the data format.

I'd choose a format as sketched here. Catmandu::SRU should neither parse XML but only unpack the SRU response and direct XML records to a Catmandu::Importer::XML.

Catmandu::Development explains to use the devel branch. It looks like there is no such branch but it has been renamed to dev (?).

How about a path operator ... that selects of fields and subfields?:

# { foo => "  x ", bar => { foo = "  y "} }
trim("...foo") # { foo => "x", bar => { foo = "y"} }

the config branch already uses Dave's Config::Onion and gives us config merging, local configs etc BUT this breaks the subkey feature in catmandu, e.g. catmandu.list.yml was loaded into a 'list' subhash (is this a problem for anyone?)

Would be nice to have some crap detector software to find bad characters or encoding problems.

E.g.
--find patterns of double encoded UTF08 in data input
--find valid but illegal control fields (e.g. these darlings kept me busy for some hours today: http://unicode-search.net/unicode-namesearch.pl?term=separator)

Any ideas which modules could be of help?

Create Catmandu::Registry for handling bundles:
Catmandu::Registry

• loads list of bunldes
• register & reveive events
• manages list of services

Bundle of functionality

• load/unload
• contains new model
• contains Plack middleware/Plack application
• config files
• event handling / messaging
• templates
• files
• js, css

... Dist::Zilla, Dist::Milla, or Minilla. See this nice article about Module::Build: http://www.dagolden.com/index.php/2140/

The 'encoding' of an Catmandu::Importer cannot be changed:

sub encoding { ':utf8' }

Is there a reason not to make an ordinary attribute for this?

thanks

there are way too many general functions in there; look into Type::Tiny, Path::Tiny etc. as a replacement

Just for you info: I'm working on that.

Step 1: Separate

Catmandu::Exporter::BibTeX

Catmandu::Exporter::XLS

Catmandu::Importer::Atom

Catmandu::Store::DBI (?)

Step 2:

Add to Task::Catmandu

Many data conversion tables consist of simple mapping tables given as spreadsheets/csv. Fields included in a mapping table are moved/renamed andy fields not notnot included are removed. How about using csv as input format for such mapping fixes?

At http://librecat.org/catmandu/2013/07/05/catmandu-fix-cmd.html it is documented to use the cmd fix like this:

catmandu convert JSON --fix 'cmd("./catjson.py")'

This should be simplified to

catmandu convert JSON --fix catjson.py

If the fix equals to a filename that is executable, then the cmd fix should be applied with this file. Normal fix files should not be executable.

Why can't I just type

catmandu import JSON test.json to DBI dbi:SQLite:test.sqlite

The primary and most common option to Catmandu::Importer is file and the primary, mandatory option to Catmandu::Store is data_source, so the call above should be expanded to

catmandu import JSON --file test.json to DBI --data_source dbi:SQLite:test.sqlite

Typical usage should be simple and non-typical usage should be possible.

I encountered a few small issues when working through http://librecat.org/tutorial/index.html ...

People who have not used cpan a lot will have to spend an enormous amount of time confirming each dependency when doing sudo cpan Catmandu. The following lines make sure dependencies are installed fully automatically:

    $ sudo cpan
    o conf prerequisites_policy follow
    o conf build_requires_install_policy yes
    o conf commit
    install Catmandu
    q

Some code examples did not work immediately because of missing semicolons at the end of lines:

    printf "SUM [ Iterator * 2] = %d\n" , $result
    use Catmandu::Exporter::YAML

this removes the dependency on Template

We just had a Catmandu workshop and the question arose how to sort lists and how to remove duplicates in a fix. I think sort should be limited to simple lists of strings but uniq may be more complex if it also applies to nested structures (not the most common use case).

It would be very helpfull if we could do things like

$ catmandu convert JSON --url http://somewhere.org/data.json to YAML

require_ok is not testing

ONIX
EAD
METS

And all the rest
http://en.wikipedia.org/wiki/Metadata_standards