gitpan / test-filereferenced Goto Github PK

NAME
    Test::FileReferenced - Test against reference data stored in file(s).

SYNOPSIS
     use Test::FileReferenced;
 
     # Optional:
     Test::FileReferenced::set_serializer('mydump', \&My::Dumper::Load, \&My::Dumper::Dump);
 
     is_referenced_ok( complex_data_structure(), "complex data structure" );
 
     is_referenced_in_file ( data_structure(), "data structure", "data_structure" );
 
     # Optional:
     Test::FileReferenced::at_exit();

DESCRIPTION
    Test::FileReferenced helps testing routines returning complex data
    structures. This is achieved by serializing test's output (using
    YAML::Any), and allowing the Developer to compare it with reference
    data.

    In case there are differences between reference and actual result,
    comparison can be made using traditional UNIX diff-like (diff, vimdiff,
    gvimdiff, kdiff) utilities.

    In such case, Test::FileReferenced - after the test completes - will ask
    the Developer to run diff on result and reference data. If all
    differences ware intended, Developer may just replace reference data
    with actual test results.

SUBROUTINES
    is_referenced_ok ( $data, $name, $comparator )
        Compare $data with reference stored under key $name in default
        reference file.

        If $comparator is a CODE reference, it is used to compare results.
        If this parameter is not given, Test::More::is_deeply is used.

        Returns:

        Value returned by comparison routine. By default (when is_deeply is
        used) it will be 1 if the test passed, and 0 if it failed.

    is_referenced_in_file ( $data, $file_basename, $name, $comparator )
        Compare $data with reference stored in custom file:
        $file_basename.yaml (assuming the serializer is YAML::Any).

        If $comparator is a CODE reference, it is used to compare results.
        If this parameter is not given, Test::More::is_deeply is used.

        Both $name and $comparator are optional parameters.

        Returns:

        Value returned by comparison routine. By default (when is_deeply is
        used) it will be 1 if the test passed, and 0 if it failed.

    set_serializer ( $extension, $load_coderef, $dump_coderef )
        Changes default serializing functions to ones provided by the
        Developer. $extension must also be provided, so Test::FileReferenced
        can automatically create the default reference file, if needed.

        You do not need to use this function, if You are happy with
        YAML::Any usage.

        Returns: undef

    at_exit ()
        If there ware failed tests, "at_exit()" will dump results from the
        test in temporary file, and then prompt to inspect changes.

        If there ware no failures, "at_exit()" will check, if results file
        (from any previous run) exists, and if so - remove it. Nothing will
        be printed in this case.

        Normally this function does not need to be run explicitly, as
        Test::FileReferenced will run it from it's "END {}" sections.

        Returns: undef

REFERENCE FILES
    Reference files are data dumps using - by default - YAML::Any.

    Default reference file
        Default reference file contains data for all "is_referenced_ok"
        calls in the test. Each test case has it's own key in the file. For
        the following example test:

         is_referenced_ok(\%ENV, 'env');
         is_referenced_ok(\@INC, 'inc');

        ...we have the following reference file:

         ---
         env:
           LANG: pl_PL
           LANGUAGE: pl_PL
           LC_ALL: pl_PL.UTF-8
         inc:
           /usr/lib/perl5/site_perl
           /usr/lib/perl5/vendor_perl/5.10.1
           /usr/lib/perl5/vendor_perl
           /usr/lib/perl5/5.10.1

        Name for the reference file is based on the tests's filename, with
        *.t* replaced with extension native to the used dumper. Example: if
        default serializer (YAML::Any) is used, foo/bar.t will use
        foo/bar.yaml.

    Custom reference files
        Custom reference files are used by "is_referenced_in_file" function.
        Each file contains reference data for single test case. For the
        following example test:

         is_referenced_in_file(\%ENV, 'env', 'environment');

        ...we have the following reference file, named environment.yaml:

         ---
         LANG: pl_PL
         LANGUAGE: pl_PL
         LC_ALL: pl_PL.UTF-8

TEST FAILURES
    If there are differences between referenced, and actual data, at the end
    of the test prompt will be printed, similar to:

     Resulting and reference files differ. To see differences run one of:
           diff foo-results.yaml foo.yaml
       gvimdiff foo-results.yaml foo.yaml
 
     If the differences ware intended, reference data can be updated by running:
             mv foo-results.yaml foo.yaml

    If there is no foo.yaml yet (first test run, for example) then the
    message will be similar to:

     No reference file found. It'a a good idea to create one from scratch manually.
     To inspect current results run:
            cat foo-results.yaml

     If You trust Your test output, You can use it to initialize deference file, by running:
             mv foo-results.yaml foo.yaml

    In this case, the first time is_referenced_ok is used, it will dump the
    following diagnostic message:

     No reference file found. All calls to is_referenced_ok WILL fail.

    This is to ensure, that the User get's the idea, that something is not
    OK, even if - for some reason - the END block does not run.

CUSTOM COMPARISON ROUTINES
    For the moment, it's an undocumented, experimental feature. Use at Your
    own risk.

TDD
    Test-driven development is possible with Test::FileReferenced. One of
    the ways, is to follow the following steps:

    Initialize reference files
        To initialize the reference file(s), run a script similar to the
        example bellow:

         #!/usr/bin/perl -w
         use strict;
         use Test::More tests=>3;
         use Test::FileReferenced;

         is_referenced_ok(undef, "First test");
         is_referenced_ok(undef, "Second test");

         is_referenced_in_file(undef, "foo", "Second test");

        This will allow You to create an empty default reference file for
        the test, and one ('foo.yaml') custom reference file.

    Fill reference files
        At this point, test should pass cleanly. Our goal is to write the
        data structures, that We expect to have, into reference files
        created above.

        After doing this, test will no longer pass.

    Generate test data
        At this point, test fails because test script provides incorrect
        data: undef's have to be replaced with actual data - probably
        generated by calls to tested subroutines.

    Implement tested code
        At this point, test still fails. Tested subroutines have to be
        properly implemented. Once this is done, test should pass, and the
        process is completed.

CAVEATS
    Most caveats listed here will - most probably - apply to any other Test
    module. They have been listed for convenience, as they have been been
    found to be the most common issues a Developer might run into, while
    using Test::FileReferenced.

    Random ordering
        Note, that Test::FileReferenced does not sort the data. If Your data
        is returned in random order (order is not actually important), You
        should use the following:

         is_referenced_ok( [ sort @randomly_ordered_data ], "Test 01" )

    Date and/or time
        Your reference data is 'frozen' as it is in given time point. If
        results contain some elements derived from date/time, they will be
        different each time You run the test. This will most likely create
        false negative results.

    Host-based data
        If Your test data contains some host-related data (URLs), tests will
        pass on Your host, but will probably fail on other machines.

TODO
    Make result files as unique as possible
        Result files should be unique (add PID? Timestamp?), so it is
        possible to run the same test in two copies at a time. At the moment
        race conditions may happen. This does not seem to be a common use
        case, but still.

        Will be fixed in next (0.02) version.

    External tmp directory
        At the moment, result files are written in the same directory as
        tests, which may not always be writable. This should be solved by
        using '/tmp', or any other User-supplied directory.

        Will be fixed in next (0.02) version.

    Propose better commands
        Currently, Test::FileReferenced assumes that the User has Unix-like
        commands, like diff, mv and cat.

        On systems, that do not have them, module should work fins, yet the
        usefulness of the prompt will be reduced.

        I do not know how (if) this is important - if You need this to be
        improved, please let me know (patches welcomed).

SEE ALSO
    Test::More

    Test::FileReferenced::Deep (WIP!)

    Test::FileReferenced::Framework (WIP!)

COPYRIGHT
    Copyright 2010, Bartłomiej Syguła ([email protected])

    This is free software. It is licensed, and can be distributed under the
    same terms as Perl itself.

    For more, see my website: http://natanael.krakow.pl/