In the process of tweaking the code for Rasterio 1.0 and Python 3, I was hitting tests/test_dateline.py::test_4326_crossing FAILED

So reverted back to Python 2.7 and Rasterio 0.36, and I'm still hitting that same error locally and on travis CI.

_______________________________________________ test_4326_crossing ________________________________________________

    def test_4326_crossing():
        crs = {'init': 'epsg:4326'}

        boundsArr = densify(make_bounds_array(BoundingBox(170., 45., 190., 50.)))
        assert winding_order(boundsArr) == False

        transformedBounds = transform_bounds(boundsArr, crs)

>       assert winding_order(transformedBounds) == True
E       assert False == True
E        +  where False = winding_order(array([[ 170.  ,   45.  ],\n       [ 175.  ,   45.  ],\n       [ 180.  ,   45.  ...     [ 170.  ,   48.75],\n       [ 170.  ,   47.5 ],\n       [ 170.  ,   46.25]]))

tests/test_dateline.py:23: AssertionError

@dnomadb can you shed some light on what's being tested here and why it's no longer working as expected?

Right now pxm has bash scripts, python scripts, external python commands and bash functions to assert various things about raster data. All of these methods have slightly different modes of operation (some raise exceptions, some return a special value, some use exit codes, etc) and are not very easy to test.

I'd like to move all the boolean tests from pxm to subcommands of raster-tester. Basically anytime we need to ask a yes/no question about a raster or set of rasters, this command should be the place to go.

So far we have

• compare
• isempty

But we could pull in

• aligned
• crosses-dateline
• is_evil
• is_lossy
• validate

This would also give us an opportunity to define and standardize exactly what a Yes/No response should look like and how the bash scripts should handle them

• exit codes
• what gets output to stdout/stderr

https://circleci.com/blog/continuously-deploying-python-packages-to-pypi-with-circleci/

When using GDAL 2.1, we get pixel counts that differ enough to cause some tests to fail:

_______________________________________________ test_cli_okcompare ________________________________________________

    def test_cli_okcompare():
        runner = CliRunner()
        result = runner.invoke(cli, ['compare', 'tests/expected/blobby.tif', 'tests/fixtures/notblobby.tif', '--upsample', '8', '--compare-masked', '--downsample', '64'])
>       assert result.exit_code == 0
E       assert -1 == 0
E        +  where -1 = <Result ValueError('Mask has 2 pixels that vary by more than 16',)>.exit_code

tests/test_cli.py:40: AssertionError
_________________________________________ test_cli_okcompare_bad_no_error _________________________________________

    def test_cli_okcompare_bad_no_error():
        runner = CliRunner()
        result = runner.invoke(cli, ['compare', 'tests/expected/blobby.tif', 'tests/fixtures/notblobby.tif', '--upsample', '8', '--downsample', '64', '--no-error'])
        assert result.exit_code == 0
>       assert result.output == 'not ok - Band 1 has 363 pixels that vary by more than 16\n'
E       assert 'not ok - Ban...ore than 16\n' == 'not ok - Band...ore than 16\n'
E         - not ok - Band 1 has 436 pixels that vary by more than 16
E         ?                     -
E         + not ok - Band 1 has 363 pixels that vary by more than 16
E         ?                       +

tests/test_cli.py:53: AssertionError
_____________________________________________ test_cli_okcompare_rgb ______________________________________________

    def test_cli_okcompare_rgb():
        runner = CliRunner()
        result = runner.invoke(cli, ['compare', 'tests/expected/blobby_rgb.tif', 'tests/fixtures/notblobby_rgb.tif', '--upsample', '8', '--compare-masked', '--downsample', '64', '--flex-mode'])
>       assert result.exit_code == 0
E       assert -1 == 0
E        +  where -1 = <Result ValueError('Band 1 has 128 pixels that vary by more than 16',)>.exit_code

tests/test_cli.py:59: AssertionError
___________________________________________ test_cli_okcompare_rgb_rev ____________________________________________

    def test_cli_okcompare_rgb_rev():
        runner = CliRunner()
        result = runner.invoke(cli, ['compare', 'tests/fixtures/notblobby_rgb.tif', 'tests/expected/blobby_rgb.tif', '--upsample', '8', '--compare-masked', '--downsample', '64', '--flex-mode'])
>       assert result.exit_code == 0
E       assert -1 == 0
E        +  where -1 = <Result ValueError('Band 1 has 128 pixels that vary by more than 16',)>.exit_code

tests/test_cli.py:65: AssertionError

None of these failures occur when using GDAL 1.11. This is the first GDAL 2 regression I've seen but I guess it had to happen sometime.

cc @dnomadb @sgillies

I haven't dug into the reason behind the differences - perhaps some difference in the way the upsampling is performed in GDAL 2? In any case, we'll want to make raster-tester robust under both GDAL versions.

This might be a packaging issue with rasterio wheels on OSX but it only affects this repo so ticketing here...

When install raster-tester locally with the rasterio==0.36 wheels, the tests segfault

tests/test_cli.py::test_cli_okcompare Segmentation fault: 11

Specifically, this command segfaults

$ raster-tester compare tests/expected/blobby.tif tests/fixtures/notblobby.tif --upsample 8 --compare-masked --downsample 64
Segmentation fault: 11

But this command works fine (dropping the --compare-masked)

$ raster-tester compare tests/expected/blobby.tif tests/fixtures/notblobby.tif --upsample 8 --downsample 64
...

If I install rasterio from source ($ pip uninstall -y rasterio && pip install --no-binary :all: rasterio) the segfault no longer occurs. Nor does it occur on local linux containers or the circleci box.

I haven't dug into why the --compare-mask flag would trigger this yet. 😕 Any ideas @sgillies ?

currently raster_tester doesn't run on python 3 or against rasterio 1.0 - need to investigate why and fix things, preferably in a backwards-compatible way.

Running pxm tests and there is some floating point drift causing failures with this error

File "/Users/mperry/venv/lib/python2.7/site-packages/raster_tester/utils.py", line 11, in exception_raiser
    raise ValueError(message)
ValueError: [{'bounds': {
'src2': BoundingBox(left=15419488.8419, bottom=-4109254.64063, right=15458624.6004, top=-4070118.8821299993),
'src1': BoundingBox(left=15419488.8419, bottom=-4109254.6406300003, right=15458624.6004, top=-4070118.88213)}
}]

One potential solution: modify compare_properties to do more lenient comparisons w/in a tolerance when a and b are floats
/cc @dnomadb

Switching to tox for multi-python testing has broken pytest-coverage

Coverage.py warning: No data was collected.

This looks interesting, it might be worth checking out as it gives a single similarity metric that's theoretically insensitive to minor changes

http://bertolami.com/index.php?engine=blog&content=posts&detail=perceptual-hashing
https://github.com/ramenhut/insight

See rasterio/rasterio#679 (comment)

problem: If you have a 4-band RGBA tiff with JPEG compression, doing a downsample read on band 4 causes a segfault.

solution:
don't read band 4 with downsampling, use a different GDAL version, or use config vars in your rasterio env

GTIFF_IMPLICIT_JPEG_OVR=NO

The GDAL version 2.0.0 through 2.0.2 and 2.1.0 are affected. GDAL 1.X, 2.1.1 and 2.0.3 will be 👌 So the risk is pretty minimal. @dnomadb you can close this whenevs, mostly just a PSA

The purpose of raster-tester is to ask yes/no questions about raster data. If the answer is no, that's not an exception, it's just one of the possible expected outcomes. We should specify this behavior in more detail to standardize it.

The following is just a proposal, would love to get feedback/discussion before we start implementing.

If the answer to the question is no:

• the function should return a tuple with boolean and a message like (False, "These rasters are bad and they should feel bad")
• the click command should print the message (to stderr or stdout?), along with the rasters that were being evaluated, not ok: a.tif vs b.tif (These rasters are bad and they should feel bad)
• the click command should exit with code 1

If the answer to the question is yes:

• the function should return a tuple with boolean and a message like (True, "These rasters are perfect in every way")
• the click command should print the message to stdout, along with the rasters that were being evaluated, ok: a.tif vs b.tif (These rasters are perfect in every way)
• the click command should exit with code 0

cc @sgillies @dnomadb

It's already on pypi: https://pypi.python.org/pypi/raster-tester

Any reason we shouldn't make it public?

The only one I can think of is that we have references to pxm and other proprietary stuff in the issues and comments. Maybe we can rename it, make it a rio plugin, clean it up a bit and put in a fresh repo?

In other words "does this raster have any nodata in it.

Should this be:

(a) a separate command
(b) a flag on isempty (eg --inverse)

cc @perrygeo

Ultimately we want something hooked into the upload process that will check for red flags and report meaningful errors to the user.

As a first step, we could add a command to this repo which checked for:

• projection
• tiling
• format
• compression
• nodata/alpha issues
• ???

Manually testing these permutations is a pain and it would be nice to ensure this (and other rio plugins) work across the board. Need to figure out the best way to make that happen with CircleCI.