Order or `grid_cells` and `grid_coordinates()` about atlite HOT 2 CLOSED

I never addressed this properly, because I was extremely busy and unhappy with personal matters; and it's quite subtle.

The short answer is: Yes, of course, if we change the ordering of the y dimension of the coordinate system, then it is necessary that we change the order of the columns in the matrix and, since grid_coordinates/grid_cells returns the basis of the columns it must change accordingly as well.

Do people expect this? I don't know. I thought it was obvious. Quite obviously, I'm wrong.

The code you are referring to is hideous because you are not following the convention that y is ascending in the layoutmatrix. And yes, it seems like sparse does not help you too much to compensate that.

What you could do, for example, is to switch to this convention early on, ie. use

    availabilities = xr.DataArray(
        np.array(availabilities),
        coords=[
            buses,
            y[::-1], # GDAL uses a north to south latitude
            x
        ],
        dims=['bus', 'y', 'x']
    ).isel(y=slice(None, None, -1))

for https://github.com/PyPSA/pypsa-eur/blob/f5382021a42dcd3fb55cfe18d3648972857d653f/scripts/build_renewable_profiles.py#L343-L350 .

Or embrace it even earlier just after calling extractMatrix ie use

    return gk.raster.extractMatrix(availability)[::-1] # GDAL uses a north to south latitude

Or just don't use sparse and stick with the final matrix structure as csr by following my initial advise #20 (comment) .

I don't understand the possible solution, you mention. The index order is fixed now as ascending y and x, we do not need any sort operations for this. I do not think it is less good than the other possible convention. I can come up with pros and cons for each, if we want to compile a list.

There will always be different conventions, it is important to communicate them clearly, but ultimately it is up to the users to follow them!

And I would even go a bit further and say that any mechanisms that try to predict and compensate for user errors automatically will on the long run lead to further and then less obvious mistakes.

from atlite.

Adressed in 12a86b3 in #20 .

A few comments:

Do people expect this? I don't know. I thought it was obvious. Quite obviously, I'm wrong.

I doubt it. One would have to care about the index order in cutouts, know that it changes and know that the functions are internally designed that way (no doc string mentioning this).

What you could do, for example, is to switch to this convention early on, ie. use
...

Thanks for the advice. Since the changes affect more than just this piece it is insufficient.
I prefer to keep the meta data and data close together with sparse, making the conversion step more obvious via reindex_like suits my style more than multiple list reversals with [::-1].

There will always be different conventions, it is important to communicate them clearly, but ultimately it is up to the users to follow them!

Of course. But we never exposed or documented them.

And I would even go a bit further and say that any mechanisms that try to predict and compensate for user errors automatically will on the long run lead to further and then less obvious mistakes.

I fully agree!

from atlite.