OutOfMemoryError using interpolate! about geoarrays.jl HOT 4 CLOSED

Happy Newyear! ;)

You have a huge grid that's mostly empty (~25M cells of which ~500k have data, so only 2%) and default Kriging probably won't handle the 500k datapoints.

I'd try at least using another cellsize (this is not necessarily in meters) to get a smaller, filled raster.

Furthermore you could tune the Kriging solver (with something like distance/max neighbors). You could also look at https://github.com/juliohm/GeoStats.jl#estimation-problems to find a less demanding solver (such as InvDistWeight).

from geoarrays.jl.

I have a different grid I am actually interested in working with, but that layer showed the same behaviour.

Here is a summary of the grid, it has some holes but not as bad as the file above.

#num missing cells
julia> count(ismissing,canopy)
14571

#num filled cells
julia> count(!ismissing,canopy)
1118743

#total grid size
julia> size(canopy)[1] * size(canopy)[2]
1133314

#% empty cells
julia> 14571/1133314*100
1.2856984030904057

I tried a couple different solvers as you suggested, they don't error but take very long. I interrupted all of these after they did not complete after 10 mins.

GeoArrays.interpolate!(canopy,InvDistWeight())

GeoArrays.interpolate!(canopy,InvDistWeight(:variable => (neighbors=3,)))#parms from tests

GeoArrays.interpolate!(canopy,LocalWeightRegress())

I am trying to replicate some default behavior from a different software that interpolates automatically.I don't know what kind of interpolation is being used. That software works on the 1m cell size, so I would prefer to keep that the same size.

I'll try to look at some parameters and/or get some @btime info.
The layer is not that big so I am confused at the performance.

from geoarrays.jl.

By default InvDistWeight uses all neighbors for lookup, so the complete domain, which will grow explosively (power of 4) if you go from a 100 by 100 to a 1000 by 1000 raster. You need to specify neighbors and the correct variable. Last thing could be documented better, see the source @ https://github.com/evetion/GeoArrays.jl/blob/master/src/interpolate.jl#L4

GeoArrays.interpolate!(canopy,InvDistWeight(:z => (neighbors=3,)))
GeoArrays.interpolate!(canopy, InvDistWeight(:variable => (neighbors=3,)), 1, :variable)

from geoarrays.jl.

OK, that worked.
I also needed to add a using StaticArrays

Here is a fully reproducible example that works in case someone finds it useful later:

using GeoArrays
using Random
rng = MersenneTwister(1234);
example = GeoArray(Array{Union{Missing, Float64}}(rand(rng, 500, 500)))

#~2% missing rate
for i in 1:5000
example.A[rand(rng,1:500),rand(rng,1:500)] = missing 
end

count(ismissing,example)
count(!ismissing,example)

using GeoStats
using StaticArrays
using InverseDistanceWeighting

GeoArrays.interpolate!(example,InvDistWeight(:z => (neighbors=3,)))

from geoarrays.jl.