juliageo / commondatamodel.jl Goto Github PK

View Code? Open in Web Editor NEW

9.0 9.0 3.0 2.94 MB

CommonDataModel.jl is a module that defines types common to NetCDF and GRIB data

License: MIT License

Julia 99.14% Python 0.86%

commondatamodel.jl's People

Contributors

Stargazers

Watchers

Forkers

tcarion keduba

commondatamodel.jl's Issues

Calling `Array` on a zero dimensional variable returns a scalar

using NCDatasets
fn = tempname() * ".nc"
ds = NCDatasets.Dataset(fn, "c")
NCDatasets.defVar(ds, "a", Int32, ())

Then Array returns a scalar:

julia> Array(ds["a"])
-2147483647

It should always return an Array (even if its zero-dimensional)

Run Aqua tests

Currenty aqua tests are not actually run, and there are a lot of ambiguities.

This affects all packages depending on CommonDataModels and also using Aqua. It would also be good to run the other Aqua checks at the same time.

julia> Aqua.test_ambiguities(CommonDataModel)
22 ambiguities found
Ambiguity #1
broadcasted(f::Function, A, B::CommonDataModel.ReducedGroupedVariable) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/groupby.jl:503
broadcasted(f::Function, A::CommonDataModel.ReducedGroupedVariable, B) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/groupby.jl:505

Possible fix, define
  broadcasted(::Function, ::CommonDataModel.ReducedGroupedVariable, ::CommonDataModel.ReducedGroupedVariable)

Ambiguity #2
broadcasted(::typeof(LinearAlgebra.:*ₛ), out, beta) @ LinearAlgebra /opt/julia/share/julia/stdlib/v1.10/LinearAlgebra/src/generic.jl:10
broadcasted(f::Function, A::CommonDataModel.ReducedGroupedVariable, B) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/groupby.jl:505

Possible fix, define
  broadcasted(::typeof(LinearAlgebra.:*ₛ), ::CommonDataModel.ReducedGroupedVariable, ::Any)

Ambiguity #3
broadcasted(::typeof(LinearAlgebra.:*ₛ), out, beta) @ LinearAlgebra /opt/julia/share/julia/stdlib/v1.10/LinearAlgebra/src/generic.jl:10
broadcasted(f::Function, A, B::CommonDataModel.ReducedGroupedVariable) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/groupby.jl:503

Possible fix, define
  broadcasted(::typeof(LinearAlgebra.:*ₛ), ::Any, ::CommonDataModel.ReducedGroupedVariable)

Ambiguity #4
checkbounds(::Type{Bool}, RA::CommonDataModel.ResizableArray, inds...) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/memory_dataset.jl:5
checkbounds(::Type{Bool}, A::AbstractArray, I::Base.LogicalIndex) @ Base multidimensional.jl:849

Possible fix, define
  checkbounds(::Type{Bool}, ::CommonDataModel.ResizableArray, ::Base.LogicalIndex)

Ambiguity #5
checkbounds(::Type{Bool}, RA::CommonDataModel.ResizableArray, inds...) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/memory_dataset.jl:5
checkbounds(::Type{Bool}, A::AbstractArray, I::Base.LogicalIndex{<:Any, <:AbstractVector{Bool}}) @ Base multidimensional.jl:847

Possible fix, define
  checkbounds(::Type{Bool}, ::CommonDataModel.ResizableArray, ::Base.LogicalIndex{<:Any, <:AbstractVector{Bool}})

Ambiguity #6
checkbounds(::Type{Bool}, RA::CommonDataModel.ResizableArray, inds...) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/memory_dataset.jl:5
checkbounds(::Type{Bool}, A::AbstractArray{<:Any, N}, I::AbstractArray{Bool, N}) where N @ Base abstractarray.jl:687

Possible fix, define
  checkbounds(::Type{Bool}, ::CommonDataModel.ResizableArray{<:Any, N}, ::AbstractArray{Bool, N}) where N

Ambiguity #7
checkbounds(::Type{Bool}, RA::CommonDataModel.ResizableArray, inds...) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/memory_dataset.jl:5
checkbounds(::Type{Bool}, A::AbstractArray, i::Union{AbstractArray{<:CartesianIndex}, CartesianIndex}) @ Base multidimensional.jl:681

Possible fix, define
  checkbounds(::Type{Bool}, ::CommonDataModel.ResizableArray, ::Union{AbstractArray{<:CartesianIndex}, CartesianIndex})

Ambiguity #8
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(f, itr::Base.SkipMissing{<:AbstractArray}) @ Base missing.jl:395

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::Base.SkipMissing{<:AbstractArray})

Ambiguity #9
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(f, a::AbstractArray) @ Base array.jl:2682

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::AbstractArray)

Ambiguity #10
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(f, d::AbstractDict) @ Base abstractdict.jl:471

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::AbstractDict)

Ambiguity #11
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(f, a::Array{T, N}) where {T, N} @ Base array.jl:2670

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::Array{T, N}) where {T, N}

Ambiguity #12
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(f, t::Tuple) @ Base tuple.jl:464

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::Tuple)

Ambiguity #13
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(f, s::AbstractString) @ Base strings/basic.jl:666

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::AbstractString)

Ambiguity #14
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(f, Bs::BitArray) @ Base bitarray.jl:1825

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::BitArray)

Ambiguity #15
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(pred, s::AbstractSet) @ Base abstractset.jl:518

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::AbstractSet)

Ambiguity #16
filter(ncv::CommonDataModel.AbstractVariable, indices...; accepted_status_flags) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfconventions.jl:58
filter(f, s::Union{SubString{String}, String}) @ Base strings/substring.jl:279

Possible fix, define
  filter(::CommonDataModel.AbstractVariable, ::Union{SubString{String}, String})

Ambiguity #17
getindex(v::CommonDataModel.AbstractVariable, name::Union{AbstractString, Symbol}) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfvariable.jl:533
getindex(v::CommonDataModel.MemoryVariable, ij...) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/memory_dataset.jl:56

Possible fix, define
  getindex(::CommonDataModel.MemoryVariable, ::Union{AbstractString, Symbol})

Ambiguity #18
getindex(ds::Union{CommonDataModel.AbstractDataset, CommonDataModel.AbstractVariable}, n::CommonDataModel.CFStdName) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/dataset.jl:243
getindex(v::CommonDataModel.MFCFVariable, ind...) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/multifile.jl:252

Possible fix, define
  getindex(::CommonDataModel.MFCFVariable, ::CommonDataModel.CFStdName)

Ambiguity #19
getindex(ds::Union{CommonDataModel.AbstractDataset, CommonDataModel.AbstractVariable}, n::CommonDataModel.CFStdName) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/dataset.jl:243
getindex(v::CommonDataModel.MemoryVariable, ij...) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/memory_dataset.jl:56

Possible fix, define
  getindex(::CommonDataModel.MemoryVariable, ::CommonDataModel.CFStdName)

Ambiguity #20
getindex(v::CommonDataModel.AbstractVariable, name::Union{AbstractString, Symbol}) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/cfvariable.jl:533
getindex(v::CommonDataModel.MFCFVariable, ind...) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/multifile.jl:252

Possible fix, define
  getindex(::CommonDataModel.MFCFVariable, ::Union{AbstractString, Symbol})

Ambiguity #21
reduce(f::Function, gv::CommonDataModel.GroupedVariable) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/groupby.jl:428
reduce(::typeof(hcat), A::AbstractVector{<:AbstractVecOrMat}) @ Base abstractarray.jl:1718

Possible fix, define
  reduce(::typeof(hcat), ::CommonDataModel.GroupedVariable{TV, TF, TGM, TM, var"#s128"} where {var"#s128"<:(AbstractVecOrMat), TV, TF, TGM, TM})

Ambiguity #22
reduce(f::Function, gv::CommonDataModel.GroupedVariable) @ CommonDataModel ~/.julia/dev/CommonDataModel/src/groupby.jl:428
reduce(::typeof(vcat), A::AbstractVector{<:AbstractVecOrMat}) @ Base abstractarray.jl:1715

Possible fix, define
  reduce(::typeof(vcat), ::CommonDataModel.GroupedVariable{TV, TF, TGM, TM, var"#s128"} where {var"#s128"<:(AbstractVecOrMat), TV, TF, TGM, TM})

Test Failed at /home/raf/.julia/packages/Aqua/9p8ck/src/ambiguities.jl:77
  Expression: num_ambiguities == 0
   Evaluated: 22 == 0

ERROR: There was an error during testing

A function to determine to which dataset a variable belongs to

In the CF conventions different variables can be related using shared dimension names and special attributes: for example, bounds or the ancillary_variables attribute. If a function gets only the CommonDataModel.AbstractVariable as an argument it would be useful to query to which dataset this variable belongs to in order to access related variables from the dataset. Another use case would be to get the coordinate variables associated with a given data variable.

Would you agree @tcarion that this would be a useful function to add? What could be a good function name? Maybe dataset(v) or parent_dataset(v) ?
I have a slight preference for just using dataset(v). As all function names, I would keep the function unexported to avoid ambiguities.

In some case, the could lead to circular definition (the type definition of CommonDataModel.AbstractVariable depends on CommonDataModel.AbstractDataset and CommonDataModel.AbstractDataset depends on CommonDataModel.AbstractVariable), but they can be avoided using parametric types, e.g.

CommonDataModel.jl/test/memory_dataset.jl

Line 7 in 01a7ff0

parent_dataset::TP

julia 1.6.7: union type issue on Windows

This currently fails on windows with julia < 1.8

using NCDatasets
filename = tempname()
NCDataset(filename,"c") do ds
    # test Union{Missing,T}
    data = [missing,1.,2.]
    defVar(ds,"foo",data,("dim",), fillvalue = -9999.)
end

See JuliaLang/julia#53136

Other platforms and windows with julia >= 1.8 are not affected.

Binary operations are not internal

If I have a dataset ds, then the result of ds + ds is a matrix, not a dataset.
I have followed the example of the groupby function, that calculates anomalies, and we end up with a matrix, not a dataset. This prevents the use of the result again in the gropuby function if we want to calculate the standard deviations in order to get the standardized anomalies. The groupby function needs a dataset and does not work with a matrix.

CRS

In both GRIBB and NetCDF there is a need to extract CRS data. e.g:

https://discourse.julialang.org/t/extracting-projection-from-grib-files/109818/3

It would be good if CommonDataModel.jl handled this, probably adding methods to GeoInterface.crs as function to retrieve it.

I am aware of the difficulty (and in edge-cases actual impossibility) of converting CF standard crs data to Well Known Text or Proj. But we should try. Currently the workaround is to load and call ArchGDAL.jl or GMT.jl to get the crs manually, but an additional large binary dependency for this small task is less than ideal.

I made an attempt at starting this a few years ago:
https://github.com/rafaqz/Rasters.jl/blob/netcdf_crs/src/sources/ncdatasets.jl#L407-L508

But really something like that should live here or in a package similar to CFTime.jl - like CFCRS.jl... that CommonDataModel.jl could also depend on.

This is a good guideline for an implementation:
https://trac.osgeo.org/gdal/wiki/NetCDF_ProjectionTestingStatus

(notice even GDAL has an incomplete implementation - but it should take care of 99.9% of the use cases)

One stop-gap solution is to add an extension for ArchGDAL here and just call it on the files when GeoInterface.crs is called on any AbstractVariable.

DiskArrays.jl compatability

It would be good if DiskArrays.jl chunking, broadcasting, and iteration capabilities could propagate through any AbstractVarable here.

They don't have to be AbstractDiskArray because there are macros in DiskArrays.jl that can be applied to the types.

Broadcast over `CFVariable` is very slow

In Rasters we hit a bug where I had forgotten to re-wrap CFVariable in our internal CFDiskArray made specifically for fixing this CommonDataModel bug.

This one small change made writing fiies 100x faster
https://github.com/rafaqz/Rasters.jl/pull/633/files

Broadcasting to/from a CFVariable means reading/writing for every single pixel

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.