Refactor type hierarchy about conformalprediction.jl HOT 6 CLOSED

Yes, this sound good.

To clarify, MLJModelInterface.jl remains the home for MLJ's basement-level model interface. I just mention LearnAPI.jl because that is where I hope to move to later.

So we should add two new abstract types to MLJModelInterface: Set <: Supervised and ProbabilisticSet <: Supervised. We already have Interval <: Supervised. Would you be willing to make the PR? The additions go here and here.

from conformalprediction.jl.

Addressed in #21

@ablaom is this what you had in mind? Instead of ConformalSet <: MMI.Supervised we'd ideally have ConformalSet <: MMI.Set in the future.

from conformalprediction.jl.

Yeah, mostly:

• At present Interval is a subtype of Supervised.
• I don't see a need for a separate ConformalInterval <: Interval subtype in MLJModelInterface. If it were useful to you, that could live in ConformalPrediction.jl, no?
• I would make make Set and ProbababilisticSet subtypes of Supervised.

In other words, we currently have

Probabilistic <: Supervised
Deterministic <: Supervised
Interval <: Supervised

and I would propose extending this with

Set <: Supervised # if you think you even need this
ProbabilisticSet <: Supervised

(In the new LearnAPI.jl, we would add two target proxy types, Set and ProbabilisticSet. These are possible values of a trait target_proxy and there would be no need to restrict to supervised models, but that's another story...)

from conformalprediction.jl.

Would that work for you?

from conformalprediction.jl.

Hi Anthony! Yes, absolutely that works. In fact, that's what I had in mind and also reflect the way things are currently implemented (I think the diagram needs some clarification perhaps).

Also, good point ProbabilisticSet <: Supervised, have added a placeholder for this now in #26. To be clear, at this point we have the following:

"An abstract base type for conformal models that produce interval-valued predictions. This includes most conformal regression models."
abstract type ConformalInterval <: MMI.Interval end

"An abstract base type for conformal models that produce set-valued deterministic predictions. This includes most conformal classification models."
abstract type ConformalSet <: MMI.Supervised end                    # ideally we'd have MMI.Set

"An abstract base type for conformal models that produce set-valued probabilistic predictions. This includes most conformal classification models."
abstract type ConformalProbabilisticSet <: MMI.Supervised end       # ideally we'd have MMI.ProbabilisticSet

"An abstract base type for conformal models that produce probabilistic predictions. This includes some conformal classifier like Venn-ABERS."
abstract type ConformalProbabilistic <: MMI.Probabilistic end

For set-valued predictions, I am at this point still just subtyping <: MMI.Supervised, since the specific types still need to be added to LearnAPI.jl. In particular, for conformal models of type ConformalSet currently produce prediction of type UnivariateFinite (linking our related discussion on discourse).

I will close this issue now and open a new one related to set-valued predictions specifically.

Thanks!

from conformalprediction.jl.

Thanks @ablaom 👍🏽

Yes, I'll make the PR as soon as I can (have a few commitments these next two weeks, but will get to it eventually 😅)

from conformalprediction.jl.