Roadmap for release about traitlets HOT 34 CLOSED

Ping @jasongrout @ellisonbg

from traitlets.

We still depend on ipython_genutils. Is it ok to make a release and keep this?

100%. It's a mild annoyance that the package exists, but it's not important to drop its use any time soon.

Should we make a beta? Should it be 5.0.0 or 4.1.0 ?

The two primary points on versioning for me:

• It's absolutely necessary that we break zero existing APIs (All of IPython and Jupyter 4.0 must continue to work without modification). This will continue to be a requirement for at least a year or two.
• This deprecates roughly 100% of the existing traitlets API.

Given the strict backward-compatibility requirements, we shouldn't have to make 5.0, but the near-complete API change might be better represented with 5.0.

from traitlets.

@minrk can you clarify why future major versions of traitlets can't break
APIs for that long of a period. Presumably IPython/Jupyter 4.0 should be
pinned to 4.x series of traitlets so they will keep working no matter what
happens in 5.0 traitlets.

However, if future versions of any other package (ipywidgets,
bqplot, matplotlib) are going to depend on both traitlets and the notebook
(I think they have to) then ipywidgets can't depend on an API breaking
traitlets until a future version of the notebook does as well. We can't
have people using 4.0 of the notebook with a 5.0 of
ipywidgets/traitlets that are API breaking. We do have to be very careful
about this type of dependency conflict.

Is this what you were thinking about?

But I don't see why ipywidgets and notebook future versions couldn't start
to both depend on an API breaking traitlets before 2 years.

Cheers,

Brian

On Sat, Oct 10, 2015 at 5:46 AM, Min RK [email protected] wrote:

We still depend on ipython_genutils. Is it ok to make a release and keep
this?

100%. It's a mild annoyance that the package exists, but it's not
important to drop its use any time soon.

Should we make a beta? Should it be 5.0.0 or 4.1.0 ?

The two primary points on versioning for me:

• It's absolutely necessary that we break zero existing APIs (All of
  IPython and Jupyter 4.0 must continue to work without modification). This
  will continue to be a requirement for at least a year or two.
• This deprecates roughly 100% of the existing traitlets API.

Given the strict backward-compatibility requirements, we shouldn't have
to make 5.0, but the near-complete API change might be better represented
with 5.0.

—
Reply to this email directly or view it on GitHub
#104 (comment).

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

from traitlets.

Presumably IPython/Jupyter 4.0 should be pinned to 4.x series of traitlets so they will keep working no matter what happens in 5.0 traitlets.

I disagree with this. We don't pin any other dependencies just assuming they are going to break everything on a major release, and in general that's not common practice in Python due to the flat package namespace. Plus, the whole point of the split is that there are stable APIs between our various components, and they should be able to make major releases independently of each other. Since Python doesn't do javascript-style package installation, stable means over time, not just version numbers. If we are doing lock-step 4->5 of everything, there wasn't much of a point to separating the repos.

We can't have people using 4.0 of the notebook with a 5.0 of ipywidgets/traitlets that are API breaking...Is this what you were thinking about?
But I don't see why ipywidgets and notebook future versions couldn't start to both depend on an API breaking traitlets before 2 years.

One package depending on a breaking change means forcing a release of a dozen packages before it can be used - we use traitlets everywhere. A breaking change in traitlets means forcing a new release of every package we maintain. If, however, there's a new API that doesn't break the old one, we can start using it as soon as it's available without issue, which is the current state of this repo. The new API has been added next to the old one, and we can keep it there to allow a smooth transition. The only reason now to remove the old API is cleanliness, which isn't good enough anymore. We shouldn't break old APIs unless a strong case can be made for the direct benefit of its removal, even with new major releases. This is something we have historically been very bad at, and traitlets being the bottom of the stack makes it the single most important place to be careful. Two years might be an exaggeration, but I don't think one is.

My main point: due to the repo split, our deprecations cycles need to get a lot longer.

from traitlets.

I disagree with this. We don't pin any other dependencies just assuming
they are going to break everything on a major release, and in general
that's not common practice in Python due to the flat package namespace.
Plus, the whole point of the split is that there are stable APIs
between our various components, and they should be able to make major
releases independently of each other. Since Python doesn't do
javascript-style package installation, stable means over time, not just
version numbers. If we are doing lock-step 4->5 of everything, there wasn't
much of a point to separating the repos.

Good point. For package that sit at the bottom/mid of our dependencies,
this makes sense. But the reason we did the repo split was the enable
higher level components (ipywidgets) to move quickly and break things.
Traitlets is challenging because it is at the very bottom, but also used
extensively by ipywidgets in advanced ways that require its evolution (not
breakage necessarily though).

We can't have people using 4.0 of the notebook with a 5.0 of
ipywidgets/traitlets that are API breaking...Is this what you were thinking
about?
But I don't see why ipywidgets and notebook future versions couldn't start
to both depend on an API breaking traitlets before 2 years.

One package depending on a breaking change means forcing a release of a
dozen packages before it can be used - we use traitlets everywhere. A
breaking change in traitlets means forcing a new release of every package
we maintain. If, however, there's a new API that doesn't break the old one,
we can start using it as soon as it's available without issue, which is the
current state of this repo. The new API has been added next to the old one,
and we can keep it there to allow a smooth transition. The only reason now
to remove the old API is cleanliness, which isn't good enough anymore.
We shouldn't break old APIs unless a strong case can be made for the direct
benefit of its removal, even with new major releases. This is something we
have historically been very bad at, and traitlets being the bottom of the
stack makes it the single most important place to be careful. Two years
might be an exaggeration, but I don't think one is.

I full agree that in this case we can make traitlets fully BW compat and we
should. Even though we don't want to be forced to do lock-step releases of
all packages, that doesn't mean we can't, when needed. We are not there
yet, but commiting to 2 years of no API breaks feels like an overreaching
and unrealistic one given the fast pace of the project. If we find that N
months from now everything is being held up by our inability to move APIs
forward, then we could do another synchronized release of a larger subset
of project.

My main point: due to the repo split, our deprecations cycles need to get
a lot longer.

For foundational package that rely on each other, such as traitlets,
message spec, nbformat etc. I agree. I think I am mostly reacting possibly
over-generalized idea of "no API breakage for 2 years". We still have a lot
of things to break in parts of our code base...

Cheers,

Brian

—
Reply to this email directly or view it on GitHub
#104 (comment).

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

from traitlets.

But the reason we did the repo split was the enable
higher level components (ipywidgets) to move quickly and break things.

Yes, I don't mean that this applies to everything, but traitlets is probably where we should be the most conservative about breakage. Part of the reasoning of the split is that we have components at widely varying levels of stability—some very stable, some very unstable. Widgets are highly unstable and experimental, and I view as free to break all the things, while traitlets was quite stable. Most of the other packages lie somewhere in-between.

Traitlets is challenging because it is at the very bottom, but also used
extensively by ipywidgets in advanced ways that require its evolution (not
breakage necessarily though).

Yup, traitlets is a tricky one because of that. I'm AOK with new APIs, and as the work here has demonstrated so far, we can make new APIs without unnecessarily breaking the old ones. They can be deprecated and stay deprecated for quite some time. Deprecating doesn't need to put a ticking clock on removal, either—we don't have to remove deprecated APIs until maintaining them becomes a problem.

Even though we don't want to be forced to do lock-step releases of all packages, that doesn't mean we can't, when needed.

Quite true, but hopefully it will never (or rarely) be needed. I doubt we will ever be in a position to need a release of traitlets that would break the stable version of any other project, for instance. I would expect sensible deprecation cycles to allow releases of other packages to have been made naturally in the meantime.

I think I am mostly reacting possibly over-generalized idea of "no API breakage for 2 years". We still have a lot of things to break in parts of our code base...

I didn't mean to imply that I was trying to dictate a strict rule, especially one that would apply to every one of our repos the same, sorry about that. I was just trying to convey the idea that we should be conservative about breaking widely used low-level APIs on our most stable, core packages. In general, I think we should do our best to do more of exactly what's been done here: make new APIs adjacent to old ones and deprecate more gently, with the deprecation time (time matters much more than version numbers for deprecations) taking into account how much code needs to be updated to use the new APIs. I think at least a year for _foo_changed to keep working is a reasonable ballpark estimate for something like that. I didn't mean for the statement to be stronger than that, though.

from traitlets.

All sounds good, thanks for clarifying. It is nice that we are evolving
traitlets in a BW compat manner ;-)

On Tue, Oct 13, 2015 at 3:27 AM, Min RK [email protected] wrote:

But the reason we did the repo split was the enable
higher level components (ipywidgets) to move quickly and break things.

Yes, I don't mean that this applies to everything, but traitlets is
probably where we should be the most conservative about breakage. Part
of the reasoning of the split is that we have components at widely varying
levels of stability—some very stable, some very unstable. Widgets are
highly unstable and experimental, and I view as free to break all the
things, while traitlets was quite stable. Most of the other packages
lie somewhere in-between.

Traitlets is challenging because it is at the very bottom, but also used
extensively by ipywidgets in advanced ways that require its evolution (not
breakage necessarily though).

Yup, traitlets is a tricky one because of that. I'm AOK with new APIs, and
as the work here has demonstrated so far, we can make new APIs without
unnecessarily breaking the old ones. They can be deprecated and stay
deprecated for quite some time. Deprecating doesn't need to put a ticking
clock on removal, either—we don't have to remove deprecated APIs until
maintaining them becomes a problem.

Even though we don't want to be forced to do lock-step releases of all
packages, that doesn't mean we can't, when needed.

Quite true, but hopefully it will never (or rarely) be needed. I doubt we
will ever be in a position to need a release of traitlets that would break
the stable version of any other project, for instance. I would expect
sensible deprecation cycles to allow releases of other packages to have
been made naturally in the meantime.

I think I am mostly reacting possibly over-generalized idea of "no API
breakage for 2 years". We still have a lot of things to break in parts of
our code base...

I didn't mean to imply that I was trying to dictate a strict rule,
especially one that would apply to every one of our repos the same, sorry
about that. I was just trying to convey the idea that we should be
conservative about breaking widely used low-level APIs on our most stable,
core packages. In general, I think we should do our best to do more of
exactly what's been done here: make new APIs adjacent to old ones and
deprecate more gently, with the deprecation time (time matters much more
than version numbers for deprecations) taking into account how much code
needs to be updated to use the new APIs. I think at least a year for
_foo_changed to keep working is a reasonable ballpark estimate for
something like that. I didn't mean for the statement to be stronger than
that, though.

—
Reply to this email directly or view it on GitHub
#104 (comment).

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

from traitlets.

I completely agree with all @minrk said, and making a beta is probably a good idea then.

Regarding the length of the deprecation period, I wanted to add that this could be variable depending on the API in question.

• clearly, things like on_trait_change, and the magic methods should be supported for a while.
• however, other jupyter repos are currently making direct calls to things that should probably be considered private API (like _notify_trait), and when these are fixed, I imagine that we could remove the compatibility layer that we are currently keeping for those, much earlier than the on_trait_change method.

from traitlets.

@SylvainCorlay absolutely, well put. Related to that: let's try to keep track of where we are using private APIs across our repos, and see if we can get that close to zero.

from traitlets.

Is there any plans to support GUI like enthought-traits?

from traitlets.

Probably not in this package, but there may be a desire for a downstream project like traits-ui. Something like that can certainly be created as a totally separate project that depends on traitlets.

from traitlets.

@SylvainCorlay, I worked through a new solution for @default in #108 that's much less intrusive.

from traitlets.

@minrk Thank you for the explanation. Is there any plans to support container notification? I noticed that:
We don't support:

• A full set of trait types. Most importantly, we don't provide container
  traits (list, dict, tuple) that can trigger notifications if their
  contents change.

from traitlets.

@fyrestone yes there are plans for observable containers. Some of the changes that will be released in this version will allow custom trait types to fire custom events on element changes.

from traitlets.

@SylvainCorlay Great! Thank you.

from traitlets.

@SylvainCorlay, Brian mentioned that you'd talked about the @default decorator and that you were interested in basically using it to reassign the associated function under a different name (i.e. _foo_default) on the class. I'm a little skeptical though, because I would imagine that manipulating the class like this will cause problems when HasDescriptors isn't the base class.

from traitlets.

@minrk What do you think is missing for 4.1 at the moment?

from traitlets.

Slight amendment to the description: the type argument in observe is change rather than trait_change

from traitlets.

@rmorshea good catch - corrected.

from traitlets.

With respect to that topic though, what exactly would be the difference between change and All? I guess the way I see it, the two are sort of synonymous. For instance, when thinking about something like notifying for element changes in a list, an element_change type would be "subchange" which may or may not follow from the fact that the old_value != new_value. In other words, in order for there to have been a "subchange" there would have to have been a "change".

I started to consider this approach to notifier types in #111 where trait types would be responsible for generating subchanges (and passing them to the HasTraits instance) iff there was a change where old_value != new_value.

In short, what kind of notifications would All refer to that would not be considered "subchanges"?

from traitlets.

@SylvainCorlay I think we need some more documentation and examples of the new API, and perhaps a migration doc.

I've also been using master for a bit, and have been meaning to investigate our deprecation messages. At the moment, they are very uninformative, and if we can do a better job indicating what needs to be updated, we should.

from traitlets.

@rmorshea what I meant by change does not cover element change. It is important that we have this type of event - also for backward compatibility.

@minrk would you make the migration document part of the sphinx doc or a wiki?

from traitlets.

In the documentation, we have a link to milestones tags on GitHub PRs:
http://traitlets.readthedocs.org/en/latest/changelog.html#id1

We should probably tag the merged PR appropriately.

from traitlets.

The merged PRs are tagged. There were only a couple from the last week or two that hadn't been.

from traitlets.

What do you guys think is blocking for the release?

from traitlets.

I think I could rework #111 into the new release, but that's not really a stopping point. It could just as well be put in later.

from traitlets.

It seems there are no blockers at this point? Then it's just a matter of: release notes and a release manager? @SylvainCorlay, are you typically the RM for traitlets? Goes to show how much I'm in the loop :)

from traitlets.

@fperez I am not the release manager. I think that @minrk is the one doing the releases.

from traitlets.

Got it, thanks. In that case, I'll leave it to @minrk to decide how to balance the timing between this release and the much more demanding work on the notebook 4.1 one. But I'm glad to hear this is nearly ready to go.

from traitlets.

I'm happy trusting @SylvainCorlay and @rmorshea, which is what I've mostly been doing. The docs and such are in good shape, I think. @SylvainCorlay do you think RC or final release? Any more testing you would like before shipping?

I've done some runs of the existing codes with traitlets master, and the only thing that I've been unhappy with is the deprecation messages—they don't give any info about what code is triggering the deprecation, so it's tedious to figure out what needs updating. I'll take a pass at those today.

from traitlets.

#149 catches traitlets.config up to use the new API (it would be a bit embarrassing to deprecate an API that traitlets itself still uses)

#151 makes the deprecation warnings point to the deprecated code, rather than a random point in traitlets itself.

With those two, we might be good to go.

One aspect of the changes that I'm squeamish about is changing the signature of

def _trait_changed(...):
    super()._trait_changed(...)
    other stuff

Which makes subclassing sensitive to these changes. We have to be careful about any classes where we are updating to use the new APIs and think about whether subclasses can reasonably override the signal methods. If they are, we have to update to the new APIs in backward-compatible ways, even on the event listener methods.

from traitlets.

Since the issue reported by @minrk earlier is now solved, should we release or make a second beta?

from traitlets.

Yup, let's do another beta, and hopefully release by the end of the week. I'll try to do some more testing of the transition to make sure there's a backward-compatible path forward for the packages.

from traitlets.

4.1 is out, yay!

from traitlets.