Implementing a more robust issue/PR tagging system about jupyter HOT 18 CLOSED

Hey there, I'm going through old issues and it seems to me that it makes sense to close this one just from the perspective that various subprojects have settled into their own conventions for tagging, and though this discussion hasn't had any activity for 8 years, anyone can refer back to it if they have new thoughts and ideas about how to proceed with tagging.

Thanks everyone and happy hacking!

from jupyter.

The naming of our tags to do that is maybe not good enough. For a few years we've been using "Sprint-Friendly" (https://github.com/search?q=org%3Ajupyter+label%3Asprint-friendly). We use to have also "quickfix" that we merged with sprint-friendly.

We'll try to do a better job at tagging.

from jupyter.

I think this is an important issue, but it's usually really hard to know ahead of time how difficult an issue is to address, so I'm not sure how feasible the latter labels are to do in practice. I'm sure we can strike a better balance than we do currently, though.

from jupyter.

Safia,

As I've been working on improving the contribution documentation, I realize

that we can benefit greatly from implementing a tagging system on our
issues and our PRs. I imagine that on on each issue or PR we would have the
following tags.

1. A tag representing whether the issue related to a front-end
   (Javascript/CSS), backend (Python), or documentation problem.

With our repos quickly being split into separate ones for Python and
JavaScript, I think this is less important. But for repos that have both, I
am +1 on it.

2. A tag representing the difficulty level of submitting a PR on that
   issue. We can determine difficulty using some of these metrics.
   • The amount of Jupyter related knowledge needed to submit a PR on
     this
   • The amount of Javascript/Python/CSS knowledge needed to submit a
     PR on this
   • Other metrics probably
3. A tag representing the effort level to submit a PR for particular
   issue. I imagine the following divisions being issued on this.
   • easy — issues that can be knocked down in a couple of hours of
     work by a semi-experienced developer, very similar to the sprint-friendly
     tag that we use now
   • medium — issues that require a bit more time than the easy level
     issues and are for beginners or semi-frequent contributors who want a
     challenge
   • hard — issues that are probably best addressed by someone who has
     a lot of experience with the code base

In line with this, I think we should avoid using the sprint-friendly tag
to represent issues that are easy to tackle. If I am a new contributor, I
don't really understand what sprint-friendly means in the context of
difficulty but a tag like difficulty: easy is much easier to understand.

I love these ideas. I have never liked the sprint-friendly tag. My only
concern is that until we have a project manager to help with issue triage,
it will be difficult for us to keep up with tagging everything. My only
question is what to call the category 2 tags?

There is room for more tags to be included but I think the three below
cover the basics.

There are some pros and cons to implementing this tagging system as
outlined below.
Cons

• More time has to be dedicated to organizing and tagging issues from
  the owners of each repo
• It would be difficult to maintain consistency in the way that we tag
  difficulty an effort level as it depends on the person tagging

Pros

• It shows new contributors (and existing ones) that we care about
  making the issues approachable for them
• It will help us figure out how we are allocating our manpower (for
  example, is Safia taking too many of the easy issues when she can tackle
  some more difficult ones? Should Steve focus on front-end issues while Bob
  focuses on back-end?) so that we can work through things more efficiently

And I'm sure there are other pros and cons but I can't think of them at
the moment. I'm welcome to hearing them though!

cc/ @minrk https://github.com/minrk @Carreau
https://github.com/Carreau @ellisonbg https://github.com/ellisonbg
@fperez https://github.com/fperez or anyone else who should be included
in the discussion

—
Reply to this email directly or view it on GitHub
#74.

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

from jupyter.

@captainsafia I agree with @minrk that it is very difficult to accurately gauge difficulty in practice. One thing that I have seen work well on the Requests project is to tag issues as Contributor Friendly (i.e. well defined issues suitable for sprints or newcomers) and Willing to Pair or Pair with Safia tags to issues that a developer would be willing to assist someone that has some dev experience to take on a larger task.

from jupyter.

Thanks for the experienced perspective.

With our repos quickly being split into separate ones for Python and JavaScript, I think this is less important. But for repos that have both, I am +1 on it.

Good point! I don't think we would have a similar tag on a repository level, as in I don't imagine needing to tag issues within a repository with which part of the code base it relates to.

My only concern is that until we have a project manager to help with issue triage, it will be difficult for us to keep up with tagging everything.

This was a concern that I had as well. Keeping things organized is always hard especially if you don't have one person/a group of people who are dedicated to doing it.

My only question is what to call the category 2 tags?

I was thinking "difficulty level."

I have seen work well on the Requests project is to tag issues as Contributor Friendly (i.e. well defined issues suitable for sprints or newcomers) and Willing to Pair or Pair with Safia tags to issues that a developer would be willing to assist someone that has some dev experience to take on a larger task.

This might be a good idea as well, to have an almost binary difficulty tagging scheme. Either we know for sure something is friendly for contributors or we know that we can work with contributors (or just do it internally). What do other folks things?

from jupyter.

FYI,

Here i how rust organise labels:

https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#issue-triage https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#issue-triage

• Yellow, A-prefixed labels state which area of the project an issue relates to.
• Magenta, B-prefixed labels identify bugs which belong elsewhere.
• Green, E-prefixed labels explain the level of experience necessary to fix the issue.
• Red, I-prefixed labels indicate the importance of the issue. The I-nominated label indicates that an issue has been nominated for prioritizing at the next triage meeting.
• Orange, P-prefixed labels indicate a bug's priority. These labels are only assigned during triage meetings, and replace the I-nominated label.
• Blue, T-prefixed bugs denote which team the issue belongs to.
• Dark blue, beta- labels track changes which need to be backported into the beta branches.

M

On Dec 2, 2015, at 04:14, Safia Abdalla [email protected] wrote:

Thanks for the experienced perspective.

With our repos quickly being split into separate ones for Python and JavaScript, I think this is less important. But for repos that have both, I am +1 on it.

Good point! I don't think we would have a similar tag on a repository level, as in I don't imagine needing to tag issues within a repository with which part of the code base it relates to.

My only concern is that until we have a project manager to help with issue triage, it will be difficult for us to keep up with tagging everything.

This was a concern that I had as well. Keeping things organized is always hard especially if you don't have one person/a group of people who are dedicated to doing it.

My only question is what to call the category 2 tags?

I was thinking "difficulty level."

I have seen work well on the Requests project is to tag issues as Contributor Friendly (i.e. well defined issues suitable for sprints or newcomers) and Willing to Pair or Pair with Safia tags to issues that a developer would be willing to assist someone that has some dev experience to take on a larger task.

This might be a good idea as well, to have an almost binary difficulty tagging scheme. Either we know for sure something is friendly for contributors or we know that we can work with contributors (or just do it internally). What do other folks things?

—
Reply to this email directly or view it on GitHub #74 (comment).

from jupyter.

Another thing that I have noticed in the past is that in practice it is very difficult to separate out the "difficulty" from "amount of time". IOW, these two rankings tend to be highly correlated, so it isn't obvious we need both.

Also, I think elements of the rust approach are nice but beyond our organizational capacity currently. We could explore something like that after we hire the project manager.

from jupyter.

@ellisonbg that's a good point. I think there is also a distinction with respect to how much code is expected. While there are some that may be 'hard', the difficulty is actually finding where the one-line fix goes or how to spell the regex, and there are others that are "write a bunch of new API modules" or "refactor completion" that may not be intellectually challenging, but are a lot of code to write. I have no idea how to communicate that, nor how well we can actually estimate it in most cases.

from jupyter.

In the past, quite a few of the things tagged 'sprint friendly' and 'quickfix' haven't really been either. 'quickfix' in particular was ambiguous, because there are many things that can be fixed quickly if you understand the issue and know where the relevant code is, but are not at all easy for a newcomer to the codebase to deal with.

I think it's worth having some kind of tag for 'easy' issues that we can point newcomers to as an on-ramp to the project. But given the difficulties of classification, I'm not sure it's worth making any more tags trying to indicate difficulty or time required.

from jupyter.

Good points from @takluyver here. I think that is part of the reason I
haven't felt like our quickfix and sprint friendly tags haven't worked
well. Things that are actually easy/trivial tend to get fixed quickly by
us.

On Mon, Dec 7, 2015 at 1:05 PM, Thomas Kluyver [email protected]
wrote:

In the past, quite a few of the things tagged 'sprint friendly' and
'quickfix' haven't really been either. 'quickfix' in particular was
ambiguous, because there are many things that can be fixed quickly if you
understand the issue and know where the relevant code is, but are not at
all easy for a newcomer to the codebase to deal with.

I think it's worth having some kind of tag for 'easy' issues that we can
point newcomers to as an on-ramp to the project. But given the difficulties
of classification, I'm not sure it's worth making any more tags trying to
indicate difficulty or time required.

—
Reply to this email directly or view it on GitHub
#74 (comment).

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

from jupyter.

On Dec 8, 2015, at 18:23, Brian E. Granger [email protected] wrote:

Good points from @takluyver here. I think that is part of the reason I
haven't felt like our quickfix and sprint friendly tags haven't worked
well. Things that are actually easy/trivial tend to get fixed quickly by
us.

You mean by Min ? Right ?

I try to force myself not to fix the easy one that can wait, and sometime as
Min not to fix the issues I open.

M

from jupyter.

I think it is worth tagging "sprint friendly". Whether a new contributor, sprinter, or team member completes the issue, tagging does give an indication of issues that are well defined and scoped.

If we wish to onboard new contributors of all experience levels, it would be helpful to have the "sprint friendly" option. All are free to claim and complete them (I'm fairly confident that there will always be new issues ☀️ ). By having the issues tagged, experienced team members will have some time freed up for less defined issues or deep troubleshooting.

from jupyter.

We do all definitely fix easy things at times. It's tricky, because we don't see all that many casual contributors looking for bugs to fix, so if a problem is bothering someone, it feels lousy to leave it unfixed to provide a task for someone who might show up weeks later, when we can fix it in five minutes.

So the problems that get left are those which don't much bother anyone, and those aren't terribly exciting to work on.

from jupyter.

@takluyver I'm including documentation in my definition of bugs and issues. Many of these doc issues will be sprint friendly and a reasonable onboarding entry point.

I'm all for keeping the fix process fluid and moving forward. I'm not advocating anyone change their personal workflow which clearly works. 😄

from jupyter.

I'm sceptical of the notion that docs work makes a good entry point to a project. We certainly need input from people less familiar with the project about where the docs are confusing or incomplete, and there are some people who will be good at writing docs, but I think it takes a good understanding of a project to write good docs. I suspect that the idea of getting newcomers to help with docs is a daydream of programmers who don't want to write docs themselves ;-)

from jupyter.

there are some people who will be good at writing docs, but I think it takes a good understanding of a project to write good docs

Yes, but I've mentored several events where I was helping first-time contributors get started on a project and the easiest way to get started is to augment the documentation. Indeed, most of my PRs on new projects are related to documentation, specifically around issues I experience around using the library or setting up the dev environment.

We do all definitely fix easy things at times. It's tricky, because we don't see all that many casual contributors looking for bugs to fix, so if a problem is bothering someone, it feels lousy to leave it unfixed to provide a task for someone who might show up weeks later, when we can fix it in five minutes.

I agree with this 100%. It definitely gets into chicken and egg territory though, do we not have any casual contributors due to the lack of entry-level sprint-friendly tags or do we not have many entry-level issues because there are no casual contributors?

Things that are actually easy/trivial tend to get fixed quickly by us.

Yep! I noticed this pretty quickly when I was getting involved as well. I'm always on the prowl for easy issues to fix in repositories I haven't worked with before and they are usually solved pretty quickly by someone on the core team. I, personally, feel that it makes more sense to have some things open for beginners so that knowledge of the codebase is propagated across multiple individuals.

Another thought on the lack of new contributors, as we start to host events like JupyterDay and JupyterCon, we can anticipate that slowly and surely we will have more contributors (hopefully). We definitely should put effort into organizing our issues, if not for other developers, than for ourselves. It does have a high start-up cost, but I believe that in the long run it'll pay off.

I also think that if we do want to get more people involved and helping out, we can start by sending off a weekly digest of the current sprint-friendly issues across the ecosystem and the point of contact for those issues that first-time contributors can reach out to personally for assistance.

I also think it would make sense to bring this conversation to the mailing list as well. For the most part, everyone in this this has extensive experience with the project or open-source so we do have a bias.

That was a pretty big brain dump to catch up on the conversation, but I think I covered mostly everything.

from jupyter.

Safia just pointed me here. I would welcome a "easy" or "contributor needed" tag. Pretty much -0 on all other tags.

If you want people to start contributing you need to put out some starting points, and then mentor them. The main purpose of the easy issues is to get people hooked on how fun it is to contribute. Mostly I think it is helping people to get over being nervous about screwing up in public.

Some fraction of newcomers will stay for good. It means that the core-devs end up writing less code and doing more reviewing/mentoring/community building. It can work extremely well but costs some effort.

I got my start in open-source with scikit-learn and because it was a success I think they do a pretty good job (confirmation bias anyone? :)). Andy (and the other devs) spends a large amount of time creating easy issues and then mentoring people when they pick them up. Pretty sure he could fix them more quickly than he can create the issues. If you are there for questions it is also not a problem if the "easy" issue turns out not to be easy. If the PR gets abandoned you can easily pick it up and finish it (in scikit-learn some other new comer might come along and pick it up)

I don't think that the easy issues are just for people learning to code, but also for people like me who are fairly confident but don't know the "style" (both in a technical and personality sense) of the project. Learning your way around a new code base takes commitment and effort. Sweeten the deal by giving people a sense of achievement (a merged PR, even if it is an easy one). I'd love to contribute more to jupyterhub, but right now I only send tiny patches when something is broken for me.

I'll go out on a limb and claim that most (easy) issues don't need urgent (a few days) repair. So it is worth leaving them out there as a "trap". Once you get a reputation for having issues for newcomers you end up with the state of scikit-learn were it is virtually impossible to find an easy issue that doesn't have a PR on the go already.

from jupyter.