Spec process proposal about specs HOT 18 CLOSED

Some thoughts prompted by:

i'm going to implement something new that just landed in the spec and it's not going to work exactly like the spec says, we have to make some changes. do i stop my work and make a proposal? do i do the implementation and then make the proposal or draft? ideally for something like this i would do the implementation and then propose the deltas directly to the spec, kind of more like an update than a new proposal.

In my head solving this type of problem seems out of scope for the spec process itself to handle. It feels so much cleaner if all protocol level changes come about by the same dumb interface: make a proposal, work to move it to a draft, merge to the spec. As an observer I don't need to worry about how the protocol is being worked on to anticipate what is going to happen to the spec, I can simply rely on all new ideas passing through the pipeline. Adding additional spec-modifying channels to cater to unblocking work will lead to a spec process tightly tied to our initial working patterns. My guess is this leads to a brittle spec process and an increase in the chance that some subtle and critical mistake makes its way into the spec.

The universal dumb interface approach seems feasible to me because I don't think the spec process will end up being the best and only interface for us to synchronize the efforts of protocol and implementation developers, at least not until the project has matured and slowed down a lot. We don't need a spec process that handles issues of this fine level of granularity if we have some "layer 2" process, i.e. something like that used by @porcuquine in proofs land, to sync research and dev in other areas of the protocol like consensus or repair. We can work together to write proposals and code fairly concurrently (i.e. wait to write a draft until some code seems to work) and minimize the number of patching proposals submitted to the spec.

Without such "layer 2" higher context working relationships, i.e. once OSS contributors propose a higher percentage of spec changes, the dumb "just make a new proposal" interface might get annoyingly laggy. But maybe that comes with the territory for a stable project?

These are just initial thoughts, interested to see if anyone disagrees strongly.

from specs.

• Do we have agreement on this? if so, can we update the proposal with changes?
• Can we set a deadline for which this comes in place?
• Do we need an aligning call

Personal request:

• let's try to be succinct when writing these issues, it's really time consuming going through a lot of text (if you need long text, pls have bulletpoint summary, use summarizing phrases at the beginning of long paragraphs)

from specs.

@decentralion: great work! Maybe we can move this into "instructions.md" or something like that in the repo.

We should also make a distiction between specs and technical documentation. Where that distinction is that the technical documentation will contain all the context and additional exploration of other ideas as well as link any references to the defining Filecoin component spec.

@whyrusleeping: what do you think?

from specs.

Thanks for writing this up!

@bvohaska:

Maybe we can move this into "instructions.md" or something like that
in the repo.

My understanding is that this issue is intended to be an instance of the
very process that it describes. So, once it progresses from “proposal”
to “draft”, it can be merged into drafts/113-metaprocess.md or
similar. Then, a root README or instructions.md could link to that
file.

We should also make a distiction between specs and technical
documentation. Where that distinction is that the technical
documentation will contain all the context and additional exploration
of other ideas as well as link any references to the defining Filecoin
component spec.

Agreed in the long term; in the short term, I’d propose keeping these in
one artifact for now. Writing, expanding, and maintaining the spec alone
will be a fair amount of effort. Adding separate tech docs would
additionally require keeping those in sync with the spec and the
go-filecoin implementation—three potential stores of information, not
just two. I worry that this wouldn’t carry its weight in the short term.
Note also that @phritz has expressed the importance of communicating
design intent in the spec itself for implementers, so the distinction
won’t be as black-and-white as “math here, prose there”.

@decentralion:

Proposals do not need to be at a very high level of detail, and should
be written early in the process towards changing the spec.

I think that I agree, but let’s flesh this out a bit more. Here are some
concrete variations on the same theme:

• If I discover a problem in the spec, but I’m not sure of the
  solution, should I post a “proposal”? (In #110 @whyrusleeping
  suggested that this should at least be an issue on the repository,
  as opposed to, say, a Discourse post.)

• If a group of people are working over an extended period of time on
  developing a solution to a well-defined problem, should these
  conversations take place in a “proposal”?

• Is it the case that every proposal MUST specify the design intent,
  SHOULD specify the requirements, and MAY specify a proposed
  solution? (in the sense of RFC 2119)

• Should this very discussion take place in the proposal issue (as it
  currently is)?

As I understand it, the answer to each of these questions is “yes”.

Drafts should merge easily (i.e. consensus from the team is NOT
required to merge the PR).

To confirm: what’s implicit here is that the drafts/*.md file is a
“living document” that is intended to be discussed in and updated by
pull requests, until it reaches a state where maintainers agree that it
should be merged. But although the draft may evolve over time, it
remains at the same (high) level of detail throughout its lifecycle.
Is this all correct?

The draft can be left in the repository for historical tracking.

Yes, and: (a) to the draft should be prepended a header that lists the
commit hash in which the draft was accepted into the spec, and (b) the
draft file should thenceforth be frozen. This way, people can continue
to make sense of the diff proposed in the draft even as the actual spec
changes structurally.

Once we have a v1 spec, we'll tighten up the requirements for merging
a draft,

…and dedicate some effort to fixing up the existing spec, which will be
at varying levels of detail, to the desired precision, yes?

from specs.

Thanks! This feels like the right direction. Some suggestions:

• Let's make the goals explicit:
  • enable people to implement a filecoin node from the spec
  • capture design intent so that anyone evaluating a change or creating an implementation can benefit from it
  • benefit from the review/input from many different points of view (implementers, researchers, etc.) each of which might bring new, potentially useful information. Accrue this benefit early in the process (ie, get feedback early and often)
  • implementation stays in sync with the spec
• Additional requirements:
  • fidelity of detail should be commensurate with level of confidence/finality. Let's not waste peoples' time writing the bits up if something is in flux or likely to change.
  • process must be open and easy to hook into (no updates in slack channels, exactly one canonical place to do a thing, etc) [you have something like this but didn't have the open part, which maybe is implied]
  • process itself amendable by the process
• Let's have some anti-requirements, starting with:
  • bikeshedding on presentation. let's not waste time on making it pretty or too much about how it is structured. it must be readable and understandable yes, but let's worry a lot more about content than presentation (ie lets not have bikeshedding on sections, numbering, whatever)
  • scatter/smear information across a variety of things (PRs, docs, directories, etc)
  • add overhead that gets in the way of writing and building. If something is painful let's change it. The goal is not process, it's a protocol description.
  • sections in template documents that are of speculative usefulness. Let's start slim and add what we need based on real need.
• process
  • i think you need to say a few words about how a thing qualifies for the next stage, eg how does a proposal become a draft? how do we decide to merge a draft into the spec?
  • what's the lifecycle process? i think we should call out the explicit use cases:
    • i want to know proposals as they happen
    • i want to know about drafts as proposals graduate to them
    • i want to know when the spec changes so i can go implement
    • i'm making a proposal and i want the explicit input of some specific people or WGs
    • i feel strongly about a proposal (also, draft) and want to know when/how the decision is going to be made so i can come and make a fuss
    • i'm a WG captain and planning work way ahead i want to know about potential changes that aren't even at the proposal stage yet, for example if we are thinking of radically reworking a part of the protocol i might want to limit investment in the current implementation until the dust settles
    • i think there's a problem with the protocol where do i file a bug?
    • i'd like to ask for clarification about part of the protocol, where do i ask?
    • suppose i want to clear up some language in the spec, can I just propose a PR?
    • suppose i want to make one small change, say rework some enums in a message. do i have to make a proposal?
    • i'm going to implement something new that just landed in the spec and it's not going to work exactly like the spec says, we have to make some changes. do i stop my work and make a proposal? do i do the implementation and then make the proposal or draft? ideally for something like this i would do the implementation and then propose the deltas directly to the spec, kind of more like an update than a new proposal.
• proposals:
  • are all issues in that repo proposals? i would suggest the repo is most useful if the answer is yes, but it's unclear from reading what your intent is and where bugs if any would go in that case.
  • we should have a template for proposals. it should probably read like a super light weight design sketch with a clear problem statement, set of requirements, and a proposed solution and maybe alternatives. These four sections get at the core of intent. Here's how I've articulated it elsewhere: https://github.com/filecoin-project/notes/blob/master/designdocs.md#what-is-a-designdoc
• catching up
  • hooray, we need to do this
  • i propose the very first deliverable which is P0 and the number one priority if it is not already is a description of the spec as it is. all hands on deck, get something in there even if it is at high level of fidelity. new people are coming on every day and we are really feeling the pain.
  • you need to say what happens to everything that is in specs now, all the PRs and whatnot. Seems like they either get moved to proposals, to drafts, into the spec, or get shot in the head. i would love to see this happen soon after we have the item above.

Re some of the other comments:

• i definitely have a preference for keeping everything together, esp while it is in such flux as it is and will be for the next 6 months

from specs.

• i want to know proposals as they happen

Suggestion: “Watch” the repository. You’ll be notified of new issues and
pull requests, which correspond to proposals and drafts respectively.

• i want to know about drafts as proposals graduate to them

As above, watch the repository. (This doesn’t provide a way to watch for
only proposals or drafts.)

• i want to know when the spec changes so i can go implement

Not sure about this. Phabricator has Herald, which is what you want.
I’m not familiar with a GitHub equivalent. You could probably implement
a pure Git solution with a post-receive that would pick up changes to
the spec file or subtree, but this would only trigger when you fetch
from your local copy of the repository.

• i'm making a proposal and i want the explicit input of some
  specific people or WGs

Suggestion: @-reference them in the proposal issue. For working groups,
we could create GitHub teams, which can also be @-referenced.

• i feel strongly about a proposal (also, draft) and want to know
  when/how the decision is going to be made so i can come and make a
  fuss

Suggestion: For proposals, subscribe to the proposal issue. For drafts,
same comments as in “I want to know when the spec changes so I can go
implement”.

• i'm a WG captain and planning work way ahead i want to know
  about potential changes that aren't even at the proposal stage
  yet, for example if we are thinking of radically reworking a part
  of the protocol i might want to limit investment in the current
  implementation until the dust settles

Not sure about this. If the answer to “are all issues in the repository
proposals?” is “no”, and discussion of things that might become
proposals tends to happen in the repository, then watching the
repository suffices.

• i think there's a problem with the protocol where do i file a bug?

What we said earlier was: general questions (even serious ones) go to
Discourse; if you can point out a particular part of the spec that looks
wrong, then post an issue on this repository. (I’m not necessarily
endorsing this, just recalling it as a starting point.)

• i'd like to ask for clarification about part of the protocol, where
  do i ask?

As above: Discourse. If the clarification is generally useful, it can
give rise to a pull request (see next item).

• suppose i want to clear up some language in the spec, can I just
  propose a PR?

Suggestion: Yes, assuming that the change does not change the semantics
of the protocol.

• suppose i want to make one small change, say rework some enums in a
  message. do i have to make a proposal?

Suggestion: Yes. Anything that changes the semantics of the protocol
should go through the proposal process so that people watching the
repository can provide input.

• i'm going to implement something new that just landed in the spec
  and it's not going to work exactly like the spec says, we have to make
  some changes. do i stop my work and make a proposal? do i do the
  implementation and then make the proposal or draft? ideally for
  something like this i would do the implementation and then propose the
  deltas directly to the spec, kind of more like an update than a new
  proposal.

This is an interesting one. +1. I’m not sure.

• are all issues in that repo proposals? i would suggest the repo is
  most useful if the answer is yes, but it's unclear from reading what
  your intent is and where bugs if any would go in that case.

Why is the repository more useful if the answer is “yes” than if the
answer is “no, but all proposal issues are tagged with [proposal] and
all draft update PRs are tagged with [draft-new] or [draft-update]
or [draft-acceptance] as the case may be (or just [draft])?

As noted above, a benefit of also using issues for discussions of things
that may become proposals is that people watching the repository can
have more lead time.

• we should have a template for proposals. it should probably read
  like a super light weight design sketch with a clear problem
  statement, set of requirements, and a proposed solution and maybe
  alternatives. These four sections get at the core of intent. Here's
  how I've articulated it elsewhere:
  https://github.com/filecoin-project/notes/blob/master/designdocs.md#what-is-a-designdoc

+1.

• you need to say what happens to everything that is in specs now, all
  the PRs and whatnot. Seems like they either get moved to proposals, to
  drafts, into the spec, or get shot in the head. i would love to see
  this happen soon after we have the item above.

+1. @whyrusleeping said that most open PRs can die, save about two of
them. As to the fate of spec.md and drafts/, I have no comment.

from specs.

are all issues in that repo proposals? i would suggest the repo is
most useful if the answer is yes, but it's unclear from reading what
your intent is and where bugs if any would go in that case.

Why is the repository more useful if the answer is “yes” than if the
answer is “no, but all proposal issues are tagged with [proposal] and
all draft update PRs are tagged with [draft-new] or [draft-update]
or [draft-acceptance] as the case may be (or just [draft])?

To clarify I meant that watching the repo is most useful if each issue corresponds to a proposal or draft and there are no other uses for issues. This as opposed to using issues to also track bugs, have discussions that are not specific to a proposal/draft, etc. If issues are exactly one of a proposal or draft then the signal is really clear. If issues are a mixture of all kinds of spec-related stuff (eg, "write OKRs") then I have to filter/process it and figure out what is or is not relevant to my interests. Since I'm already doing this for a zillion repos I was kind of hoping there would be a super clear, high fidelity signal from the specs repo's issues. I'm not sure it's possible to restrict the use cases this narrowly or whether it's even a good idea, I'm just pointing out the signal/noise problem. We want the lifecycle events to be really easy to notice and find -- this is one way of achieving it.

i'm making a proposal and i want the explicit input of some
specific people or WGs

Suggestion: @-reference them in the proposal issue. For working groups,
we could create GitHub teams, which can also be @-referenced.

This doesn't work very well for busy people, they get so many issue updates and @s they tend to get lost in the shuffle. We've been trying https://github.com/filecoin-project/pm#how-to-use-this-repo to accommodate that fact for cross-WG communication, to mixed results. I'm not sure we have a great solution for this problem.

I would really love to see a fast-and-loose spec update protocol available to devs for stuff that is still emergent. In the next few months we are the only implementation and it'll be so much faster to go make small changes as direct spec PRs (eg, re-organizing enums or updating some details that changed because of implementation concerns). I can understand the thought that we should practice the way it should be now, but there's a real cost and I'm doubtful on the benefit of practicing extra process before we need it.

Beyond the next few months I would also like to see a fast-and-loose process for emerging / unsettled protocol features: maybe when something is in draft an implementation should work on it and they can feed back any changes that were necessary into the draft as direct PRs? Perhaps we're getting a little ahead of ourselves here, but I want to make it easy to make small changes because we're going to have a lot of them before the protocol reaches 1.0.

Basically: we should embrace the fact that any time we go to implement a draft there are going to be some things we overlooked that we only catch at implementation time. We should make feeding those changes back in easy because it's a good thing to make improvements to the spec based on what we learn, and if you make good things hard/onerous they happen less often.

from specs.

Thanks for the responses; this is really helpful.

Since I'm already doing this for a zillion repos I was kind of hoping
there would be a super clear, high fidelity signal from the specs
repo's issues.

Do email filters suffice? Notifications for this repository should have
subjects prefixed with [filecoin-project/specs] and also the List-ID
email header.

If this doesn’t suffice, how do other projects in this space handle this
issue? We surely don’t want to invest the time right now into switching
to Phabricator or setting up a bespoke notification system. I hope that
we can combine the existing primitives in ways that are good enough to
unblock us in the short term.

Basically: we should embrace the fact that any time we go to implement
a draft there are going to be some things we overlooked that we only
catch at implementation time.

Certainly. In some cases, it may be reasonable to keep a draft in draft
stage until the go-filecoin implementation is complete, so that once the
draft is accepted into the spec the implementation can immediately be
updated. But this probably doesn’t apply in all cases, so…

We should make feeding those changes back in easy because it's a good
thing to make improvements to the spec based on what we learn, and if
you make good things hard/onerous they happen less often.

…+1.

from specs.

Do email filters suffice? Notifications for this repository should have
subjects prefixed with [filecoin-project/specs] and also the List-ID
email header.

I'm talking about signal within the issues themselves. If an issue in the specs repo always contains either a draft or a (pointer to a?) proposal then every issues is relevant to people who want to keep track of what's going on with the spec. However if issues are drafts and proposals, but there are also issues that are discussions about OKRs, notes from meetings, debate about where to host the next meetup, ideas we should re-visit in the future, discussions of when and how to shave why's head, etc then it's really hard for me as someone interested in the spec to actually see what's happening with the spec. I have to sift through / ignore / filter all those other issues. I'm saying one way to ensure that it's really easy for people to locate and keep track of proposals and drafts is to have only proposals and drafts in the issues. In that model, issues in the specs repo are for drafts and proposals and only drafts and proposals. Super focused, clean, clear, no clutter.

from specs.

However if issues are drafts and proposals, but there are also issues
that are [other things] then it's really hard for me as someone
interested in the spec to actually see what's happening with the spec.
I have to sift through / ignore / filter all those other issues

Sorry, I’m not following. Certainly any single issue should not be both
a proposal and a meeting note transcript. If you want to filter the list
of issues to see what’s actually happening with the spec, just filter by
the label (e.g., label:WG-go-filecoin on the PM repo); that’s why
we have labels. This just seems like a total non-issue to me—surely
clicking on the “proposal” label is not too much to ask. I must be
misunderstanding something; what am I missing?

from specs.

what am I missing?

Usability.

from specs.

Link to draft meta-spec:

https://github.com/filecoin-project/specs/blob/spec-update-process/drafts/spec-update-process.md

Feel free to take a crack at adding/removing content. I added a note about having a draft spec template. Might make it easier for people to collect their thoughts and not focus on formatting too much. Let's get all these thoughts into a draft that we can merge with master. I'm thinking about having something in drafts/ master within a week or so. It'll be only a draft so it will not be taken as truth until we have consensus from all stakeholders which may take a while.

from specs.

Do we have agreement on this?

Sure, I'm all for whatever the team doing the work decides is the right thing.

Do we need an aligning call

Not from my end, you have my bits.

However one final word on usability / "how hard can it be to just click a label": you can't filter on labels in your email, so if labels are the thing that differentiates signal (updates to drafts, proposals, specs) from noise (background/coordination chatter) my email notifications do not give me the useful signal that I want. I have to manually filter out the noise, either by opening notifications or going to the web site and clicking around.

from specs.

Hi all! Sorry I've been absent on this thread the past week--was on vacation.
I've created a pull request that merges a more fleshed out proposal to master; you can see it here: #119

Some features of the proposal:

• Drafts all have owners, and explicit status:
  • WIP
  • PENDING APPROVAL
  • APPROVED PENDING MERGE
  • REJECTED
  • MERGED (at $hash)
• While a draft is WIP, the bias is towards approve and merge changes fast, so info does not languish in side branches or PRs. (Feedback+discussion happens in the issue for the proposal, not in the pull request.)
• To keep the spec repo high-signal, we don't allow non-proposal issues. (Then general questions/concerns should go in discuss.filecoin.io instead.)

In keeping with the proposed process, I'd ask that someone approve this pull swiftly, so that we can use master/drafts as the source of truth. If you would like to see specific changes,
please send pull requests rather than bikeshedding on this thread. If you have major concerns/questions, then posting in this thread is appropriate.

If people are broadly happy with this proposal, then we can merge it as official as soon as @whyruslepeing and @bvohaska approve. Then we can setup issue templates to discourage people from posting non-proposal issues, and redirect them to discuss.filecoin.io.

from specs.

We now have one.

from specs.

@pooja: Where is this documented?

from specs.

@bvohaska: here i believe: https://github.com/filecoin-project/specs/blob/master/process.md

from specs.

Yup! ^ @whyrusleeping If you were thinking of something else here

from specs.