Rethink CCPP framework development cycle and branch structure about ccpp-framework HOT 6 OPEN

Hi @mkavulich et al., after discussion from the group here at NCAR/CGD, we just wanted to share our overall thoughts:

• As mentioned in previous meetings, our biggest fear is needing a critical bug fix for some release version of a host model, and then having that be postponed for weeks or months because the PR is waiting on testing from all of the participating host models. Right now, the UFS is the only host model doing offline testing that could hold up a PR, but in the future a PR could be held up by CAM-SIMA testing as well.

  One way to avoid this problem is to simply ensure that all testing of the main branch is done within a certain time period (which is easily done if everything is automated), or, if that isn't doable, that PRs can occasionally be labeled "urgent" and fast-tracked within each host model's testing workflow.

  However, if speeding up testing (or at least ensuring a hard timeline) isn't feasible, then we'll basically have to have more than one branch, with main still being the branch that has the most rigorous testing (i.e. every participating host institution signs off).
  It should also be noted that the overall testing burden will likely continuously increase as more host models add their own required tests to the main branch, so this may become a problem in the future even if we greatly improve our efficiency for the current set of tests.

• Adding new branches/forks for each host model is certainly feasible, but our concern is that it will quickly lead to code divergence, especially if host models find that there isn't really a good reason to push certain features back to main (e.g. "This feature will only ever be used in our host model, so no need to push it back").

  Also I think it's important to note that the CCPP-framework arguably shouldn't follow the organization of CCPP-physics. Host models will always have particular science goals or objectives that may cause their physics schemes to be specific to a particular host institution (even if they are still inter-operable from a CCPP software standpoint). However, the goal of the framework is that all host models are using literally the exact same framework code. If we collectively end up using both different physics AND different frameworks then I think the core goal of interoperability will become almost impossible.

• On the other hand, having a development branch within the NCAR/CCPP-framework repo will both reduce the risk of code divergence and allow for hashes/tags to be created without necessarily waiting for testing from every host model (while still doing at least some minimal testing via the current automated testing system we have now).

  Having a development branch may occasionally cause noticeably difficult merges, especially if development is moving quickly, but to be honest the current workflow of only having main will have the exact same problem, as will having separate branches or forks whenever they are synced to main. In other words, I don't think there really is a way to avoid that particular problem.

Given this, assuming there is no way to significantly speed up testing, we are leaning towards having a shared development branch. Of course we are always happy to discuss this further if folks still have concerns. Thanks!

from ccpp-framework.

I like this. The development branch should be named develop following standard practices.

from ccpp-framework.

This all seems sensible to me, but isn't the governance of development outside of "main" none of our business?

from ccpp-framework.

After further discussion at the framework meeting (see Meeting Minutes for details), we have three options:

• Some version of what is proposed in this issue: a shared "develop" branch that is merged into "main" with some regular cadence
• Separate develop branches for each host model
• Status quo - main branch only

In an effort to keep track of people's ideas, I've started a pro/con chart here. I have populated it with themes and ideas I pulled from our meeting, but feel free to edit with your thoughts. I will post a screenshot of the final list here in this issue when we're "done" for record-keeping; here's what it looks like now:

from ccpp-framework.

@peverwhee Thanks again for listing this all out and organizing the pros and cons. After some time to digest the topic, I've come up with some (mildly rambly) thoughts:

Off the bat, I am leaning towards solution 2 (separate branches for each host model). Specifically, a modified proposition that would use forked repositories rather than simple branches within the main repository: this would bring ccpp-framework more in line with how ccpp-physics is handled, and would allow CESM/SIMA and other individual projects to work at their own development pace and only take contributions from the main branch/repository when they are ready.

My reason for wanting forks is that this will allow for issues and PRs specific to each dycore to be appropriately separated so that A. Issues and PRs will be seen more easily by the people most able to fix them, and B. The main repository will remain less cluttered and easier to maintain.

In order for this to actually work, we would need buy-in from all groups maintaining forks that

1. They will follow all CCPP standards in their fork as defined in the CCPP Technical Documentation
2. They will run all tests from the main branch on their own fork on a regular basis (preferably these would pass for all changes) and
3. They will offer in-kind help to ccpp-framework code managers for any trouble merging changes to the main branch, and make reasonable efforts to adopt any requested changes from ccpp-framework code managers.

As a counterpoint to the testing requirements from the main branch, groups with forks would also have the right to insist on any tests they require being run on all development to the main branch. In fact, the more automated testing at more steps, the better!

As for what kind of cadence to merge changes back into the main repository, I don't really know what makes sense.

from ccpp-framework.

Updated pro/con list with contributions from @mkavulich

from ccpp-framework.