Decide policy on multidev mutability about example-drops-8-composer HOT 2 OPEN

The use-case for having a usable pr multidev might be better served by simply using a ci-BUILD_NUMBER multidev to run all tests, and make a pr-PULL_REQUEST_ID multidev only for agency use -- never for test running.

I've often said that we're using these environments as both MultiDev and MultiTest. (Dev being an environment for working on a branch/PR over time and Test being an environment for reviewing tag/commit)

This question highlights how those overlapping usages don't always play well together. Making both pr-N and ci-N could help resolve that tension.

But it also has a high likelihood of increasing confusion for Pantheon customers overwhelmed by the number of moving pieces in this set up. I'd rather not cross that bridge right now if we don't have to. Such a decision feels to me like one that is best answered by a focus group of developers using these repos on real projects.

Though we do already have a precedent for a commit building to two environments. Commits on master build to ci-N and master on Pantheon.

For this specific issue, I'd like us to find a mechanism that can be configured per site. Yes, the starting point of the WordPress and Drupal repos should be the same. But if an individual team wants to restore after testing or not, it should be easy to switch.

One way we could do that would be by putting the mechanism for restoring/recloning into the running of ./vendor/bin/behat itself, perhaps with an exec() from an @afterSuite method in a `FeatureContext.php.

IMO a 30 or 70 line bash file wrapping ./vendor/bin/behat is a problem to be solved. When I do Jump Starts and walk people through running Behat tests for the first time it is a pain to explain which lines need to be copy/pasted out of those files so that ./vendor/bin/behat can be run from a local machine.

But maybe I'm being short sited by only thinking about this question of multidev mutability from the perspective of Behat.

I think there is a good chance that real projects will add more testing jobs that have the potential to change data (performance tests, load tests, visual regression tests). For instance, I can imagine that a project would want a Circle job to add 1000 pieces of content and make assertions about data coming from New Relic. The team would probably want those 1000 dummy posts gone before sending the multidev url to the client.

That team might ask us "So, when all of the build steps are done and every Circle job has run, will the Multidev be exactly how Dev would be if we click merge on the pull request?" I think the answer to the question probably should be "yes".

So all that to say, for teams that want it, a restore/reclone should run at the end of build_deploy_and_test or in a separate job following (not in the Behat bash wrapper). My hesitation at this point is that I'm concerned about the overall complexity of the build process and doing the same thing (setting up one multidev) twice in one run feels like a net increase in complexity.

from example-drops-8-composer.

doing the same thing (setting up one multidev) twice in one run feels like a net increase in complexity.

Yeah, it's hard to cover all of these use-cases without some increase in complexity. Backup/restore has more hidden complexity (works fine until you don't notice the tests erased a db change).

Another thing we might consider is the previous idea of running the ci-* tests on Circle instead of Pantheon. The on-circle tests wouldn't have the same fidelity as the on-Pantheon tests, but they'd run faster. If a subset of the tests ran in parallel on Circle (for pr-* builds as well), you'd get an early warning for some failures.

Of course, this does not satisfy the needs of anyone who wants to run a destructive test that only runs on Pantheon. If we used a Lando docker image on Circle, that might help some.

from example-drops-8-composer.