[SDK Stability] Add plugin-specific scaffolding test interface to decide if a plugin version should be updated or not about operator-sdk HOT 1 OPEN

@itroyano Thanks for raising the issue. Some info on how this is done today:

When a new plugin is released (for ex. go/) and we would like to pull that into SDK, the developer:

1. Creates a new branch and tries updating the go.mod.
2. Updates the template for scaffolding plugin in cli.go -

   operator-sdk/internal/cmd/operator-sdk/cli/cli.go

   Line 84 in de40e50

   gov2Bundle, _ := plugin.NewBundleWithOptions(

3. After which, the testdata (https://github.com/operator-framework/operator-sdk/tree/de40e5091cc33bbd08592aa1da0007d9bfad6eb9/testdata) is updated using make generate. During the updation of testdata, the operator-sdk command then calls in the new plugin version and scaffolds out the files using the operator-sdk <commands> (

   operator-sdk/hack/generate/samples/generate_testdata.go

   Lines 35 to 60 in de40e50

   	flag.StringVar(&binaryPath, "bin", testutils.BinaryName, "Binary path that should be used")
   	flag.Parse()
   	// Make the binary path absolute if pathed, for reproducibility and debugging purposes.
   	if dir, _ := filepath.Split(binaryPath); dir != "" {
   	tmp, err := filepath.Abs(binaryPath)
   	if err != nil {
   	log.Fatalf("Failed to make binary path %q absolute: %v", binaryPath, err)
   	}
   	binaryPath = tmp
   	}
   	wd, err := os.Getwd()
   	if err != nil {
   	log.Fatal(err)
   	}
   	// samplesPath is the path where all samples should be generated
   	samplesPath := filepath.Join(wd, "testdata")
   	log.Infof("writing sample directories under %s", samplesPath)
   	log.Infof("creating Helm Memcached Sample")
   	helm.GenerateMemcachedSamples(binaryPath, samplesPath)
   	log.Infof("creating Go Memcached Sample with Webhooks")
   	golang.GenerateMemcachedSamples(binaryPath, samplesPath)

   ).

With this change if there is any issue during scaffolding the tests do not pass, and CI errors. There is no automation here though, but we do catch scaffolding issues before merging anything.

This works well currently with Helm and Go plugins. But with ansible (recently moved out external plugin), generation of testdata gets a bit complicated since we scaffold the templates in the external repository: https://github.com/operator-framework/ansible-operator-plugins/blob/4e8c78d21b6017ab0a280e3d274d97edf50b8fc4/hack/generate/samples/ansible/generate.go#L107. The scaffolded operator is tested in the plugin repository for any plugin update (https://github.com/operator-framework/ansible-operator-plugins/blob/main/test/e2e/ansible/suite_test.go) before updating it in SDK.

Having improvements in testing external scaffolds would be helpful, however to do so we need to discuss on how we would like to split tests between the external plugin and SDK repos.

from operator-sdk.