Should missing snapshots be automatically created? about cupaloy HOT 11 CLOSED

Hi! Would it be possible to configure so that missing snapshots are not automatically created?

from cupaloy.

That's certainly possible, can I ask what your use case is? Just looking to understand better what the solution should look like.

At the moment the easiest thing would be the addition of a DontCreateNew configurator allowing you to:

snapshotter := cupaloy.New(cupaloy.DontCreateNew)

And then use that snapshotter in your tests.

However I have been thinking about a way to apply options to the library methods i.e. to set DontCreateNew globally and be able to use cupaloy.SnapshotT as normal in tests. I assume that would be preferable?

from cupaloy.

I'm not sure if this is exactly what I need but perhaps I could help you understand better what my use case is.

I am using bazel as part of my workflow. The implication is that bazel test runs in a separate test environment so files written do not get persisted. This means that some of the error messages are not very accurate (ie. it says snapshot created when it's not actually created)

To resolve this, I would like to create a script that runs go test which will generate/update the snapshots. I want the intent to be clear that I'm only going to create new snapshots whenUPDATE_SNAPSHOTS=true go test ./... is called (preferably also not fail the tests. I could use Snapshot instead of SnapshotT to take over error handling but I want the name to be t.Name() still...this is a separate issue)

When I run the test through bazel, I want to be able to recognize a) that the snapshots don't currently exist or b) current outputs don't match snapshots and show the diffs. (go test should have the same behavior if UPDATE_SNAPSHOTS is not set)

Let me know if I can further clarify anything, perhaps I'm not expressing my thoughts as best as I could.

from cupaloy.

So this sounds like you want two new options: don't create new snapshots, and/or don't fail on updating snapshots?
(If new snapshots are not created then I assume the test should still fail, in which case this is simply a change in the error message?)

With these options you could check in your test setup whether the tests are being run through bazel and if so apply the relevant options.

from cupaloy.

@janivihervas I see you have a fork to implement similar behaviour.
I'd love to get some feedback from you whether the above solutions would be any good (this workflow is clearly something cupaloy should support without too much friction).

from cupaloy.

Sure! My fork doesn't fail tests if new snapshots are created or updated when UPDATE_SNAPSHOTS=true. This is because I'm heavily using code bots that will commit the build results back to the PR and I'm using them for code formatting, test snapshots, screenshots etc. Example config can be seen here: https://github.com/janivihervas/jwt-auth-proxy/blob/master/.circleci/config.yml (updates go.mod and go.sum files, formats, updates snapshots)

Basic idea is so that instead of this flow

1. Developer creates a PR, new build starts
2. Build fails because snapshots got updated or code wasn't formatted correctly etc.
3. Developer checks the build logs to see why the build failed
4. Developer will format the code or update the snapshots (which can take a long time to update) and pushes a new commit
5. Build succeeds, PR merged

I can do this:

1. Developer creates a PR, new build starts
2. Build succeeds and commits new and updated snapshots, correctly formatted code etc back to PR. This will start a new build which will succeed and not update anything anymore
3. Developer notices code bot changes, reviews what was changed and merges the PR

So in my opinion cupaloy could have a toggle that will not fail the tests if any of the snapshots got updated or created. Only updating and not creating new ones sounds a bit odd, because there was a change after all and at least I don't see a reason to differentiate between the two. And this is how for instance jest works: --updateSnapshot will update and create new snapshots.

My fork could be updated with a new environment variable (FAIL_ON_UPDATE etc, defaults to "true") in order to keep the change backwards compatible. I have been meaning to create a PR about it, but I've been just swamped with other stuff. If you don't mind, I could try to find time this weekend to do it.

from cupaloy.

Ah interesting yeah that's definitely something that should be supported.

Now I'm wondering how to support this and similar behaviour without hard coding a load of environment variables that are checked. Ideally what I want is for the environment variable checking code to be user written and then, based on that check, a global cupaloy option can be enabled/disabled. E.g. something like

if //check environment variable {
  cupaloy.Global = cupaloy.New(cupaloy.FailOnUpdate(false))
}

put into an init() or test setup method.

Unfortunately given how tests are compiled, this would have to be duplicated in every package.
I'll have a think about alternative ways to support this.

from cupaloy.

I think if cupaloy can be configured in code it'd be good. This will give developers using cupaloy freedom to determine how to enable/disable FailOnUpdate/ShouldUpdate/etc (whether through env variables, go flags, etc). I wrap cupaloy snapshots in another function to control/configure its behavior to be consistent across all tests.

from cupaloy.

Agreed, the one problem I'm running up against is that (AFAIK) go compiles each test package separately and so, if you want consistent behavior across tests, then you'll have to copy/paste that configuration into every single package.
This is behavior that you always want so is irritating to need to e.g. always run go test with a flag or remember to copy/paste a config into all new test files.

I've got some time this weekend so I'll have a play with this idea and see how it might work.

from cupaloy.

Let me know what you think of the referenced proposal ^

I think this provides some solution to your use case @byang223: in the init function you'll be able to check whether or not you're being run by Bazel and if so decide whether or not to change the global config.

from cupaloy.

I made a comment regarding whether the test will fail when it is a new snapshot. Ideally I would like to be able to configure cupaloy so that I can explicitly say I want to create new snapshots vs check current snapshots. Create/updating snapshots would ideally not fail the test.

from cupaloy.