When backupStaticAttributes is enable, static data from a first test case leaks to the subsequent tests cases about phpunit HOT 10 CLOSED

I do not see a bug here.

from phpunit.

This is causing me problems as well.
I can workaround the issue by autoloading all affected classes before the test suite runs, but this is annoying and requires me to identify which classes are affected (which is difficult) and list them in my test bootstrap (which is a failure of encapsulation).

I think this is a real bug.

I'm not sure how this could be fixed. I can't see a "class loaded" event to which PHPUnit could subscribe. Perhaps @eriksencosta 's suggestion of wrapping the autoloader would work. However, any autoloaders added after the wrapping occurred would be missed, which seems quite likely to happen (e.g. Zend controller tests bootstrap Zend in the test "setup" method, at which point a lot of autoloaders are added).

from phpunit.

This bug still exists. I think this was closed by mistake due to an unrelated "#1" in the commit auto-linked above. Please could you re-open, to track the issue?

This may be difficult or impossible to fix, but it should still be tracked :-)

from phpunit.

@RichardBradley The example code repo from the OP no longer exists. Are you able to provide a new isolated test case?

from phpunit.

OK, will do

from phpunit.

I have created a repo with a reproduction case here: https://github.com/thehereward/phpunit-backup-static-attributes-bug

from phpunit.

To summarise the bug:

• PHPUnit has a feature "backupStaticAttributes" which is documented to "run your tests in a way where changes to [static] variables do not affect other tests"
• The implementation of this feature works by enumerating all loaded classes and copying the "before" values of their static fields into an array, then restoring those values after each test has run
• However, this doesn't work in many cases, because "get_declared_classes()" is called by the "backupStaticAttributes" feature before the test is run. If the test causes a new class to be auto-loaded during the test run, then its static fields will not be included in the "before" values, and any value which is added to a static field by that test will leak into all other tests for the rest of the suite run.

You may dismiss this by saying that the static field didn't exist before the test was run, so "backupStaticAttributes" couldn't catch it, but that does not account for how this feature appears to be intended to operate from the point of view of a PHPUnit user.

If I have a static class which looks like, for example:

  class MyGlobalStateHolder {
    static $scriptStartTime = time()
  }

... and if I want to modify this static field for the duration of a single test:

  class MyTimeDependentTest {
    protected function setUp() {
      // for the duration of this test only, pretend that it is 1984
      MyGlobalStateHolder::$scriptStartTime = 441763200;
    }
  }

Then, as things are documented I should expect the "backupStaticAttributes" feature to cause this field change to be isolated to this single test. This bug means that whichever test first loads the class in question will "win" and its static changes will leak to all other tests in the same run.

It is not clear to me that this is easily fixable, but it is a real bug.
Perhaps the fix should be documentation of the limitations of the "backupStaticAttributes" feature. or perhaps the "backupStaticAttributes" feature needs to be removed as not workable.

Please could you reopen the issue?
Do you have any idea how this might be fixed?

from phpunit.

Perhaps the restoreStaticAttributes() method could check whether any new classes have been autoloaded in the course of the test and, if so, issue a warning that "backupStaticAttributes" doesn't work in the presence of autoloaders?

That wouldn't fix the issue, but at least it would give a warning when it occurred, which might save people lots of time tracking down mysterious bugs that depend on the order in which tests are run.

from phpunit.

@RichardBradley Thanks a lot for the elaborate description and thanks @thehereward for redoing an isolated test case. I understood the problem now, and I think I also noticed that behavior already.

Especially with autoloaded classes, statics in newly loaded classes will not be reset, and worse, the list of statics to backup increases over time (because a new static of the previous test becomes a preexisting static to backup in all subsequent tests).

I think that's worth to investigate and fix. However, due to various issue references (unintentionally unquoted test data provider #1 as well as PHP stack trace frame #1), this issue continues to get automatically closed by GitHub. Can we move this into a new issue, using the last comments as a good, initial summary?

from phpunit.

OK, now recreated at #1372

from phpunit.