paratestphp / paratest Goto Github PK

All tests filenames must end with Test.php (as is enforced in the SuiteLoader).

I could be wrong, but should this always be done? Should it be done only if a .xml.dist file isn't given?

I should be able to run multiple test suites where the order of tests is explicit as specified in
http://phpunit.de/manual/current/en/organizing-tests.html#organizing-tests.xml-configuration

I tried to run paratest on the composer test suite, and at some point it fails with:

Warning: simplexml_load_file(): file:///C:/Users/seld/AppData/Local/Temp/PT_29D1.tmp:1: parser error : Document is empty in src\ParaTest\Logging\JUnit\Reader.php on line 27

I checked and indeed that file is just empty. I'm not sure how to debug it further to be honest.

Hi there,

I have been trying to get paratest working with our phpunit and selenium test suite and have hit a problem.
When I run paratest, the selenium - Chrome window opens and one test starts executing, paratest is then blocked until that test finishes and the browser window closes. The next test then starts.
Another issue I encountered was that when --filter was being passed to pup unit it was wrated in /\b \b/ which prevents phpunit from working. I removed /\b \b/ from the command to get phpunit to a state where it would start executing tests - again, I don't know if this is related to the lack of concurrency.
Stepping through the code, it looks like it is being blocked when reading from the process pipes used with proc_open.

I would be grateful if you have any advice in solving the above concurrency issue.

Is there a way to test concurrency with paratest?

For example a scenario something like this:

public function testMoneyIsNotLostByConcurrentTransfers(){
    $accountRepository = DataAccessLayer::getBankAccountRepository();
    $accountOfTom = $accountRepository->create(array(
        'owner' => 'Tom',
        'balance' => new Money(10000)
    ));
    $accountOfBob = $accountRepository->create(array(
        'owner' => 'Bob',
        'balance' => new Money(10000)
    ));
    $accountOfSusanne = $accountRepository->create(array(
        'owner' => 'Susanne',
        'balance' => new Money(10000)
    ));

    $this->runParallel(
        function () use ($accountOfTom, $accountOfBob){
            $accountOfTom->transfer($accountOfBob, new Money(5000));
        },
        function() use ($accountOfTom, $accountOfSusanne){
            $accountOfSusanne->transfer($accountOfTom, new Money(5000));
        }
    );

    $this->assertEquals($accountOfTom->getBalanceAmount(), 10000);
    $this->assertEquals($accountOfBob->getBalanceAmount(), 15000);
    $this->assertEquals($accountOfSusanne->getBalanceAmount(), 5000);
}

For testing concurrency a parallel test runner is needed... But if I am right paratest calls the test files only in parallel, not the code... Is it possible to build something upon paratest which can run code parallel?

In our SF2 project we have the phpunit.xml with wildcard

   <testsuites>
        <testsuite name="Project Test Suite">
            <directory>../src/*/*Bundle/Tests</directory>
            <directory>../src/*/Bundle/*Bundle/Tests</directory>
            <directory>../src/*/Bundle/*/*Bundle/Tests</directory>
        </testsuite>
    </testsuites>

the error I have is
Suite Path ../src/*Bundle/Tests could not be found

How do you solve this? Am I missing something?

Thanks to all.

Small issue I've noticed in the output for paratests : the percentage is often over 100.
Exemple:

$ $(pwd)/vendor/bin/paratest -c $(pwd)/app/phpunit.xml.dist $(pwd)/src/
Running phpunit in 5 processes with /home/travis/build/project/vendor/bin/phpunit
Configuration read from /home/travis/build/project/app/phpunit.xml.dist
...............................................................  63 / 159 ( 39%)
............................................................... 126 / 159 ( 79%)
............................................................... 189 / 159 (118%)
......................
Time: 53 seconds, Memory: 12.75Mb
OK (211 tests, 1091 assertions)

As you can see here, the percentage shown is 118%. Any idea how this can happen?
Could it be because of skipped / incomplete / @depend tests ?

When running the composer test suite with paratest, you have to give it --bootstrap=tests/bootstrap.php or it fails to load some abstract test classes properly, and results in this:

Fatal error: Class 'Composer\Test\TestCase' not found in C:\Users\seld\Web\composer\tests\Composer\Test\Autoload\AutoloadGeneratorTest.php on line 23

As you can see in the AutoloadGeneratorTest class extends the TestCase class. This class is made loadable via the composer autoloader in the bootstrap file, but paratest doesn't pick it up automatically it seems.

I've encountered an issue that I've narrowed down to using multiple @group tags per test method with Paratest. The same tests run successfully with phpunit.

I installed Paratest by cloning the repository, and then doing a php composer.phar install. A "composer.phar upgrade" yields the same result.

The directory structure is basically:
test_case/ (The commands are ran from this directory.)
test_case/paratest ( git clone'd paratest )
test_case/tests/Tests (The two sample test files are here.)

Sample_Test.php:

Hi, two questions about paratest:

*** 1. Is there a way to run phpunit in --debug mode with paratest, so that I can see the name of each individual test running as paratest is executing? I tried:

paratest --phpunit "phpunit --debug"

but that continues to run the tests like the --debug switch was never specified.

*** 2. Is there an option to have paratest continue running even after a phpunit crash? I am using HHVM to run various framework test suites and some of the tests in those suites cause fatal errors. So I get the

"Log file /tmp/PT_G8Mg9B is empty. This means a PHPUnit process has crashed."

type of message. Instead, is there a way to (1) spawn another instance of phpunit that continues with the next test after the crash? (2) run each test in its own process so that one crash does not affect all of the other tests (sort of like phpunit --process-isolation, I guess).

Thank you.

I think this issue continues #10.

If I use dataProvider in tests I get incorrect xml output (it different from standard phpunit). The difference is that phpunit output has another <testsuite> tag and testsuite has a different attribute name. See example below.

• PHPUnit 3.7.13 from Composer,
• Paratest 0.4.1 or dev-master (545c4a7)

Code example:

<?php

class simpleTest extends PHPUnit_Framework_TestCase
{
    public function provider()
    {
        return array(
            array('foo'),
            array('bar'),
        );
    }

    /**
     * @dataProvider provider
     */
    public function test1()
    {
        $this->assertTrue(false);
    }
}

vendor/bin/phpunit --log-junit=phpunit-junit.xml SimpleTest.php

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="SimpleTest" file="/home/a.bayandin/PhpstormProjects/Online4/tests/SimpleTest.php" tests="2" assertions="2" failures="2" errors="0" time="0.001455">
    <testsuite name="SimpleTest::test1" tests="2" assertions="2" failures="2" errors="0" time="0.001455">
      <testcase name="test1 with data set #0" assertions="1" time="0.001205">
        <failure type="PHPUnit_Framework_ExpectationFailedException">SimpleTest::test1 with data set #0 ('foo')
Failed asserting that false is true.

/home/a.bayandin/PhpstormProjects/Online4/tests/SimpleTest.php:18
</failure>
      </testcase>
      <testcase name="test1 with data set #1" assertions="1" time="0.000250">
        <failure type="PHPUnit_Framework_ExpectationFailedException">SimpleTest::test1 with data set #1 ('bar')
Failed asserting that false is true.

/home/a.bayandin/PhpstormProjects/Online4/tests/SimpleTest.php:18
</failure>
      </testcase>
    </testsuite>
  </testsuite>
</testsuites>

vendor/bin/paratest -p 1 --phpunit=vendor/bin/phpunit --log-junit=paratest-junit.xml SimpleTest.php

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="2" assertions="2" failures="2" errors="0" time="0.001525" file="">
    <testcase name="test1 with data set #0" class="" file="" line="" assertions="1" time="0.001247">
      <failure type="PHPUnit_Framework_ExpectationFailedException">SimpleTest::test1 with data set #0 ('foo')
Failed asserting that false is true.

/home/a.bayandin/PhpstormProjects/Online4/tests/SimpleTest.php:18
</failure>
    </testcase>
    <testcase name="test1 with data set #1" class="" file="" line="" assertions="1" time="0.000278">
      <failure type="PHPUnit_Framework_ExpectationFailedException">SimpleTest::test1 with data set #1 ('bar')
Failed asserting that false is true.

/home/a.bayandin/PhpstormProjects/Online4/tests/SimpleTest.php:18
</failure>
    </testcase>
  </testsuite>
</testsuites>

I've tried to run paratest own tests and the're failing:

PHPUnit 4.0.12 by Sebastian Bergmann.

Configuration read from .../phpunit.xml.dist

....E..........................................................  63 / 190 ( 33%)
...........................................FF...EEESSSESEE..E
Running phpunit in 5 processes with .../vendor/bin/phpunit

Configuration read from .../phpunit.xml.dist

E
Running phpunit in 5 processes with .../vendor/bin/phpunit

Configuration read from .../phpunit.xml.dist

. 126 / 190 ( 66%)
........F............................................EI.I...... 189 / 190 ( 99%)
.

Time: 5.42 minutes, Memory: 18.25Mb

There were 10 errors:

1) ParaTest\Coverage\CoverageMergerTest::testSimpleMerge
file(someFile): failed to open stream: No such file or directory

.../test/ParaTest/Coverage/CoverageMergerTest.php:16
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

2) ParaTest\Runners\PHPUnit\SuiteLoaderTest::testLoadFileGetsPathOfFile
Trying to get property of non-object

.../src/ParaTest/Runners/PHPUnit/SuiteLoader.php:85
.../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php:207
.../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php:104
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

3) ParaTest\Runners\PHPUnit\SuiteLoaderTest::testLoadFileShouldLoadFileWhereNameDoesNotEndInTest
Trying to get property of non-object

.../src/ParaTest/Runners/PHPUnit/SuiteLoader.php:85
.../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php:207
.../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php:111
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

4) ParaTest\Runners\PHPUnit\SuiteLoaderTest::testLoadDirGetsPathOfAllTestsWithKeys
Trying to get property of non-object

.../src/ParaTest/Runners/PHPUnit/SuiteLoader.php:85
.../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php:117
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

5) ParaTest\Runners\PHPUnit\SuiteLoaderTest::testGetTestMethodsReturnCorrectNumberOfSuiteTestMethods
Trying to get property of non-object

.../src/ParaTest/Runners/PHPUnit/SuiteLoader.php:85
.../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php:164
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

6) ParaTest\Runners\PHPUnit\SuiteLoaderTest::testGetTestMethodsOnlyReturnsMethodsOfGroupIfOptionIsSpecified
Undefined index: configuration

.../src/ParaTest/Runners/PHPUnit/SuiteLoader.php:85
.../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php:184
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

7) ParaTest\Runners\PHPUnit\SuiteLoaderTest::testExecutableTestsForFunctionalModeUse
Trying to get property of non-object

.../src/ParaTest/Runners/PHPUnit/SuiteLoader.php:85
.../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php:194
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

8) ParaTest\Runners\PHPUnit\RunnerIntegrationTest::testRunningTestsShouldLeaveNoTempFiles
unserialize(): Error at offset 0 of 235 bytes

.../src/ParaTest/Runners/PHPUnit/Runner.php:328
.../src/ParaTest/Runners/PHPUnit/Runner.php:224
.../src/ParaTest/Runners/PHPUnit/Runner.php:85
.../it/ParaTest/Runners/PHPUnit/RunnerIntegrationTest.php:25
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

9) ParaTest\Runners\PHPUnit\RunnerIntegrationTest::testLogJUnitCreatesXmlFile
unserialize(): Error at offset 0 of 235 bytes

.../src/ParaTest/Runners/PHPUnit/Runner.php:328
.../src/ParaTest/Runners/PHPUnit/Runner.php:224
.../src/ParaTest/Runners/PHPUnit/Runner.php:85
.../it/ParaTest/Runners/PHPUnit/RunnerIntegrationTest.php:40
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

10) PerformanceTest::testRunningSuitesWithLongBootstrapsIsFasterWithTheWrapperRunner
RuntimeException: Cannot parse output: {var_export(
Running phpunit in 2 processes with .../vendor/bin/phpunit

Configuration read from .../phpunit.xml.dist

.........F..E.F..F.............)}. The last executed command was `.../bin/paratest --bootstrap .../test/slow_bootstrap.php --phpunit .../vendor/bin/phpunit --runner WrapperRunner --processes 2 .../test/fixtures/tests`.

.../functional/PerformanceTest.php:104
.../functional/PerformanceTest.php:93
.../functional/PerformanceTest.php:34
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

--

There were 3 failures:

1) ParaTest\Runners\PHPUnit\SuiteLoaderTest::testLoadThrowsExceptionWithInvalidPath
Failed asserting that exception of type "PHPUnit_Framework_Error_Notice" matches expected exception "\InvalidArgumentException". Message was: "Trying to get property of non-object" at
#0 .../src/ParaTest/Runners/PHPUnit/SuiteLoader.php(85): PHPUnit_Util_ErrorHandler::handleError(8, 'Trying to get p...', '/mnt/hd/home/al...', 85, Array)
#1 .../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php(59): ParaTest\Runners\PHPUnit\SuiteLoader->load('/path/to/nowher...')
#2 [internal function]: ParaTest\Runners\PHPUnit\SuiteLoaderTest->testLoadThrowsExceptionWithInvalidPath()
#3 .../vendor/phpunit/phpunit/src/Framework/TestCase.php(952): ReflectionMethod->invokeArgs(Object(ParaTest\Runners\PHPUnit\SuiteLoaderTest), Array)
#4 .../vendor/phpunit/phpunit/src/Framework/TestCase.php(818): PHPUnit_Framework_TestCase->runTest()
#5 .../vendor/phpunit/phpunit/src/Framework/TestResult.php(686): PHPUnit_Framework_TestCase->runBare()
#6 .../vendor/phpunit/phpunit/src/Framework/TestCase.php(754): PHPUnit_Framework_TestResult->run(Object(ParaTest\Runners\PHPUnit\SuiteLoaderTest))
#7 .../vendor/phpunit/phpunit/src/Framework/TestSuite.php(675): PHPUnit_Framework_TestCase->run(Object(PHPUnit_Framework_TestResult))
#8 .../vendor/phpunit/phpunit/src/Framework/TestSuite.php(675): PHPUnit_Framework_TestSuite->run(Object(PHPUnit_Framework_TestResult))
#9 .../vendor/phpunit/phpunit/src/Framework/TestSuite.php(675): PHPUnit_Framework_TestSuite->run(Object(PHPUnit_Framework_TestResult))
#10 .../vendor/phpunit/phpunit/src/TextUI/TestRunner.php(421): PHPUnit_Framework_TestSuite->run(Object(PHPUnit_Framework_TestResult))
#11 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(179): PHPUnit_TextUI_TestRunner->doRun(Object(PHPUnit_Framework_TestSuite), Array)
#12 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(132): PHPUnit_TextUI_Command->run(Array, true)
#13 /usr/local/bin/phpunit(582): PHPUnit_TextUI_Command::main()
#14 {main}.

phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

2) ParaTest\Runners\PHPUnit\SuiteLoaderTest::testLoadBarePathWithNoPathAndNoConfiguration
Failed asserting that exception of type "PHPUnit_Framework_Error_Notice" matches expected exception "\RuntimeException". Message was: "Trying to get property of non-object" at
#0 .../src/ParaTest/Runners/PHPUnit/SuiteLoader.php(85): PHPUnit_Util_ErrorHandler::handleError(8, 'Trying to get p...', '/mnt/hd/home/al...', 85, Array)
#1 .../test/ParaTest/Runners/PHPUnit/SuiteLoaderTest.php(68): ParaTest\Runners\PHPUnit\SuiteLoader->load()
#2 [internal function]: ParaTest\Runners\PHPUnit\SuiteLoaderTest->testLoadBarePathWithNoPathAndNoConfiguration()
#3 .../vendor/phpunit/phpunit/src/Framework/TestCase.php(952): ReflectionMethod->invokeArgs(Object(ParaTest\Runners\PHPUnit\SuiteLoaderTest), Array)
#4 .../vendor/phpunit/phpunit/src/Framework/TestCase.php(818): PHPUnit_Framework_TestCase->runTest()
#5 .../vendor/phpunit/phpunit/src/Framework/TestResult.php(686): PHPUnit_Framework_TestCase->runBare()
#6 .../vendor/phpunit/phpunit/src/Framework/TestCase.php(754): PHPUnit_Framework_TestResult->run(Object(ParaTest\Runners\PHPUnit\SuiteLoaderTest))
#7 .../vendor/phpunit/phpunit/src/Framework/TestSuite.php(675): PHPUnit_Framework_TestCase->run(Object(PHPUnit_Framework_TestResult))
#8 .../vendor/phpunit/phpunit/src/Framework/TestSuite.php(675): PHPUnit_Framework_TestSuite->run(Object(PHPUnit_Framework_TestResult))
#9 .../vendor/phpunit/phpunit/src/Framework/TestSuite.php(675): PHPUnit_Framework_TestSuite->run(Object(PHPUnit_Framework_TestResult))
#10 .../vendor/phpunit/phpunit/src/TextUI/TestRunner.php(421): PHPUnit_Framework_TestSuite->run(Object(PHPUnit_Framework_TestResult))
#11 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(179): PHPUnit_TextUI_TestRunner->doRun(Object(PHPUnit_Framework_TestSuite), Array)
#12 phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php(132): PHPUnit_TextUI_Command->run(Array, true)
#13 /usr/local/bin/phpunit(582): PHPUnit_TextUI_Command::main()
#14 {main}.

phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

3) GroupTest::testGroupsSwitchExecutesMultipleGroups
Failed asserting that '
Running phpunit in 5 processes with .../vendor/bin/phpunit. Functional mode is on

Configuration read from .../phpunit.xml.dist

' matches PCRE pattern "/OK \(3 tests, 3 assertions\)/".

.../functional/GroupTest.php:37
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:179
phar:///usr/local/bin/phpunit/phpunit/TextUI/Command.php:132

FAILURES!                                                                       
Tests: 186, Assertions: 295, Failures: 3, Errors: 10, Incomplete: 2, Skipped: 4.

Maybe related to problem discovered in #79

ParaTest has definitely solved some issues for me - particularly in the functional testing realm, and I am glad to see it has done the same for a bunch of other people.

I think it is becoming too cumbersome to rely on PHPUnit as a hard dependency. The benefit of this approach is you can run existing PHPUnit suites concurrently, but PHPUnit can introduce a breaking change - and I don't think it is worth the investment to have ParaTest fully support all PHPUnit configuration options.

I have been drawing up some ideas for a new test framework that has concurrency baked in.

Any thoughts on converting ParaTest to a [READONLY] and beginning work on a new framework altogether (particularly one that can take advantage of php 5.5 coroutines and generators?)

The error below happened on this travis build

1) PHPUnitTest::testRunWithFatalRuntimeErrorWithTheWrapperRunnerOutputsError
Failed asserting that 'PHP Catchable fatal error:  Argument 1 passed to ParaTest\Runners\PHPUnit\Worker::assign() must be an instance of ParaTest\Runners\PHPUnit\ExecutableTest, null given, called in /home/travis/build/brianium/paratest/src/ParaTest/Runners/PHPUnit/WrapperRunner.php on line 86 and defined in /home/travis/build/brianium/paratest/src/ParaTest/Runners/PHPUnit/Worker.php on line 47
PHP Stack trace:

It also happens on my dev (on around 70% of the runs) machine when running

bin/paratest -p 2 --runner WrapperRunner test/fixtures/tests/EnvironmentTest.php

Not sure of the specific details how this works but basically Laravel needs to create an instance of app to allow for full testing which at a point creates a session which needs to create a random number using Illuminate\Support\Str.php

There was 1 error:

1) SampleTest::testFoo
RuntimeException: Unable to generate random string.

C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Support\Str.php:190
C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Support\helpers.php:764
C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Session\Store.php:14
C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Session\SessionServiceProvider.php:108
C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Foundation\Application.php:605
C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Foundation\Application.php:564
C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Foundation\start.php:208
C:\Users\lotus\projects\laravel_testing\bootstrap\start.php:61
C:\Users\lotus\projects\laravel_testing\app\tests\TestCase.php:16
C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Foundation\Testing\TestCase.php:39
C:\Users\lotus\projects\laravel_testing\vendor\laravel\framework\src\Illuminate\Foundation\Testing\TestCase.php:29

FAILURES!
Tests: 1, Assertions: 0, Failures: 0, Errors: 1.

Here is the test.

<?php
class SampleTest extends TestCase {
    public function testFoo()
    {
        $this->assertTrue(true);
    }
}

I removed all other tests including the exampleTest.php shipped with Laravel. Changing to extend PHPUnit_Framework_TestCase directly from the test class resolves the issue for Paratest but certain testing in Laravel requires TestCase to be extended.

I know, that according to docs the logging section (see http://phpunit.de/manual/3.7/en/appendixes.configuration.html) of phpunit.xml file isn't supported.

Example:

<logging>
    <log type="coverage-html" target="build/coverage" title="BankAccount"
        charset="UTF-8" yui="true" highlight="true"
        lowUpperBound="35" highLowerBound="70"/>
</logging>

It would be great if at least coverage-related logging would be supported as well to compliment --coverage-html, --coverage-clover and --coverage-php command-line options.

When i define a test suite that has several <directory> entries, Paratest takes only the first one and ignores the others:

With a definition like this

    <testsuites>
        <testsuite name="ParaTest Fixtures">
            <directory>/src/Tests1/</directory>
            <directory>/src/Tests2/</directory>
        </testsuite>
    </testsuites>

only /src/Tests1/ directory will be scanned.

Hello,

For my unit tests, i need to have a specific "HTTP_HOST", so I put this in my phpunit.xml:

I use "$_SERVER['HTTP_HOST']" in my bootstrap.php (declared in the phpunit.xml).

It works with phpunit, but when I run it with paratest, this variable is not set.

Is this a bug or did i make something wrong?

Thank you.

I think we are at a point where we can tag a new version. There have been some solid additions since the last tagged version. The change log goes back to April 7.

I am thinking it would be beneficial to start including a change log. I'll take a run at making it, and maybe someone can review it? The biggest change is the addition of the TEST_TOKEN feature, and then a lot of various enhancements and bug fixes. Does a v0.5.0 release sound appropriate?

If we run tests using phpunit or paratest - the results are expected:

ivan@ivan-ubuntu-1204d:~/paratest$ bin/paratest 

Running phpunit in 5 processes with /home/ivan/paratest/vendor/bin/phpunit

Configuration read from /home/ivan/paratest/phpunit.xml.dist

...............................................................  63 / 157 ( 40%)
............................................................... 126 / 157 ( 80%)
...............................

Time: 02:51, Memory: 6.25Mb

OK (157 tests, 487 assertions)

If we run paratest -f - it doesn't match to expectations:

ivan@ivan-ubuntu-1204d:~/paratest$ bin/paratest -f

Running phpunit in 5 processes with /home/ivan/paratest/vendor/bin/phpunit. Functional mode is on

Configuration read from /home/ivan/paratest/phpunit.xml.dist

...............................................................  63 / 157 ( 40%)
............................................................... 126 / 157 ( 80%)
..............................

Time: 01:21, Memory: 8.75Mb

OK (156 tests, 431 assertions)

Seems like all the tests that have @depends are skipped. And actually that should be logged.

I've been debugging a weird test stall issue. It involves a phpunit bootstrap running liquibase migrations. Liquibase outputs a lot of information to stderr, so I patched Worker to use "exec $wrapperBinary 2>&1 | tee worker-$token.log" to debug whats happening between the processes, and it fixed the issue. Then I looked at the code and saw the stderr is only read if there's an error. Seems like the stderr buffer fills up and all subsequent stderr prints get stalled after.

Not sure about the right thing to do here, but I think stderr should be read in a loop too, buffering its output in the worker class to be available in case of an error.

Have following phpunit.xml:

<filter>
    <whitelist>
        <directory suffix=".php">test</directory>
    </whitelist>
</filter>

<logging>
    <log type="coverage-html" target="./log/codeCoverage" charset="UTF-8"
        yui="true" highlight="true"
        lowUpperBound="50" highLowerBound="80"/>
</logging>

with directory structure proposed in http://net.tutsplus.com/tutorials/php/parallel-testing-for-phpunit-with-paratest/

I changed some test to take longer time (bigger sleep) and instead of general code coverage I see only coverage for longest running test after all.

Running these tests with phpunit gives expected result with 100% code coverage for each test and method inside.

Please see screenshots in attachment

As symfony 2.4 will be soon releasing it's first RC, it would be nice to allow developers to use paratest with it.

This would mean updating composer.json process and console dependencies to something broader.

What do you think?

The parser needs to load the older PSR-0 convention of converting _ to directory separator. The problem right now lies in assuming namespaces. in Parser.php

private function getClassName()
{
      $class = str_replace('.php', '', basename($this->path));
      $namespace = $this->getNamespace();
      if($namespace)
          $class = $namespace . '\\' . $class;
      return $class;
}

Basically if $class does not exist, walk up the path until a class is found. If not found by project root, then bail with an exception.

That's what happens on just checked out 1aa8a8e revision:

ivan@ivan-ubuntu-1204d:~/paratest$ bin/paratest -f functional/

Running phpunit in 5 processes with /home/ivan/paratest/vendor/bin/phpunit. Functional mode is on

Configuration read from /home/ivan/paratest/phpunit.xml.dist

..........................................

Time: 01:12, Memory: 4.50Mb

OK (42 tests, 62 assertions)
ivan@ivan-ubuntu-1204d:~/paratest$ bin/paratest functional/

Running phpunit in 5 processes with /home/ivan/paratest/vendor/bin/phpunit

Configuration read from /home/ivan/paratest/phpunit.xml.dist

......................................

Time: 02:49, Memory: 3.75Mb

OK (38 tests, 56 assertions)

After specifying --log-junit I've found that in my case these tests are called twice:

• GroupTest::testGroupSwitchOnlyExecutesThoseGroupsInFunctionalMode
• GroupTest::testGroupSwitchOnlyExecutesThoseGroupsWhereTestHasMultipleGroups
• OutputTest::testInstantFeedbackIsDisplayedWhenAndFunctionalModeDsiplayed
• PHPUnitTest::testFunctionalWithBootstrapUsingShortOption

UPD:

the reason for that is obvious: there is a GroupTest::testGroupSwitchOnlyExecutesThoseGroups test which is being called like --filter testGroupSwitchOnlyExecutesThoseGroups which matches to testGroupSwitchOnlyExecutesThoseGroupsInFunctionalMode as well.

calling PHP_Timer methods statically is causing issues in strict mode. This should be updated to actually instantiate an instance of the timer in ResultPrinter

Paracode seems to always exit with 0 but should exit with a different code if tests failed (best would be to return whatever phpunit returned)

I'm getting this a lot when I run Selenium tests in parallel:

PHP Warning:  Invalid argument supplied for foreach() in /Users/jlipps/Code/sausage/vendor/brianium/paratest/src/ParaTest/Runners/PHPUnit/ResultPrinter.php on line 68

Thought it might be nice to have a start message like the cool kids:

paratest running <path> with 5 processes in functional mode

..............

Time: 0:40, Memory 1.25 MB

(OK, X test, X assertions)

jlipps sent me here -

phpunit has JUnit XML output to be parsed by things like Jenkins. With paraunit, I hacked it so that each phpunit process would dump it's own XML to a folder that Jenkins would parse. It would be nice to standardize this a little bit to get it to be identical to the same XML returned by a plain phpunit run.

Tag current code to indicate PHPUnit 4.x compatibility. I guess it will be 0.7.0 version.

It is really great to see things being merged in. Up to this point, the version numbers have been following something along the lines of "major.minor.patch" i.e 0.4.4. Any suggestions for managing these numbers in the future? If we stick with this method, it might be nice to update at least the minor version to account for the latest additions.

This also brings up the question: do we have a version 1 goal?

Thanks again to everyone involved.

A cool side note, paratest is pushing 5400 installs on packagist.

Hi, I get this error when I try to run paratest on a largish Symfony2 application.

I've tried various combinations of paratest options but they all yield the same error.

Any ideas why this happens?

bin/phpunit --version
PHPUnit 3.7.15 by Sebastian Bergmann.

bin/paratest --version
ParaTest version 0.4.4

Symfony version is 2.2.1.

PHP Fatal error: Cannot redeclare class Fpg\BackgroundBundle\Tests\JobServerControlTest in /home/tarjei/htdocs/frisvold/fistup/src/Fpg/BackgroundBundle/Tests/JobServerHandlerTest.php on line 70
PHP Stack trace:
PHP 1. {main}() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/bin/paratest:0
PHP 2. Symfony\Component\Console\Application->run() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/bin/paratest:35
PHP 3. ParaTest\Console\ParaTestApplication->doRun() /home/tarjei/htdocs/frisvold/fistup/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
PHP 4. Symfony\Component\Console\Application->doRun() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/src/ParaTest/Console/ParaTestApplication.php:27
PHP 5. Symfony\Component\Console\Command\Command->run() /home/tarjei/htdocs/frisvold/fistup/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:193
PHP 6. ParaTest\Console\Commands\ParaTestCommand->execute() /home/tarjei/htdocs/frisvold/fistup/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:240
PHP 7. ParaTest\Console\Testers\PHPUnit->execute() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/src/ParaTest/Console/Commands/ParaTestCommand.php:30
PHP 8. ParaTest\Runners\PHPUnit\Runner->run() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/src/ParaTest/Console/Testers/PHPUnit.php:32
PHP 9. ParaTest\Runners\PHPUnit\Runner->load() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/src/ParaTest/Runners/PHPUnit/Runner.php:27
PHP 10. ParaTest\Runners\PHPUnit\SuiteLoader->load() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/src/ParaTest/Runners/PHPUnit/Runner.php:64
PHP 11. ParaTest\Runners\PHPUnit\SuiteLoader->initSuites() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/src/ParaTest/Runners/PHPUnit/SuiteLoader.php:43
PHP 12. ParaTest\Parser\Parser->__construct() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/src/ParaTest/Runners/PHPUnit/SuiteLoader.php:82
PHP 13. ReflectionClass->__construct() /home/tarjei/htdocs/frisvold/fistup/vendor/brianium/paratest/src/ParaTest/Parser/Parser.php:22
PHP 14. Composer\Autoload\ClassLoader->loadClass() /home/tarjei/htdocs/frisvold/fistup/vendor/composer/ClassLoader.php:0

the phpunit.xml or phpunit.xml.dist that bootstraps a test runner should work in paratest as well

Assume we have a standart Symfony2 phpunit.xml.dist. There are pathes like

<directory>../src/*/*Bundle/Tests</directory>
<directory>../src/*/Bundle/*Bundle/Tests</directory>

As far as i understand it is not supported by SuiteLoader for now.
It is easy ta add such a support by adding symfony/finder to composer.json and using it in loadPath.
I can do the PR if any.

Hi There,

I have an issue which I want test my automation with paratest, but how can I test my all cases in one command? because if I do like this :

saucelab/vendor/bin/paratest -f

my terminal always show :

[RuntimeException]
No path or configuration provided (tests must end with Test.php)

Hard for me if I cannot use phpunit.xml because all my configuration stored there.

same question posted here #16

note : i am using the newest code from repo anyway

Thank you very much!

Unit tests have the potential to run in parallel on multiple cores. [1]
However, there is some overhead:

• creating a process for each test is costly
• the bootstrap of the test is repeated every time
  So I'm experimenting on a proof of concept [2] where N workers are created at the start and they take new 'phpunit ...' commands to execute on their standard input. [3]
  How this may fit inside paratest one day? Creating a new Runner would be ok?

[1] https://github.com/giorgiosironi/paratest-example/blob/master/dualcore.txt
[2] https://github.com/giorgiosironi/paratest/commits/reuse_bootstrap
[3] https://github.com/giorgiosironi/paratest/blob/726509462a553da98962313a635bc566366c1a7d/bin/phpunit-wrapper

Testing in Windows, some of these new features that rely on environment variables do not behave as expected. The last tagged version of 0.4.4 is the last version to support Windows.

My coworkers are having this error everytime they run paratest. If they use phpunit, it works fine. On my computer it works fine anyway.

[Exception] String could not be parsed as XML

Any idea why ?

@ChubV - who contributed to some work on feature/windows-compat - brought up a good question. Do we have a certain road map for this project? Can we leverage the wiki or milestones in a similar fashion to the bootstrap project to have some plans for this?

https://github.com/twitter/bootstrap/wiki/Roadmap
https://github.com/twitter/bootstrap/issues/milestones?state=open

Do we have a 1.0.0 goal? Do we care about these things? This tool has been really helpful for me, and I am glad a lot of people are getting use out of this.

Some talking points for me would be:

• Is it worth looking into a better tool for communicating as a group? If so what tool?
• What are priorities? Some things that come to my mind: Robust configuration support, colored output, WrapperRunner, Supporting other tools like Codeception?
• Helpful to have a google group? Is that overkill right now?

I think it would at least be nice to have some form of road map. Thoughts?

My test suite runs 48 seconds locally. When executed locally with paratest it takes it 49 seconds to complete. However when executed on Travis it's executed 8 seconds.

Is there a way to detect why it's not working locally?

Also I've tried to run paratest tests locally and they're hanging in the middle of the process.

Incomplete/Skipped tests are not accounted for.
For example if you have 450 regular tests and 50 incomplete or skipped for a total of 500, you are expected to see the progress bar going 63/500, 126/500, where the 126 are the regular ones only.
So the progress bar will stop before 100% at 450 and in the end you will see
OK (450 tests, X assertions)

I think there is an issue with parallel tests using @Depends. These are likely being skipped when run individually. This could be fixed by adding the dependency to the filtered tests, essentially creating a pattern

phpunit --filter "providerTest|dependentTest" TestFile.php

This has likely been overlooked because the reporter doesn't show skipped tests

• get provider test from doc block and prepend to --filter
• add skipped test feedback to reporter