Native implementation? about paratest HOT 28 CLOSED

Definitely a yes for the new test framework ;)
You could also use PHPUnit as a library instead of wrapping it. This way, you could reuse the report writer, xdebug coverage integration, etc.. That might save a lot of work / code.

But I would beg you to put paratest in a "bugfix-only"-mode instead of readonly-mode because from my point of view, the main purpose of paratest is to execute an existing phpunit testsuite and it would still have to serve this purpose given the "new test framework" can't run these existing suites.

from paratest.

Excellent. Thanks for the feedback! At the moment, the latest tagged version of ParaTest is 0.6.1. Do you suggest keeping the minor version at 6 and just incrementing the patch number for subsequent fixes?

from paratest.

I don't really care about the numbers, but it sounds good to me.
What is most important - I think - is that if you use paratest on a "legacy" project and you run into a bug, you can still open a PR, get it merged and continue working.
Thats not

readonly

but you can nevertheless put all your time / efforts to the new project.

from paratest.

Awesome. I agree

from paratest.

I'm for using PHPUnit in native way (e.g create test runner class and then collect all data from it) instead of running the phpunit from command line. This would solve problems with new versions of PHPUnit and inability to use some options, that don't have analogs on command-line.

And we could use the PHPUnit own way of locating a tests and just split them into new test suites to be run in parallel.

Besides since PHPUnit 4.x the @sebastianbergmann is actually following SemVer so no more sudden BC breaks as before.

from paratest.

I tried something similar with "forktest". The goal was to create something like paratest but based on forks and to hook that deep into phpunit, that from the view of a listener, you couldn't say if it were executing the tests in parallel or not. Another issue I wanted to address was that the way paratest catches up the reports, it only preserves like the topmost exception trace (what is really annoying with phpunit-selenium where I use closures a lot and what matters is the previous exception).

Unfortunately, phpunit binds itself to Exceptions which have final methods for getMessage and getTrace etc. so you can't "fake" them or transport them from a fork to the parent process to input them into a TestListener. So I ended up deleting that "forktest" project. Without such dead weight, you could use an interface with getMessage and getTraceAsString without worrying wether the answer comes from an actual exception or they were serialized elsewhere and are just strings in a lightweight DTO. And that's what I would expect from a well designed, extendable testframework.

So if PHPUnit could be used, it would probably be the preferrable way. But I'm afraid it would end up with more workarounds than an application can have to keep trustworthy.

from paratest.

We can always ask @sebastianbergmann to change something within PHPUnit to make it more extensible.

from paratest.

Changing the interface for TestListeners is a major api change and will break almost every TestListener implementation that exists (I can't remember any project I worked on that didn't have any custom implementation). So, yes, we can ;) but how many of those changes will it take to make it work and how many of those changes will be accepted until one is rejected and we give up the mission?
Or am I too pessimistic?

from paratest.

Let's consider worst case: @sebastianbergmann never accepts our proposals (if any) to PHPUnit code.

Knowing that we can change paratest code in defensive fashion and be protected against PHPUnit changes. Actually we're doing that already in #84.

from paratest.

So the basic decision is:

somehow inventing the wheel again but have clean code
vs
using as much existing (phpunit) code as possible but add workaround after workaround (like in #84) to make up for phpunits not-so-clean code/architecture

right?

from paratest.

I think we end up with some kind of abstraction layer over PHPUnit and the sub-classes for PHPUnit 3.x (and below) and PHPUnit 4.x will be created. This way internally we'll speak to our own abstraction and if a problem arises with a particular PHPUnit then we update actual descendant classes in central places without affecting the application.

And actually in PHPUnit 4.x the changes to test runner weren't that major (just one ...risk.... method added).

from paratest.

The ParaTest is advertised on the SauceLabs as main tool which allows to run tests in parallel: https://saucelabs.com/php/se2/7

Maybe we can involve developers from SauceLabs to help us improve ParaTest?

from paratest.

Besides this is their (the SauceLabs) package that is using ParaTest: https://github.com/jlipps/sausage/

from paratest.

Jonathan made some contributions early on. I actually got to meet those folks and they are all pretty awesome. Fun fact: running concurrent tests against the sauce labs service is what inspired the creation of ParaTest :)

from paratest.

@julianseeger I didn't know of a better way to get ahold of you, but you have been on top of this project for a while now. Would you be interested in becoming a collaborator to help field some of the support based pull requests?

from paratest.

<----- collaborator ;)

So back to the topic:
To sum it up, it sounds to me like a split for two seperate problems or directions.
For the native implementation, probably even Sebastian Bergmann knows that there is a lot of ugly code and behavior in PHPUnit and Version 4 didn't change this. So a +1 for the native implementation.
On the other hand, many projects are using PHPUnit and even if this new testing framework would become the world-leading-solution, there would still remain a lot of phpunit-based projects to justify paratest for a couple of years.

So both may be done ... the native one + continued paratest ... right?

from paratest.

There are plenty of testing frameworks out there and the main difference IMO is the syntax. They do run tests in one way or another anyway. Creating another testing framework seems like too much for me.

What we can do is take approach that @sebastianbergmann had (at least I think this happened like this): separate large PHPUnit framework into smaller standalone modules (e.g. PHP_CodeCoverage, PHP_FileIterator, etc.). This way paratest will become standalone enough to be used along side with any testing framework out there considering that there will be an integration plugin for that testing framework.

To support existing PHPUnit users we'll of course will create PHPUnit integration plugin.

from paratest.

The way paratest works - and definitely has to work - is a massive overhead by completely wrapping the process, passing the arguments, parsing methods and annotations etc.. A native implementation would allow a deeper integration of the parallelism (probably too intrusive to get it into an existing framework). Maybe you could reuse existing components like PHP_CodeCoverage etc. but write a fresh and clean core - the most important thing and probably the most rotten thing at phpunit. One could even stick to a common syntax like $this->assertEquals(...) etc...
Actually this is what I hoped PHPUnit 4 to be :/

from paratest.

Hi,

This conversation is a little old since the last message dated Apr 6,
So I decided to ask you guys: probably, have you already switched to new implementation in separated repository? If yes, I would take a part in.

Otherwise: are we going to continue with current implementation of ParaTest (including new features)? I'm going to start using ParaTest on project I'm working on, and feel a lack of some options of phpunit cli, which should be proxied to paratest command (I'm ready to implement it, if we are going to stay with ParaTest "at least for years" :) ).

Thanks.

from paratest.

Hi @EuKov ,
speeking for myself, there is no new implementation and I'm going to stay with ParaTest for a while ;)

For your missing phpunit cli options, the easiest way is (if possible) to control phpunit with the configuration file. If that's not possible, I will be glad to help you with any pull request.

from paratest.

@julianseeger got your point about current implementation.
About cli: yeah, that's the best way, going to try.

Thank you!

from paratest.

Yeah, I started exploring a new test runner in Hack here: https://github.com/HackPack/HackUnit - I got a decent implementation running, but I am not sure I will be sticking with Hack - I am also not sure about moving forward with an xUnit style runner. I may still consider this in the future, and I will let you all know in case you are interested in working on it with me :)

from paratest.

@julianseeger, @dbaltas, @giorgiosironi and anyone who happens upon this, I just wanted to bring something to your attention in case you are interested. The past month, I have been working on a new testing tool for php 5.4+ called Peridot

I would love to get feedback, and though its not in the 1.0 release, concurrently running tests is something we would like to get in ASAP. We have already begun experimenting with a couple of methods to accomplish this, but as you know it is a tricky problem to solve in PHP :)

I would definitely appreciate a review of it, and if you don't mind getting it in front of some of your peers, that would be super helpful.

from paratest.

Syntax looks familiar with one Mocha uses.

from paratest.

Yes, the default DSL for Peridot is the BDD style. The framework in general is very inspired by Mocha, but it has definitely been tailored to PHPs strengths and style :)

from paratest.

You could write a paratest testrunner for Peridot ^^
At first glance it looks like you have wasted the "power" of php to use it's class concept but it could create a nice unification for the test style of projects with a lot of js code that's tested with mocha and the like.

from paratest.

There is actually a pretty strong object model behind the scenes :)

It just makes the language of testing less dependent on class definitions. You should checkout the concept of scopes in peridot. Its a great way of mixing objects and their behaviors in to Peridot tests.

And yes! I would love a paratest runner for Peridot! I'll have to look into it. My partner in crime and I have been discussing ways to have parallel execution built in to Peridot. We have a couple ideas we are toying with.

from paratest.

As peridot is growing and paratest can increase it's focus on phpunit, I think we can close this now.

from paratest.