herdphp / docker-phpqa Goto Github PK

phpqa generate /path/to/phpt/dir -f %function_name% -bug %#bug_number%

--TEST--
%#bug_number% 
--CREDITS--
--FILE--
<?php
# body
?>
--XFAIL--
# the reason
--EXPECT--

In the way it is, we need to put new tests in the folder phpt of this repository.

I think it could be good if we had some explanation of how to use this project to facilitate the flow that a contributor need to follow in order to send pull requests to https://github.com/php/php-src:

• git fork https://github.com/php/php-src to https://github.com//php-src
• git clone https://github.com//php-src in my machine
• git checkout the desired upstream branch
• git branch (that group of tests) and checkout this local branch
• create tests, tests the tests ;-) , and git commit
• git push to https://github.com//php-src
• open github pull from https://github.com//php-src 's branch (that group of tests) to https://github.com/php/php-src
• come back to my local repository, git checkout the desired upstream branch, rinse and repeat

@brunoric how do you suggest us to put everything together?

Best!

I'd like to request that gcov be compiled to each of the versions in the docker container , so that users can validate their tests are providing coverage for the lines they're trying to cover before submitting a pull request

Put some effort to shrink the images, either using the Alpine version of them or any other strategy.

Integrate this repository with CircleCi in order to have the docker images builds tested.

When running the suite if the test fails some debug files are created to help to understand what went wrong.

If we run multiple versions the latest is the only one that is saved. This is not optimal.

The ideal should be to generate folders or prefixes for the version that is running. Something like:
phpt/56/
phpt/71/
or
phpt/56_
phpt/71_

When running the phpqa command I get the following error before the expected output:

Error: Unrecognized command .

We should review how we display the help message. It does not contain all possible options (for example it is missing help) and it is also too long. Maybe creating the option phpqa help to output the possible commands and also making possible something like phpqa help run to output the help for the command run.

This api needs to be evaluated and discussed.

I cloned the phptestfest-php-src and executed some tests with the phpqa tool.
When I execute the test from the security folder almost all tests are fail.

Here is the log from my console.
I am not sure why this happens, but it only occurs within the docker env. If I execute the test on a Host machine with make there is only one failure.

In order to test all things, we need to add php extensions, like gd, to build and run tests

Since we are releasing frequently we need to implement a version command to easily inform to the user the version that it has installed. This will help when users are submitting issues.

I have a custom extension I wanted to test. I wonder is it in scope for this project to test things not available in PHP? I derived a docker file that would do it, if you wanna see it here:

https://github.com/Bit-Wasp/secp256k1-php/tree/master/travis

I'd be curious what you guys think! Got this working with gcov/lcov and then submit coverage to https://codecov.io/gh/Bit-Wasp/secp256k1-php. It'd be great to get that test harness in all extension repos

It is a known issue on CircleCI (https://circleci.com/docs/1.0/docker-btrfs-error/), and has an easy solution:

Normally, this is a red-herring and doesn’t affect your builds, so you can simply ignore it. You can add the --rm=false flag to docker build to avoid seeing this issue.

Should we adopt the solution or ignore the error?

When I try to install docker-phpqa on windows the following error occurs:

$ curl -s https://raw.githubusercontent.com/herdphp/docker-phpqa/master/bin/ins
taller.sh | bash
PHPQA installer started ...
Step 1: clonning the repository to /c/Users/LuizFelipe/.phpqa/bin/phpqa.sh
Cloning into 'C:/Users/LuizFelipe/.phpqa'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Step 2: linking phpqa.sh to /usr/local/bin/phpqa ...
ln: failed to create symbolic link ‘/usr/local/bin/phpqa’: No such file or directory
Step 3: applying exec permissions ...
chmod: cannot access ‘/usr/local/bin/phpqa’: No such file or directory
Success! The phpqa command was added to your /usr/local/bin folder and can be used globally.

Note: I already have Docker installed.

PHP is about to use only GPG keys verification and distrust MD5. We should remove these checks from the image building.

Reference: http://news.php.net/php.internals/99769

As mentioned by @benharold in a pull request at the phpcommunity/phptestfest.org, when the generate command doesn't receive any target directory it throws a misleading error.

phpqa run suite
Error: You need to provide a phpt or a directory with phpt files to be tested or pass suite as first parameter to run the full test suite.

Improve documentation, adding a session about how to write PHPT and how to send the created tests to PHP QA.

When tests fails the suite generates a .php file of the code to be executed. Would be great to be able to debug such file and improve bug reports or test fixes.