herdphp / docker-phpqa Goto Github PK

Docker tools to easily create and run tests for the PHP-SRC

License: Other

Shell 73.85% PHP 26.15%

php php7 php71 php-test phpt php-src php56 php-library testfest

docker-phpqa's Introduction

NOT MAINTANED ANYMORE

The development of docker-phpqa was dropped in favor of phptt which is actively maintained. Thanks to everyone that contributed with this small tool. =)

docker-phpqa

Docker tools to easily create and run tests for PHP-SRC. The intent of this library is to help anyone to create PHPT tests for PHP language engine and its core extensions.

How to get it?

You must have the latest version of docker installed on your machine.
Run our installer:

curl -s https://raw.githubusercontent.com/herdphp/docker-phpqa/master/bin/installer.sh | bash

Start PHPTesting!!!

How to use it?

To generate new PHPT test files

The generate-phpt tool wrapped by docker-phpqa is a developer tool that lives inside of PHP-SRC repository. It is meant to bootstrap PHPT creation, it has a very simple usage.

Usage:
phpqa generate [PHPT_DIR] -f <function_name> |-c <class_name> -m <method_name> -b|e|v [-s skipif:ini:clean:done] [-k win|notwin|64b|not64b] [-x ext]

Where:
-f function_name ................. Name of PHP function, eg cos
-c class name .................... Name of class, eg DOMDocument
-m method name ................... Name of method, eg createAttribute
-b ............................... Generate basic tests
-e ............................... Generate error tests
-v ............................... Generate variation tests
-s sections....................... Create optional sections, colon separated list
-k skipif key..................... Skipif option, only used if -s skipif is used.
-x extension...................... Skipif option, specify extension to check for
-h ............................... Print this message

If you want to see the help you can just use:

phpqa generate -h

If you want to create a basic PHPT test for the ucfirst function:

phpqa generate -f ucfirst -b

This will create a template PHPT file for the function ucfirst inside of your PHPT default folder. If you want to choose the PHPT directory where the files are generated you can pass it as the first argument of the generate command:

phpqa generate /path/to/phpt/dir -f ucfirst -b

To run your created tests against multiple PHP versions

Write a PHPT test in any folder;
Run the phpqa command passing as arguments the filepath (relative or full) and the PHP version to run the test (master, 72, 71, 70, 56, 55, all):

# without a version the test will run against the latest PHP stable version (PHP 7.1) codebase
phpqa run phpt/examples/closure_061.phpt
# with a version the test will run against the specified version
phpqa run phpt/examples/closure_061.phpt 56
# if you want you can pass `all` as second argument and the test will run against all available versions
phpqa run phpt/examples/closure_061.phpt all

To generate the coverage report

# without a version the coverage report will be generated for the latest PHP stable version (PHP 7.1) codebase
phpqa gcov
# with a version the coverage report will be generated for the specified version
phpqa gcov 70

Stay updated for bug fixes and new features

To update your docker-phpqa tool and images just call the update command:

phpqa update

Usage help

docker-phpqa has a small set of commands and a quick way to get help about its usage is either type the phpqa command without any arguments or calling phpqa help command.

Wiki

For tutorials, cookbooks and more information about how to use phpqa please take a look at our wiki.

PHP versions available

PHP 5.5, 5.6, 7.0, 7.1, 7.2 (RC4) and the master branch from PHP-SRC repository are fully available and supported;

Contributing

If you notice any bug or typo or if you want to improve this tool, please feel free to create an issue and/or open a pull request.

PHPTestFest2017

This repository intent to facilitate participation and contribution in the PHP Test Festival. More information about it:

Credits

The tests in the examples folder are part of the PHP interpreter codebase distributed under PHP License 3.01.
The Dockerfile in this repository is inspired and based on the official Dockerfile template and images present in the PHP Docker Hub repository also available at the Docker Library Github repository. The scripts docker-php-* are also from the same repository. These source code is distributed under MIT license.

LICENSE

This work is licensed under the terms of the MIT license.

docker-phpqa's People

Contributors

Stargazers

Watchers

Forkers

jpjoao rogeriopradoj ivanrosolen brunoric cwt137 marabesi barryosull dstockto galvao korvinszanto denis-rendler gabrieldagama royopa jeffherson stefanotorresi p365labs afk11

docker-phpqa's Issues

Create a version command

Since we are releasing frequently we need to implement a version command to easily inform to the user the version that it has installed. This will help when users are submitting issues.

Have a debug tool (xdebug or phpdbg) to allow debug of failed tests

When tests fails the suite generates a .php file of the code to be executed. Would be great to be able to debug such file and improve bug reports or test fixes.

Make it suitable for contribute to https://github.com/php/php-src sending pull requests to there

In the way it is, we need to put new tests in the folder phpt of this repository.

I think it could be good if we had some explanation of how to use this project to facilitate the flow that a contributor need to follow in order to send pull requests to https://github.com/php/php-src:

git fork https://github.com/php/php-src to https://github.com//php-src
git clone https://github.com//php-src in my machine
git checkout the desired upstream branch
git branch (that group of tests) and checkout this local branch
create tests, tests the tests ;-) , and git commit
git push to https://github.com//php-src
open github pull from https://github.com//php-src 's branch (that group of tests) to https://github.com/php/php-src
come back to my local repository, git checkout the desired upstream branch, rinse and repeat

@brunoric how do you suggest us to put everything together?

Best!

Compiling in gcov

I'd like to request that gcov be compiled to each of the versions in the docker container , so that users can validate their tests are providing coverage for the lines they're trying to cover before submitting a pull request

Integrate with CircleCI

Integrate this repository with CircleCi in order to have the docker images builds tested.

Misleading error is thrown when phpqa generate command don't receive a target directory

As mentioned by @benharold in a pull request at the phpcommunity/phptestfest.org, when the generate command doesn't receive any target directory it throws a misleading error.

Add support for PHP 5.5 and 7.2 (alpha)

Enable php extensions on docker

In order to test all things, we need to add php extensions, like gd, to build and run tests

Support for custom extensions?

I have a custom extension I wanted to test. I wonder is it in scope for this project to test things not available in PHP? I derived a docker file that would do it, if you wanna see it here:

https://github.com/Bit-Wasp/secp256k1-php/tree/master/travis

I'd be curious what you guys think! Got this working with gcov/lcov and then submit coverage to https://codecov.io/gh/Bit-Wasp/secp256k1-php. It'd be great to get that test harness in all extension repos

Improve output of phpqa [help]

We should review how we display the help message. It does not contain all possible options (for example it is missing help) and it is also too long. Maybe creating the option phpqa help to output the possible commands and also making possible something like phpqa help run to output the help for the command run.

This api needs to be evaluated and discussed.

Different results with make and phpqa

I cloned the phptestfest-php-src and executed some tests with the phpqa tool.
When I execute the test from the security folder almost all tests are fail.

Here is the log from my console.
I am not sure why this happens, but it only occurs within the docker env. If I execute the test on a Host machine with make there is only one failure.

Add documentation about PHPT

Improve documentation, adding a session about how to write PHPT and how to send the created tests to PHP QA.

CircleCI error "Error deleting container: Error response from daemon: Driver btrfs failed to remove root filesystem"

It is a known issue on CircleCI (https://circleci.com/docs/1.0/docker-btrfs-error/), and has an easy solution:

Normally, this is a red-herring and doesn’t affect your builds, so you can simply ignore it. You can add the --rm=false flag to docker build to avoid seeing this issue.

Should we adopt the solution or ignore the error?

phpqa run suite not working

phpqa run suite
Error: You need to provide a phpt or a directory with phpt files to be tested or pass suite as first parameter to run the full test suite.

Remove MD5 checks

PHP is about to use only GPG keys verification and distrust MD5. We should remove these checks from the image building.

Reference: http://news.php.net/php.internals/99769

Problem when I try to install docker-phpqa on Windows

When I try to install docker-phpqa on windows the following error occurs:

$ curl -s https://raw.githubusercontent.com/herdphp/docker-phpqa/master/bin/ins
taller.sh | bash
PHPQA installer started ...
Step 1: clonning the repository to /c/Users/LuizFelipe/.phpqa/bin/phpqa.sh
Cloning into 'C:/Users/LuizFelipe/.phpqa'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Step 2: linking phpqa.sh to /usr/local/bin/phpqa ...
ln: failed to create symbolic link ‘/usr/local/bin/phpqa’: No such file or directory
Step 3: applying exec permissions ...
chmod: cannot access ‘/usr/local/bin/phpqa’: No such file or directory
Success! The phpqa command was added to your /usr/local/bin folder and can be used globally.

Note: I already have Docker installed.

When running phpqa I get "Error: Unrecognized command ."

When running the phpqa command I get the following error before the expected output:

Error: Unrecognized command .

Shrink images

Put some effort to shrink the images, either using the Alpine version of them or any other strategy.

Output of test debug info should be per version to prevent override of files

When running the suite if the test fails some debug files are created to help to understand what went wrong.

If we run multiple versions the latest is the only one that is saved. This is not optimal.

The ideal should be to generate folders or prefixes for the version that is running. Something like:
phpt/56/
phpt/71/
or
phpt/56_
phpt/71_

Create preset for bug tests

phpqa generate /path/to/phpt/dir -f %function_name% -bug %#bug_number%

--TEST--
%#bug_number% 
--CREDITS--
--FILE--
<?php
# body
?>
--XFAIL--
# the reason
--EXPECT--