might miss issues in some contexts

This way integrating into viewers is straightforward.

Have a -v mode to print lots of info rather than summaries (OK/not OK, reason, name)

This will be required if I'm to make a cpantesters equivalent

Will be needed for coordination and queuing, along with viewing

I'd toyed with the idea of doing Test::Manual, but that's too complicated for most testing orgs. Better to leave them to their spreadsheets in version control, and have a quickie program for uploading results manually. That could then be tied to a frontend like in traditional TMSes.

Some stuff like 'tests with high outcome variance' is very hard to do in timelion/kibana. I'd like a tool to do it for me.

E.G. Indexer::DzilDist, MakeMakerDist, etc

Basically just rename the index to what the distribution is, as defined by the makefile, dist.ini, etc.

E.G. a test was just bad, a duplicate, etc. etc

for testd

This would allow us to say, put the distname into the data, so we can do something like cpantesters.

needs to be done

2 assertions commented out

Gotta whack jobs in the workers that get stuck to ensure the work continues.

Any jobs that failed to run to completion will be returned to the queue, and this worker will explicitly reject that job if encountered again

sort of a consequence of the plan_running branch

This should facilitate the situation where you just gotta have (TM) that one piece of hardware to make things work. Be sure to note this in the POD as the rationale for the feature.

Allow this to be pluggable like everything else.

And the attendant tool to put documents in there. Configuration should be limited to a URL prefix for the tracker.

Also have a tool to list active issues at particular versions (and platforms optionally). Assume if 1 issue in contiguous block of anomalies for test, it's the same thing ( will have to back-search versions for each test).

Export them after I get it all set up again.

Will need it's own versioner framework, I intend to use git.

noticed this when unit testing, it's due to not setting steps or steps_planned in the manual testing

I'm not a big fan of the latter due to undeclared deps on the system specific drivers.

Much like %mark_status=retest works in testrail::API

Should allow git based workflow going forward

no need to do this for each file

Needs to be pluggable like the rest to allow simple configuration/extension.

So, if I want to execute a plan, I want to be able to just provide the name, and optionally version and platform(s).

In the event, say, only a name is passed, we'll try and run every plan with that name, on every version and platform available. Passing version would restrict to all plans of the needed version, and we could further drill down to just one plan by having platform(s) specified.

In the future, we should hook this into the provisioner subclasses and just SKIP if we can't figure out how to provision a test plan.

A Test::DesiredCapabilities module which is then consulted by A::P::E (just run begin blocks) to eject tests which don't fit the current test plan. Should be implemented in bin/testplan to loudly reject tests which simply won't work under this paradigm.

I've been calling it A::P::E for a while, may as well make it official so we can have a cute little logo, goes with the whole troglodyne theme too

e.g.

ape plan ...
ape test ...

Need a plan and assignee index.

(plan name + SUT version) must be unique. Accepts a list of test cases that must be run, and platforms to run on.
Must understand 'platform classes', e.g. platforms that multiply and create matrices. For example:
OSes and Perl Versions.

assignee documents must provide the plan name + SUT version and the assignee. Provide a tool to assign/unassign tests on command line (accept globs, multi args, balk on already assigned), and integrate it into prove to auto assign & refuse to run when running tests assigned to others.

Add a tool to run everything assigned to me in a plan.

Create a pluggable tool to automatically create plans in reaction to events, in case you are crazy enough to not use CI for your build toolchain (bonus points).

Finally, add a tool to run forever and crank on everything assigned to it; this way we can setup a distributed test cluster.

'elapsed' => 0, everywhere in steps

perhaps an --inherit option

Should prove interesting.

Should be e-z part of plan callback

Should be straightforward. Allow any/not assignee running too.

would be a good default, and handy

This way integrators can just blithely -PElasticsearch in their testsuite (e.g. their makefile, dzil plugin etc)

The current defines are too crude to allow proper usage of the fields as keywords.

Need update to POD, and investigation of why it takes so long

Work around this BS being broken in prove in our runner module; make compatible with the YATH stuff

Seems like a sane feature request, would cut down on end-user configuration to be sure. Makes a lot of sense in places like the perl platformer.

Have a Devel::Cover::Report::Elasticsearch

we should have all this available anyways. Make sure to let the user know this is going on.

Add it as say, an --inherit option.

nobody likes losing work. Prompt the user on conflicts.

Appears as though lucene queries actually do what you want, so we should re-factor after plan_running merges.

Use to provision environments for the system under test.

Allow passing multiple times on commandline e.g.

-Pprovisioner=OpenStack,provision=CentOS6,provision=CentOS 7 32-bit,setup=MySetupClass

It seems to me the most simple thing for me to develop here would be an Openstack provisioner.
I would want some way to map provision environment to base image ID, e.g this in config:

[ProvisionAliases]
CentOS 6 64-bit=abdef-blah-blah-blah-qqnoob

Also allow direct passing of image id.

Default provisioner is to just use the local system.

Default setup script would be to tar up the dist, SCP the dist tarball over and install it w/ CPAN.
Put in namespace App::Prove::Elasticsearch::Provisioner::Setup::Default

This would naturally involve complication when passing multiple provision platforms, as we would basically have to run prove over again with the same args per platform if we don't want to do wacky things to the -j mechanism.

Otherwise we'll need to inject a prove executor shim to just read logs from running prove on the tests in the remote system, unless that is our tests explicitly intend to test from here and SSH in.

To accomodate that we should have a App::Prove::Elasticsearch::Executor set of modules too (basically remote and default).

gotta speed this up a bit if possible

Needs to be even faster. Use MCE::Shared on the caches.