tupilabs / tap4j Goto Github PK

View Code? Open in Web Editor NEW

31.0 6.0 16.0 11.25 MB

tap4j - A TAP implementation for Java

Home Page: https://tupilabs.github.io/tap4j/

License: MIT License

Java 98.72% Perl 1.28%

testformat testing xunit testanythingprotocol tap hacktoberfest

tap4j's Introduction

tap4j

tap4j - Simple implementation of the Test Anything Protocol (TAP) for Java.

tap4j was created in 2010 with the intention of implementing a producer and a consumer for TAP in Java.

1..2
not ok 1 - Something went wrong
  ---
  timestamp: 2012-12-10-20:31:01:021sss
  extensions: 
    files: 
      apache.err.log:
        File-Type: text/plain
        File-Content: <Base64-Encoded>
        File-Size: 685
        File-Name: apache.err.log
  ...
ok 2 # SKIP skipping due to previous errors

Test Anything Protocol

TAP is a test protocol that can be used to report test execution. It has been created with Perl 1, around 1988, and is the main format in Perl. A test harness executes tests and generates TAP, in the same way that a test harness can execute tests and generate other formats such as TestNG, JUnit or SubUnit.

TAP can be extended, what is not true with TestNG and JUnit. There is a specification for the protocol where, by using YAML(ish), you can include extra information about your tests, such as attachments.

Projects using tap4j

Build status

Authors and licensing

See AUTHORS or pom.xml for information regarding the authors and LICENSE or pom.xml for Licensing.

tap4j's People

Contributors

Stargazers

Watchers

Forkers

konobi accszumm ajbrowe ranford goozbach s2obcn jcflack japod digideskio fragonib benhyland altai-man ninjayoto xicanoxicano14 gaybro8777 auwong

tap4j's Issues

JUnit Test Runner doesn't output with Gradle

This is entirely Gradle's fault, but here's the upstream problem gradle/gradle#842

Fix apache-rat errors

They are ignored right now in the pom.xml, but it would be nice to add the license headers where necessary, or mark as ignored otherwise.

Junit TapListener method list bug

tap4j/tap4j-ext/src/main/java/org/tap4j/ext/junit/listener/TapListener.java

Line 126 in ae93e64

if (!testMethodsList.contains(description))

It's probably a bug to try to compare Description and JUnitTestData?
Blame backwards compatibility of containers when generics arrived for this one :/

Update/remove sourceforge docs

What makes this more difficult is that when a user googles Tap4j to search for more help, they find examples that reference the old API on sourceforge that is not compatible with the latest features.

See issue #28

Stream with subtests chopping off last yaml diagnostic

When parsing the TAP Stream below with a TAPConsumerFactory.makeTap13YamlConsumer, it seems that tap4j (using 4.0.4) does not hold onto the YAML diagnostic for anotherDummyTest. Note that it finds it correctly when there are no subtests, and the subtests themselves are handled correctly in the presence of the yaml, but it is not placing this value into the Map when there are subtests.

TAP version 13
1..2
  ---
    datetime: 20100101T000000
  ...
ok 1 - someDummyTest
  ---
    datetime: 20100101T000002
  ...
  1..5
  ok
  ok
  ok
  ok
  ok
ok 2 - anotherDummyTest
  ---
    datetime: 20100101T000005
  ...
  1..1
  ok

Unable to write tap output due folder structure not being created

As An exception is thrown when using a TestNG listener:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project Selenium-TestNG-tap4j: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test failed: There was an error in the forked process

[ERROR] org.tap4j.producer.ProducerException: Failed to dump TAP Stream: Selenium-TestNG-tap4j\target\surefire-reports\DemoSuite\test.LoginTest.tap (El sistema no puede encontrar la ruta especificada)
[ERROR] at org.tap4j.producer.TapProducer.dump(TapProducer.java:129)
[ERROR] at org.tap4j.ext.testng.listener.TapListener.generateTAPPerClass(TapListener.java:179)
[ERROR] at org.tap4j.ext.testng.listener.TapListenerClassYaml.onFinish(TapListenerClassYaml.java:42)

This happens as TapProducer is trying to write to the output folder, before surefire creating it. TapProducer should test or even creating it.

Running this project (https://github.com/witokondoria/Selenium-TestNG-tap4j) with "mvn clen test" can be used to debug this fact.

As a workaround, manually creating the folder structure (target\surefire-reports\DemoSuite) and not running the clean goal skips the error.

The same error happens using different listeners.

Migrate Maven site to GitHub pages

Move away the tap4j.org home page from the VPS to GitHub pages.

tap4j parser wrongly interprets indented test as yaml diagnostics

Hey there,

I've run across this now in several occasions where our test scripts (for unfortunate reasons is necessary) output some indented JSON into the STDOUT stream. Because of the indent and it looks like YAML, it gets interpreted as yamlish, even though it's not within '---' and '...'.

I've created a test case, verified it's failed and even got a small patch that fixes it. It may be necessary to handle it differently, so I did it up as a gist.

https://gist.github.com/3a3359333709963ccaca

Test compatibility of tap4j with Perl tools

From http://blogs.perl.org/users/confuseacat/2011/09/perl-testing-with-jenkinshudson-avoiding-some-pitfalls.html

There seems to be issues integrating the plug-in with TAP due to limitations in tap4j.

Update dependencies before the next release

Check Maven repository
Check OWASP dependency issues
Review changes.xml, reports, tabs/spaces, etc

Missing tests with skip_all

1..0 # SKIP No errors

Is not interpreted as a valid skipped test plan. Test::More produces this kind of output when skipping all tests.

A way to parse a stream of more than one test set

In my application I may have to parse from a stream of TAP produced by more than one test set. In looking around for prior work in that area, I found TAP::Stream and a message from someone working on Plan 9.

It would be nice to have a parse method that either returns a list of TestSet, or invokes a callback once for each TestSet when it comes out of the oven. It looks like TAP::Stream builds an über-testset with each set that it parsed as a subtest, though I've only glanced at it so I might not be right about that. In any case, even if tap4j's parser just returned a list or used a callback, it would be easy to build the rest of that on it.

The API as it stands is not so easy to build upon, because after seeing some input that probably means a new test set started, the caller can't easily get the preceding TestSet (an exception is thrown instead of returning it), and some of the next one has already been read from the stream, so unless the caller knows how much to push back, it isn't easy to start a new Parser to parse the next part.

It seems like the signs for one TestSet to come out of the oven would include:

seeing another TAP header
seeing another plan (if the prior one was beforeTestResult)
seeing anything after a plan that wasn't beforeTestResult (except maybe a footer)
seeing anything after a TAP footer

(By the way, what is a TAP footer? I see no mention in the spec here, is there a newer version I should be looking at?)

Those are the rules that pop into my head, though it might be better to try to match exactly what something else is doing, like TAP::Stream.

I ny my application, I might try to just put something in front of the parser that would try to match those events in the stream and feed the lines to a succession of parsers, but that has the code-duplication disadvantage of having to check for some of the exact same patterns the Parser already knows about.
and whatever I do there will probably be stupid about stuff in a YAML stream that looks like a TAP line, etc.) So it seems like really it should be in the parser and done right.

If the onPlan/onTestResult/etc. methods were virtual, a caller might be able to subclass the parser and get the wanted behavior by overriding some of them (it would start to look like SAX then), but they're currently private so that's not an option.

Allow TAP plan to be optional

ATM the plan is mandatory in TAP Streams, but some producers don't provide this sometimes.

regression in subtest parsing between 4.2.1 an 4.3

I'm trying to upgrade https://github.com/jenkinsci/tap-plugin to use the latest release of tap4j. When doing some I'm running into several failures with the plugins unit tests.

Specifically:

[ERROR] Failures: 
[ERROR] org.tap4j.plugin.flattentapfeature.TestFlattenTapResult.testStripFirstLevel(org.tap4j.plugin.flattentapfeature.TestFlattenTapResult)
[ERROR]   Run 1: TestFlattenTapResult.testStripFirstLevel:65->_test:192 expected:<5> but was:<3>
[ERROR]   Run 2: TestFlattenTapResult.testStripFirstLevel:65->_test:192 expected:<5> but was:<3>
[ERROR]   Run 3: TestFlattenTapResult.testStripFirstLevel:65->_test:192 expected:<5> but was:<3>
[ERROR]   Run 4: TestFlattenTapResult.testStripFirstLevel:65->_test:192 expected:<5> but was:<3>
[ERROR]   Run 5: TestFlattenTapResult.testStripFirstLevel:65->_test:192 expected:<5> but was:<3>
[INFO] 
[ERROR] org.tap4j.plugin.flattentapfeature.TestFlattenTapResult.testStripSecondLevel(org.tap4j.plugin.flattentapfeature.TestFlattenTapResult)
[ERROR]   Run 1: TestFlattenTapResult.testStripSecondLevel:89->_test:192 expected:<7> but was:<1>
[ERROR]   Run 2: TestFlattenTapResult.testStripSecondLevel:89->_test:192 expected:<7> but was:<1>
[ERROR]   Run 3: TestFlattenTapResult.testStripSecondLevel:89->_test:192 expected:<7> but was:<1>
[ERROR]   Run 4: TestFlattenTapResult.testStripSecondLevel:89->_test:192 expected:<7> but was:<1>
[ERROR]   Run 5: TestFlattenTapResult.testStripSecondLevel:89->_test:192 expected:<7> but was:<1>
[INFO] 
[ERROR] org.tap4j.plugin.flattentapfeature.TestFlattenTapResult.testStripSecondLevelIncompleteResult1(org.tap4j.plugin.flattentapfeature.TestFlattenTapResult)
[ERROR]   Run 1: TestFlattenTapResult.testStripSecondLevelIncompleteResult1:113->_test:192 expected:<7> but was:<1>
[ERROR]   Run 2: TestFlattenTapResult.testStripSecondLevelIncompleteResult1:113->_test:192 expected:<7> but was:<1>
[ERROR]   Run 3: TestFlattenTapResult.testStripSecondLevelIncompleteResult1:113->_test:192 expected:<7> but was:<1>
[ERROR]   Run 4: TestFlattenTapResult.testStripSecondLevelIncompleteResult1:113->_test:192 expected:<7> but was:<1>
[ERROR]   Run 5: TestFlattenTapResult.testStripSecondLevelIncompleteResult1:113->_test:192 expected:<7> but was:<1>
[INFO] 
[ERROR] org.tap4j.plugin.flattentapfeature.TestFlattenTapResult.testStripSecondLevelIncompleteResult2(org.tap4j.plugin.flattentapfeature.TestFlattenTapResult)
[ERROR]   Run 1: TestFlattenTapResult.testStripSecondLevelIncompleteResult2:134->_test:192 expected:<6> but was:<1>
[ERROR]   Run 2: TestFlattenTapResult.testStripSecondLevelIncompleteResult2:134->_test:192 expected:<6> but was:<1>
[ERROR]   Run 3: TestFlattenTapResult.testStripSecondLevelIncompleteResult2:134->_test:192 expected:<6> but was:<1>
[ERROR]   Run 4: TestFlattenTapResult.testStripSecondLevelIncompleteResult2:134->_test:192 expected:<6> but was:<1>
[ERROR]   Run 5: TestFlattenTapResult.testStripSecondLevelIncompleteResult2:134->_test:192 expected:<6> but was:<1>
[INFO] 
[ERROR] org.tap4j.plugin.stripsingleparent.TestStripSingleParent.testStripFirstLevel(org.tap4j.plugin.stripsingleparent.TestStripSingleParent)
[ERROR]   Run 1: TestStripSingleParent.testStripFirstLevel:56->_test:111 expected:<3> but was:<1>
[ERROR]   Run 2: TestStripSingleParent.testStripFirstLevel:56->_test:111 expected:<3> but was:<1>
[ERROR]   Run 3: TestStripSingleParent.testStripFirstLevel:56->_test:111 expected:<3> but was:<1>
[ERROR]   Run 4: TestStripSingleParent.testStripFirstLevel:56->_test:111 expected:<3> but was:<1>
[ERROR]   Run 5: TestStripSingleParent.testStripFirstLevel:56->_test:111 expected:<3> but was:<1>
[INFO] 
[ERROR] org.tap4j.plugin.stripsingleparent.TestStripSingleParent.testStripSecondLevel(org.tap4j.plugin.stripsingleparent.TestStripSingleParent)
[ERROR]   Run 1: TestStripSingleParent.testStripSecondLevel:72->_test:111 expected:<3> but was:<1>
[ERROR]   Run 2: TestStripSingleParent.testStripSecondLevel:72->_test:111 expected:<3> but was:<1>
[ERROR]   Run 3: TestStripSingleParent.testStripSecondLevel:72->_test:111 expected:<3> but was:<1>
[ERROR]   Run 4: TestStripSingleParent.testStripSecondLevel:72->_test:111 expected:<3> but was:<1>
[ERROR]   Run 5: TestStripSingleParent.testStripSecondLevel:72->_test:111 expected:<3> but was:<1>
[INFO] 
[INFO] 
[ERROR] Tests run: 48, Failures: 6, Errors: 0, Skipped: 0

I believe these are related to a change in how subtests are parsed between 4.2.1 and 4.3. I rigged up one of the plugin tests to print more output and looked at how TestFlattenTapResult#testStripFirstLevel behaved. This test takes a string representing TAP output like

        final String tap = "1..2\n" +
                "ok 1 1\n" +
                "  1..2\n" +
                "  ok 1 .1\n" +
                "  ok 2 .2\n" +
                "ok 2 2\n" +
                "  1..3\n" +
                "  ok 1 .1\n" +
                "  ok 2 .2\n" +
                "  ok 3 .3\n";

parses it using tap4j, and then attempts to "flatten" it. I looked at both what the test is evaluating, and the original test set before trying to flatten.

4.2.1

flattened

ok 1 1.1
ok 2 1.2
ok 3 2.1
ok 4 2.2
ok 5 2.3

original

1..2
ok 1 1
    1..2
    ok 1 .1
    ok 2 .2
ok 2 2
    1..3
    ok 1 .1
    ok 2 .2
    ok 3 .3

4.3

flattened

ok 1 1
ok 2 2.1
ok 3 2.2

original

1..2
ok 1 1
    1..2
    ok 1 .1
    ok 2 .2
ok 2 2

4.4.1

flattened

ok 1 1
ok 2 2.1
ok 3 2.2

original

1..2
ok 1 1
    1..2
    ok 1 .1
    ok 2 .2
ok 2 2

I'm not familiar with the details of TAP nor the "flattening" that the plugin does. But it appears that starting in 4.3 tap4j started ignoring everything after the start of the second subtest.

Rough cut of plugin test code:

diff --git a/pom.xml b/pom.xml
index f6f374b..9b3037b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -52,7 +52,7 @@
 		<jenkins.version>2.121.1</jenkins.version>
 		<java.level>8</java.level>
 		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-		<tap4j.version>4.2.1</tap4j.version>
+		<tap4j.version>4.4.1</tap4j.version>
 		<junit.plugin.version>1.6</junit.plugin.version>
 	</properties>
 
diff --git a/src/test/java/org/tap4j/plugin/flattentapfeature/TestFlattenTapResult.java b/src/test/java/org/tap4j/plugin/flattentapfeature/TestFlattenTapResult.java
index 09741a0..5267dfb 100644
--- a/src/test/java/org/tap4j/plugin/flattentapfeature/TestFlattenTapResult.java
+++ b/src/test/java/org/tap4j/plugin/flattentapfeature/TestFlattenTapResult.java
@@ -24,6 +24,11 @@ import org.tap4j.plugin.TapPublisher;
 import org.tap4j.plugin.TapResult;
 import org.tap4j.plugin.TapTestResultAction;
 
+import org.tap4j.representer.Tap13Representer;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+
+
 /**
  * Tests for flatten TAP result configuration option.
  *
@@ -179,7 +184,7 @@ public class TestFlattenTapResult {
                 false, // verbose
                 true,  // showOnlyFailures
                 false, // stripSingleParents
-                true, // flattenTapResult
+                false, //true, // flattenTapResult  tmp: for looking at flat vs original
                 false); //skipIfBuildNotOk
 
         project.getPublishersList().add(publisher);
@@ -189,6 +194,11 @@ public class TestFlattenTapResult {
         TapTestResultAction action = build.getAction(TapTestResultAction.class);
         TapResult testResult = action.getTapResult();
 
+        Tap13Representer repr = new Tap13Representer();
+        System.out.println("~~~~~~~~~~~~");
+        System.out.print(repr.representData(testResult.getTestSets().get(0).getTestSet()));
+        System.out.println("~~~~~~~~~~~~");
+
         assertEquals(expectedTotal, testResult.getTotal());
 
         final TestSet testSet = testResult.getTestSets().get(0).getTestSet();
@@ -200,7 +210,7 @@ public class TestFlattenTapResult {
 
             int expectedTestNumber = testIndex +1;
 
-            if (printDescriptions) {
+            if (printDescriptions || true) {
                 System.out.printf("%d: %s\n", testNumber, description);
             }

Add more space for the project sponsors in the project website

In order to attract more users and companies, it may be a good idea cite the sponsors of the project somewhere.

Subtest result string

Hello. Thanks for your work on this library.

As per this, a line that describes a subtest result can be either before or after every subtest, but if it's before, some separators(brackets, to be precise) are added.

But tap4j seem to assume first option always. For example,

ok 1 - Foo
    ok 1 - st a
    ok 2 - st b
    ok 3 - st c
    1..3
ok 2 - Some subtest
not ok 3 - Bar

must be parsed as test1("Foo"), test2("Some subtest", with three subtests("st a" et cetera), test3("Bar").
Instead, it parses like: test1("Foo", with three subtests), test2("Some subtest"), test3(Bar).

Trend graph does not display

Jenkins: 1.548
TAP Plugin: 1.17

When I open a job that has TAP results, the trend graph no longer appears. Checking the Jenkins Log shows the following NPE:

Jan 27, 2014 2:27:26 PM WARNING org.eclipse.jetty.util.log.JavaUtilLog warn

Error while serving http://glados.qa.sciencelogic.local:8080/job/EM7_CI_Unit_Tests/tapResults/graphMap
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:622)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
    at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:46)
    at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:42)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.NullPointerException
    at org.tap4j.plugin.TapResult.tally(TapResult.java:189)
    at org.tap4j.plugin.TapProjectAction.populateDataSetBuilder(TapProjectAction.java:212)
    at org.tap4j.plugin.TapProjectAction.doGraphMap(TapProjectAction.java:136)
    ... 80 more

org.tap4j.ext.testng.listener.*Yaml wont output YAMLish

With the project provided at https://github.com/witokondoria/Selenium-TestNG-tap4j and manually creating the required directories (target\surefire-reports\DemoSuite) as stated at #18, the produced output tap file doesnt contains YAMLish, even after using org.tap4j.ext.testng.listener.*Yaml listeners.

Fix build on Java > 8

Failing right now on Travis due to xjc errors. We may have to drop JMeter integration.

Old library in http://instanttap.appspot.com/

I am not sure if you are the maintainers of that site, but it seems the library used is old because it does not parse # SKIP

I have seen this problem has been solved; however, that site does not reflect this.

If you are the maintainers, it will be great to update the library: http://instanttap.appspot.com/

Diagnostics are added to all test cases, after one with diagnostics was found

From JENKINS-17859, diagnostics are added to all test cases, after one with diagnostics was found.

i.e. Given the TAP stream:

ok 42 - S::test_1
ok 43 - S::test_2
not ok 44 - Failure: test_3
—
message: 'Failed asserting that Y matches expected X.'
severity: fail
data:
got: Y
expected: X
...
ok 45 - S::test_4
ok 46 - S::test_5
ok 47 - S::test_6
ok 48 - S::test_7

After not ok 44, all other tests have the same YAML-ish diagnostics.

Error parsing test result description with # symbol in TAP

Tap4j is unable to parse correctly a test result description with a # symbol like a FQN of test method. This kind of test description is produced by tap4j-ext for JUNIT or TESTNG.

Example:

not ok 1 - org.myorg.mytestproject.MyTest#myTestMethod # SKIP Test was skipped

The regex used to parse test result header in TAP format is declared in org.tap4j.parser.Constants class:

static final String REGEX_TEST_RESULT = "\\s*(ok|not ok)\\s*(\\d*)\\s*([^#]*)?\\s*"
            + "(#\\s*(SKIP|skip|TODO|todo)\\s*([^#]+))?\\s*(#\\s*(.*))?";

and it doesn't take into account the presence of # symbol into test result description. I'm not sure if that it's allowed in TAP specification. However tap4j-ext is producing that type of test result description.

The final effect of this possible BUG is that the actual SKIP directive is ignored.

Please tell me if I can help to solve this issue in any way.

Thank you.

Optional test numbers are not working

Jakub reported the issue via e-mail with sample TAP streams. Apparently, there is some bug in the test result regex.

YAML parser isn't parsing correctly.

not ok 1 sdc-healthcheck exited cleanly

file:   child_process.js
line:   287
column: 7
stack:  
  - getCaller (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:387:17)
  - assert (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:17:16)
  - Function.equal (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:158:10)
  - Test._testAssert [as equal] (/root/sdc-system-tests/node_modules/tap/lib/tap-test.js:86:16)
  - /root/sdc-system-tests/tests/platform/001_base_build_sanity.js:11:11
  - ChildProcess.exithandler (child_process.js:287:7)
  - ChildProcess.emit (events.js:70:17)
  - maybeExit (child_process.js:361:16)
  - Process.onexit (child_process.js:397:5)
found:  
  name:    Error
  message: Command failed:
  type:    ~
  code:    1
wanted: ~
type:   
  found:  undefined
  wanted: undefined

...
not ok 2 service output is not blank

file:      child_process.js
line:      287
column:    7
stack:     
  - getCaller (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:387:17)
  - assert (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:17:16)
  - Function.inequal (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:192:10)
  - Test._testAssert (/root/sdc-system-tests/node_modules/tap/lib/tap-test.js:86:16)
  - /root/sdc-system-tests/tests/platform/001_base_build_sanity.js:12:11
  - ChildProcess.exithandler (child_process.js:287:7)
  - ChildProcess.emit (events.js:70:17)
  - maybeExit (child_process.js:361:16)
  - Process.onexit (child_process.js:397:5)
found:     
doNotWant:

...
not ok 3 no services showing as offline

file:      child_process.js
line:      287
column:    7
stack:     
  - getCaller (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:387:17)
  - assert (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:17:16)
  - inequivalent (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:210:10)
  - Function.dissimilar (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:238:10)
  - Test._testAssert (/root/sdc-system-tests/node_modules/tap/lib/tap-test.js:86:16)
  - /root/sdc-system-tests/tests/platform/001_base_build_sanity.js:13:11
  - ChildProcess.exithandler (child_process.js:287:7)
  - ChildProcess.emit (events.js:70:17)
  - maybeExit (child_process.js:361:16)
  - Process.onexit (child_process.js:397:5)
found:     
  "": 
doNotWant: /offline/g

...

This does not parse correctly under the Jenkins TAP plugin with:

Parsing TAP test result [/root/data/jenkins/workspace/sdc-system-tests/tap_output/platform__001_base_build_sanity.js.tap].

org.tap4j.parser.ParserException: Error parsing TAP Stream: Error parsing YAML [ file: child_process.js
line: 287
column: 7
stack:
- getCaller (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:387:17)
- assert (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:17:16)
- Function.equal (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:158:10)
- Test._testAssert as equal
- /root/sdc-system-tests/tests/platform/001_base_build_sanity.js:11:11
- ChildProcess.exithandler (child_process.js:287:7)
- ChildProcess.emit (events.js:70:17)
- maybeExit (child_process.js:361:16)
- Process.onexit (child_process.js:397:5)
found:
name: Error
message: Command failed:
type: ~
code: 1
wanted: ~
type:
found: undefined
wanted: undefined
]: null; mapping values are not allowed here
at org.tap4j.parser.Tap13Parser.parseFile(Tap13Parser.java:564)
at org.tap4j.plugin.TapRemoteCallable.invoke(TapRemoteCallable.java:139)
at org.tap4j.plugin.TapRemoteCallable.invoke(TapRemoteCallable.java:52)
at hudson.FilePath.act(FilePath.java:758)
at hudson.FilePath.act(FilePath.java:740)
at org.tap4j.plugin.TapPublisher.perform(TapPublisher.java:121)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:693)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:668)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:646)
at hudson.model.Build$RunnerImpl.post2(Build.java:162)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:615)
at hudson.model.Run.run(Run.java:1401)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:230)
Caused by: org.tap4j.parser.ParserException: Error parsing YAML [ file: child_process.js
line: 287
column: 7
stack:
- getCaller (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:387:17)
- assert (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:17:16)
- Function.equal (/root/sdc-system-tests/node_modules/tap/lib/tap-assert.js:158:10)
- Test._testAssert as equal
- /root/sdc-system-tests/tests/platform/001_base_build_sanity.js:11:11
- ChildProcess.exithandler (child_process.js:287:7)
- ChildProcess.emit (events.js:70:17)
- maybeExit (child_process.js:361:16)
- Process.onexit (child_process.js:397:5)
found:
name: Error
message: Command failed:
type: ~
code: 1
wanted: ~
type:
found: undefined
wanted: undefined
]: null; mapping values are not allowed here
at org.tap4j.parser.Tap13YamlParser.checkAndParseTapDiagnostic(Tap13YamlParser.java:280)
at org.tap4j.parser.Tap13YamlParser.parseLine(Tap13YamlParser.java:116)
at org.tap4j.parser.Tap13Parser.parseFile(Tap13Parser.java:557)
... 15 more
Caused by: mapping values are not allowed here
in "", line 16, column 30:
message: Command failed:
^

at org.yaml.snakeyaml.scanner.ScannerImpl.fetchValue(ScannerImpl.java:745)
at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:307)
at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:183)
at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:564)
at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:163)
at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:148)
at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:228)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:160)
at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:230)
at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:160)
at org.yaml.snakeyaml.composer.Composer.composeDocument(Composer.java:122)
at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:105)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:124)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:264)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:238)
at org.tap4j.parser.Tap13YamlParser.checkAndParseTapDiagnostic(Tap13YamlParser.java:275)
... 17 more

An empty plan, 1..0, causes parser error when it occurs right after another plan

Here is an excerpt from problematic TAP input:

# Subtest: test/exposed-harness.js
    # Subtest: main harness object is exposed
        ok 1 - tape.getHarness is a function
        ok 2 - should be equal
        1..2
    ok 1 - main harness object is exposed # time=15.769ms

    1..1
    # time=41.539ms

    1..0
    # tests 0
    # pass  0

    # ok

ok 13 - test/exposed-harness.js # time=264.253ms

The 1..0 plan above causes org.tap4j.parser.ParserException: Error parsing TAP Stream: Duplicated TAP Plan found

Subtest as first test with no plan

Further to issue #12 we have some tests that start with a subtest, but due to the a changeable nature of the thing we are testing we are not able to set a test plan up front so the first line in our TAP output is an indented subtest line.

An example of this TAP output is:

    ok 1 - subtest 1a
    ok 2 - subtest 1b
    1..2
ok 1 - Subtest 1
    ok 1 - subtest 2a
    ok 2 - subtest 2b
    1..2
ok 2 - Subtest 2
1..2
ok

a perl test that will produce this output is as follows

use Test::More;

subtest 'Subtest 1' => sub {
    ok(1,'subtest 1a');
    ok(1,'subtest 1b');
};
subtest 'Subtest 2' => sub {
    ok(1,'subtest 2a');
    ok(1,'subtest 2b');
};
done_testing();

Currently this produces an error such as:

org.tap4j.parser.ParserException: Error parsing TAP Stream: Invalid indentantion. Check your TAP Stream. Line: ok 1 - subtest 1a

A workaround for us is to stick an always true test before the first subtest but this is a bit of a PITA to be honest.

P.S. Also note this spelling of indentantion I think should probably be indentation

Add support to other attachments formats

SubUnit has support to another format (not Base64). That could give more flexibility to users using TAP with attachments. Especially if they are using Selenium and saving screenshots into the TAP Stream.

Parser has incorrect logic for ending yaml diagnostics

Somewhere near line 250 of Tap13Parser we see the following code:

if (tapLine.trim().equals("---")) {
    state.setCurrentlyInYaml(true);
    return;
} else if (tapLine.trim().equals("...")) {
    state.setCurrentlyInYaml(false);
    return;
}
// etc

However, it seems we shouldn't be trimming the the "..." to close the yaml block. Here is an example of a tap stream where this assumption is not valid:

TAP version 13
1..1
ok 1 - sometest
  ---
    datetime: 2013-11-14T15:42:54
    raw_output: |
      Running sometest
      ..........
      ..........
      ...
      Done sometest
      __________
  ...

Note that using pipe the "|" character to produce formatted text is fully supported by YAML, YAMLish, and SankeYAML. However, as you can see from the stream above tap4j trips on the "..." which is included in the raw_output field instead of the real one that closes the yaml block. Really, when we setCurrently in YAML we need to record the indentation at that time as the "YAML indentation" and probably store that in the memento. Then, instead of trimming the "..." we need to see if the line is exactly "..." with the right indentation to exit out of the yaml block. Actually, I think that the "..." is optional in YAMLish and the indentation is the most important piece of information to determine whether the yaml block is finished yet. Perhaps we should look for the change in indentation rather than the "..." and treat the "..." as a readability enhancer that is largely ignored.

Also, btw, it seems that the Jenkins TAP plugin does not respect the newlines that are present in the preformatted text (raw_output above) and it seems to assume the value can always be displayed in a single line. However, that is another issue.

Add CNAME

Wildcard redirect not working right now. Could be because it is missing the CNAME file in this repository.

Rewrite regex parsers and organize code (clean up, review, etc)

The current code dates from 2010. The initial version was written in a hurry and was ugly, but worked fine. Now, after pull requests, several bugs, and a couple of years being used in Jenkins TAP Plug-in, InstantTAP, extending TestNG and JUnit, with Selenium for attachments, behind the scenes of CJAN and even by some Python developers with Jython, it's time to review the code, clean up, enhance the parsers and prepare to cut 4.0 release.

The website has been down due to migration from GoDaddy (with redirection) to Namecheap (and now VPS hosting). The website will be restored eventually before the 4.0 release/announcement. Before that, please use tap4j.sourceforge.net.

Update the code for Java 8

To keep up to date with language updates, but we may also find performance improvements in the parser.

tap4j.org has been hacked

See screenshot… 😕

Error parsing TAP Stream: Invalid indentation

Getting this error when trying to parse TAP-results through Jenkins TAP-plugin (latest 2.1 version). TAP-file is generated via ember test --silent (which uses testem under the hood) and has 4 spaces indentation.

Is it an issue with tap4j or with file?

Using tap4j-4.2.1 release.

Error trace:

org.tap4j.parser.ParserException: Error parsing TAP Stream: Invalid indentation. Check your TAP Stream. Line:     ---
	at org.tap4j.parser.Tap13Parser.parseTapStream(Tap13Parser.java:230)
	at org.tap4j.parser.Tap13Parser.parseFile(Tap13Parser.java:193)
	at org.tap4j.plugin.TapParser.parse(TapParser.java:167)
	at org.tap4j.plugin.TapPublisher.loadResults(TapPublisher.java:540)
	at org.tap4j.plugin.TapPublisher.performImpl(TapPublisher.java:412)
	at org.tap4j.plugin.TapPublisher.perform(TapPublisher.java:371)

The TAP-file example is as follows:

skip 1 PhantomJS 2.1 - Acceptance | company settings: it shows current settings
    ---
        Log: |
            { type: 'warn',
              text: '\'DEPRECATION: Ember.K is deprecated in favor of defining a function inline. [deprecation id: ember-metal.ember-k] See http://emberjs.com/deprecations/v2.x#toc_code-ember-k-code for more details.\\n    http://localhost:7357/assets/vendor.js:17193:15\\n    raiseOnDeprecation@http://localhost:7357/assets/vendor.js:17112:17\\n    http://localhost:7357/assets/vendor.js:17193:15\\n    invoke@http://localhost:7357/assets/vendor.js:17209:21\\n    deprecate@http://localhost:7357/assets/vendor.js:17177:37\\n    deprecate@http://localhost:7357/assets/vendor.js:29019:42\\n    get@http://localhost:7357/assets/vendor.js:54526:28\\n    http://localhost:7357/assets/vendor.js:172502:30\\n    exports@http://localhost:7357/assets/vendor.js:140:37\\n    _reify@http://localhost:7357/assets/vendor.js:173:66\\n    reify@http://localhost:7357/assets/vendor.js:159:33\\n    exports@http://localhost:7357/assets/vendor.js:138:15\\n    _reify@http://localhost:7357/assets/vendor.js:173:66\\n    reify@http://localhost:7357/assets/vendor.js:159:33\\n    exports@http://localhost:7357/assets/vendor.js:138:15\\n    requireModule@http://localhost:7357/assets/vendor.js:32:25\\n    resolveInitializer@http://localhost:7357/assets/vendor.js:143623:25\\n    registerInitializers@http://localhost:7357/assets/vendor.js:143636:41\\n    http://localhost:7357/assets/vendor.js:143662:25\\n    http://localhost:7357/assets/metis.js:284:41\\n    exports@http://localhost:7357/assets/vendor.js:140:37\\n    _reify@http://localhost:7357/assets/vendor.js:173:66\\n    reify@http://localhost:7357/assets/vendor.js:159:33\\n    exports@http://localhost:7357/assets/vendor.js:138:15\\n    _reify@http://localhost:7357/assets/vendor.js:173:66\\n    reify@http://localhost:7357/assets/vendor.js:159:33\\n    exports@http://localhost:7357/assets/vendor.js:138:15\\n    requireModule@http://localhost:7357/assets/vendor.js:32:25\\n    http://localhost:7357/assets/test-support.js:5319:18\\n    require@http://localhost:7357/assets/test-support.js:5309:32\\n    loadModules@http://localhost:7357/assets/test-support.js:5301:23\\n    load@http://localhost:7357/assets/test-support.js:5252:37\\n    http://localhost:7357/assets/test-support.js:5173:22\'\n' }
    ...
skip 2 PhantomJS 2.1 - Acceptance | company settings: it allows to change settings
ok 3 PhantomJS 2.1 - JSCS - acceptance/company-settings-test.js: should pass jscs
<skipped>
ok 1087 PhantomJS 2.1 - JSCS - validators/unique-vat.js: should pass jscs
ok 1088 PhantomJS 2.1 - ESLint - validators/unique-vat.js: should pass ESLint

1..1088
# tests 1088
# pass  1078
# skip  10
# fail  0

# ok

tap4j trips over YAML/TAP output that is included in the diagnostics of its own YAML

If the diagnostics string that is produced contains more tap output with YAML in is diagnostics tap4j bails out, incorrectly deducing that the YAML in the diagnostics (which is at a different indentation level and should be ignored by the parser and just included as part of the diagnostic text) should be parsed and it falls over.

The following tap stream demonstrates this issue, with the diagnostics actually containing a tap stream that is not part of the "main" tap stream.

TAP version 13
1..1
ok 1 - sometest

datetime: 2013-11-14T15:42:54
raw_output: |
  Running sometest
  .....
  ================================================================================
  Verification failed.

      ---------------------
      Framework Diagnostic:
      ---------------------
      ContainsSubstring failed.
      --> The string must contain the substring.

      Actual String:
          TAP version 13
          1..2
            ---
              datetime: 2013-09-12T08:35:14
            ...
          ok 1 - testcases.SingleSilentTest
            ---
              datetime: 2013-09-12T08:35:15
              raw_output:
            ...
          not ok 2 - testcases.SimpleTestWithSharedFixture
            ---
              datetime: 2013-09-12T08:35:16
              raw_output:
            ...

      Expected Substring:

          ok 2 - testcases.SimpleTestWithSharedFixture
            ---
              datetime: 2013-09-12T08:35:16
            ...

  ================================================================================
  Done sometest
  __________

...

Incorrect issues link in the changes report

The changes report has broken links to issues in GitHub, as

http://tap4j.org/tap4j/changes-report.html

fix checkstyle errors

Ignored in pom.xml for now, but it would be nice to get these fixed later.

tap4j complains that a TAP stream has no plan... but it has I believe

From https://issues.jenkins-ci.org/browse/JENKINS-15791.

Apparently, the TAP stream has a TAP plan, but the parser complains that it doesn't.

Not able to find SKIP directive

In the following TAP stream

#cat /var/lib/jenkins/jobs/gh-mellanox-v1.8-PR/builds/137/tap-master-files/cov_stat.tap
not ok - coverity detected 1411 failures in all # SKIP
ok - coverity found no issues for oshmem
ok - coverity found no issues for yalla
ok - coverity found no issues for mxm
ok - coverity found no issues for fca
ok - coverity found no issues for hcoll

The parser is not able to find the SKIP directive.

Add Travis CI

Create examples for working with extensions (TestNG, JUnit, etc)

It would be very useful to have a simple Test Directory that includes a few JUnit tests along with a TestSuite consumer that can generate TAP output.

See issue #28

JUnit TapListeners always report tests as passed on version 4.0.6

Trying to use different JUnit TapListeners(with or without Yaml) and getting the same results all the time: failed tests display as "ok"

Make GitHub issues official bug tracking for the project

At moment the SF.net project page lets users post issues there. This can cause a lot of trouble for users looking for open issues before using this project.

Unify changes report files

As per title, there's more than one changes.xml, and the changes are all over the place. We need a single file (with or without a multi-module, the right is still a single file).

Fix website (tap4j.org)

After migrating from GoDaddy to Namecheap, the redirection that was in GoDaddy is obviously not working anymore.

tap4j 4.4 missing from maven central

It appears that for 4.4 only the parent was uploaded.

https://search.maven.org/search?q=tap4j

Move from multi-module to single-module project

It started as a multi-module for integration tests with Perl and integration with other test formats. But now this library is mainly used for the Jenkins plugin or other tools that need a Java TAP producer/consumer.

The Maven build plugins, and the release related plugins, sometimes behave erratic with multi-modules. Plus the changes.xml is spread in two modules and needs to be unified later (no idea how that happened, but mea culpa probably).

So planning a 5.x release line with this change, unless there are objections about dropping the other modules from users/devs.

Can you provide an example for newbies to tap4j

Hi,

I am coming to tap4j after having used a large number of other languages , ( with Perl being my one introduction to TAP) so my problems might be related to my being somewhat beginner at using Maven to manage release / builds. I want to use tap4j, but I can't find any simple examples that will allow me to generate TAP code from a few JUnit test cases. What makes this more difficult is that when a user googles Tap4j to search for more help, they find examples that reference the old API on sourceforge that is not compatible with the latest features.

In addition, I have tried running the example test cases under tap4j-ext/src/test/java/org/tap4j/ext/junit/ but have been unsuccessful at generating TAP . It would be very useful to have a simple Test Directory that includes a few JUnit tests along with a TestSuite consumer that can generate TAP output.

Lastly, the current GitHub image for this project seems to break when performing a
mvn package

I get the following
Running org.tap4j.producer.TestTapProducerFactory
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 sec <<< FAILURE!

I had to hack fix this by essentially disabling the TestTapProducerFactory unit test.

Many thanks,

subtest as first test causes ParserException (Duplicated TAP plan found)

TAP created from perl's Test::More with one subTest.
Parsed using Parser parser = new Tap13Parser(true);

subTest is second - works

1..2
ok 1 - First test
    1..2
    ok 1 - This is a subtest
    ok 2 - So is this
ok 2 - Second: An example subtest

subTest is first - causes an exception

org.tap4j.parser.ParserException: Error parsing TAP Stream: Duplicated TAP Plan found.
at org.tap4j.parser.Tap13Parser.parse(Tap13Parser.java:204)

1..2
    1..2
    ok 1 - This is a subtest
    ok 2 - So is this
ok 1 - First: An example subtest
ok 2 - Second test

...maybe perl integration tests in the other direction than the current ones make sense, i.e., creating TAP by perl's Test::More for input to the parser.

Subtests order/format change as of v4.3

I think I'm facing an issue that might be related to #56. With v4.2.1, TapProducer#dump(TestSet) resulted in something like:

1..2
ok 1 Some test case 0
    1..3
    ok 1 Some action
    ok 2 Some action
    ok 3 Some action
ok 2 Some test case 1
    1..3
    ok 1 Some action
    ok 2 Some action
    ok 3 Some action

Whereas as of v4.3, the result is:

1..2
    1..3
    ok 1 Some action
    ok 2 Some action
    ok 3 Some action
ok 1 Some test case 0
    1..3
    ok 1 Some action
    ok 2 Some action
    ok 3 Some action
ok 2 Some test case 1

It looks like this is caused by a34d494 as this change is also reflected in the test class TestIssue3504508.

I'm not that familiar with TAP in general, but I assume this is somewhat related to TestAnything/Specification#2?

I just wonder which order/format is "correct"?

Fix maven site issues before next release

See https://stackoverflow.com/questions/51091539/maven-site-plugins-3-3-java-lang-classnotfoundexception-org-apache-maven-doxia

java.lang.NoClassDefFoundError: org/apache/maven/doxia/siterenderer/DocumentContent

tupilabs / tap4j Goto Github PK

tap4j's Introduction

tap4j

Test Anything Protocol

Projects using tap4j

Build status

Authors and licensing

tap4j's People

Contributors

Stargazers

Watchers

Forkers

tap4j's Issues

4.2.1

flattened

original

4.3

flattened

original

4.4.1

flattened

original

Recommend Projects

Recommend Topics

Recommend Org