Subtest result string about tap4j HOT 6 CLOSED

Hi @Altai-man

Thanks for reporting the issue.

From what I remember, there was never a formal agreement on subtests. It appeared around the time TAP 13 was being submitted for the IETF as a draft. Some tools implemented it in a way, and others in another different way.

If memory serves me right, there used to be another older page in testanything.org, but it has been archived somewhere after the new site.

Take a look at this Perl doc: https://perldoc.perl.org/Test/More.html

Test::More is much older than node-tap, and probably used the old discussion/spec to implement it. In their example, if I understand correctly, the subtest follows whichever TAP Test Result that precedes it.

It is the first time I see brackets used in TAP. Before I had seen comments with # being used as extra information about tests (metadata), as well as the YAMLish syntax. But anything else - including the brackets - would be discarded (which made the TAP syntax a bit trickier to be parsed due to ambiguity).

So not too sure how to proceed. I would like to adapt tap4j to support what node-tap expects, but I think we would have to wait for TestAnything/testanything.github.io#36 to be completed (TAP 14). Hopefully there will be standardization on subtests there.

In the meantime, I would prefer to follow what Perl, C, Shell, PHP, MySQL support (if I remember well the producer/consumers I used in these langs were based on Perl). Hope that makes sense.

from tap4j.

Thanks for you answer, @kinow!

To be honest, I actually want to parse TAP generated by Perl 6, so I took the node link just because it was close enough to refer to.

But, hmm, I may terribly misunderstood something right now, but on Test::More page I see this:

  pass("First test");
  subtest 'An example subtest' => sub {
      plan tests => 2;
      pass("This is a subtest");
      pass("So is this");
  };
  pass("Third test");

generates:

  ok 1 - First test
      # Subtest: An example subtest
      1..2
      ok 1 - This is a subtest
      ok 2 - So is this
  ok 2 - An example subtest
  ok 3 - Third test

I.e. first ok - is a separate test, then subtest, and overall subtest result is a second ok, not the opposite. So each subtest result is coming before(!) overall subtest result, not after.

I think there may be some odd misunderstanding between us and it will be great if we could clear up this one.

By the way, Perl 6 suite(that implements what Test::More does) generates this kind of code(first each subtest, then overall subtest result) too.

from tap4j.

Oh, I probably misread it. Let me have another look at the Perl example 👍

I need to find the old pages I used while writing that part... memory tells me I had to link the subtests with the previous test result. But I rarely trust my memory.

And thanks for pointing it out so politely my mistake :-)

from tap4j.

Oh well, the link that I sent before was saying the same as you pointed out. My mind was clearly showing me what I wanted :)

I've read some links from Perl, and everything points to the same way; i.e. subtests before. And it even makes sense for me now, as TAP is supposed to be able to be parsed in a Stream, continuously. So the indentation alone should be able to tell the parser it's a subtest... and as the number of test results may be omitted if I recall correctly, the subtests going before would be simpler to parse.

I really can't recall why I implemented it the other way. Will have to have another look later at the git commit history, and try again to find any links to old pages.

But said that, I think it's important to support the Perl behaviour, which would mean supporting node-tap as well :)

Sorry for the confusion @Altai-man
B

from tap4j.

No problem!

I am happy as long as initial Perl(and so C, MySQL et cetera) behavior will be parsed too at least as a feature with an additional parameter or something like that. 👍

from tap4j.

Closed with the related PR.

from tap4j.