Newlines stops consuming tag descriptions about phpdoc-parser HOT 3 OPEN

Have you tried setting textBetweenTagsBelongsToDescription in PhpDocParser to true?

These options are similar to PHPStan's bleeding edge. Right now they're set to false for BC reasons but they will be removed in 2.0 and code will behave the new modern way (as if these options were set to true).

Also check out options in TypeParser and ConstExprParser.

from phpdoc-parser.

Yes I did check those settings, those resulted in the same behavior.
I didn't check the other parsers but I assume they have nothing Todo with the issue we are facing as it is reproduced with just the description. In fact the text processing stops after an empty line in all cases I checked.

from phpdoc-parser.

I checked again... since I was using the parser just for a single tag this caused issues. So I do have a workaround that works for us right now. As we can simply consume all tokens from the lexer.

Which brings me to another issue, which might be one you also are facing when reconstructing docblocks. When I have a docblock line like this:

/**
 * @param string $myParam this is a description of the param
 *
 *                     And this line also belongs to $myParam.
 * @param array $other this is a new param.
 */
*/

The token on line 2 contains: \n with type Lexer::TOKEN_PHPDOC_EOL but as the parser sees this as a EOL it will remove the trailing whitespace when building the description.

We fixed this by adding some post processing on the tokens:

    private function tokenizeLine(string $tagLine): TokenIterator
    {
        $tokens = $this->lexer->tokenize($tagLine);
        $fixed = [];
        foreach ($tokens as $token) {
            if ($token[1] === Lexer::TOKEN_PHPDOC_EOL) {
                if (rtrim($token[0], " \t") !== $token[0]) {
                    $fixed[] = [rtrim($token[Lexer::VALUE_OFFSET], " \t"), Lexer::TOKEN_PHPDOC_EOL, $token[2] ?? null];
                    $fixed[] = [ltrim($token[Lexer::VALUE_OFFSET], "\n\r"), Lexer::TOKEN_HORIZONTAL_WS, ($token[2] ?? null) + 1];
                    continue;
                }
            }

            $fixed[] = $token;
        }
        return new TokenIterator($fixed);;
    }

But I think this is a bug in the way phpstan parses the docblocks. This expression: self::TOKEN_PHPDOC_EOL => '\\r?+\\n[\\x09\\x20]*+(?:\\*(?!/)\\x20?+)?', seems to be to greedy.

Do you think it's worth fixing this?

from phpdoc-parser.