Add $tokens to SubLexerInterface::getTokenAt() about rql-parser HOT 4 CLOSED

Hi @jgadrow. Are you sure that this is a good place to make a decision about token type? You can do that on the parsing stage. The Lexer should just split a string into few tokens. If the identifier and string tokens look the same why do you need them both?

But if you really need it, you can decorate the StringSubLexer class like this:

<?php
class IdentifierSubLexer implements SubLexerInterface
{
    private $stringSubLexer;
    private $operatorSubLexer;

    public function __construct(SubLexerInterface $stringSubLexer, SubLexerInterface $operatorSubLexer)
    {
        $this->stringSubLexer = $stringSubLexer;
        $this->operatorSubLexer = $operatorSubLexer;
    }

    /**
     * @inheritdoc
     */
    public function getTokenAt($code, $cursor)
    {
        $stringToken = $this->stringSubLexer->getTokenAt($code, $cursor);
        if ($stringToken === null || !$stringToken->test(Token::T_STRING)) {
            return null;
        }

        $operatorToken = $this->operatorSubLexer->getTokenAt($code, $stringToken->getEnd());
        if ($operatorToken === null || !$operatorToken->test(Token::T_OPERATOR)) {
            return null;
        }

        return new Token(
            Token::T_IDENTIFIER,
            $stringToken->getValue(),
            $stringToken->getStart(),
            $stringToken->getEnd()
        );
    }
}

Please let me know if I've got you right.

from rql-parser.

It's not a look-ahead operation but a look-behind operation. So it would be:

$operatorToken = $this->operatorSubLexer->getTokenAt($code, $cursor-1);

However, this relies on expecting the length of the previous token to be exactly 1 char which is not always correct.

If we had access to the current TokenStream, we could just look at the last token on the stack (or even further if we had a use case for it).

class IdentifierSubLexer implements SubLexerInterface
{
    /**
     * @inheritdoc
     */
    public function getTokenAt($code, $cursor, array $tokens)
    {
        if (Token::T_OPERATOR === $tokens[count($tokens)-1]->getType()
            || !preg_match('/[a-z]([a-z0-9.-_]|\%[0-9a-f]{2})*/Ai', $code, $matches, null, $cursor)
        ) {
            return null;
        }

        return new Token(
            Token::T_IDENTIFIER,
            rawurldecode($matches[0]),
            $cursor,
            $cursor + strlen($matches[0])
        );
    }
}

It's a clean, dependable way to do a look-behind check which allows me to skip the regex operation entirely if I already know my token type isn't valid in this context.

from rql-parser.

So you just need to implement your own Lexer class or override existing one where you can pass $tokens to the getTokenAt method.

It would be the easiest solution in this case because I'm not allowed to break the stable API.

from rql-parser.

True, and this was an option I considered exploring (in the end, I got my front-end devs to agree to write a custom URL encoder to also encode the -_.~ characters). But there's no rule saying all open issues have to be implemented in v1.x.

Thus, I submitted this for possible inclusion in a v2.x where you would be able to alter the interface.

Also, because I know text is a terrible way to convey tone, please know that I greatly appreciate the time you have put into this package! I considered a few different ones before deciding that of the available options I liked yours the best. :)

from rql-parser.