The title says pretty much all

What is the right way to replace a group of tokens with a different set of tokens? It feels like I must be missing something. There doesn't seem to be an easy way of constructing a list of new tokens and some of the obvious ways of doing this produce odd results. For instance this transform:

def transform_source(source, **_):
    tokens = token_utils.tokenize(source)
    for ii in range(2, 4):
        del tokens[ii]

    tokens.insert(2, '"Some Text"')
    return token_utils.untokenize(tokens)

on this source:

x = Some Text

produces this transformed source:

x ="Some Text" Some Text

What is the right way to do this? Is there a reliable way?

Consider the following example where we'd like to replace

x!
(x + 1)!

by

factorial(x)!
factorial(x + 1)!

Currently, the way to do this would be to process the string until we encounter a !, and then start backtracking workout out the logic.

Suppose we had access to groups of braces, perhaps something like:

braces = {"end_index": "begin_index"}

For example, the second case above would be

braces = {4:0}

when encoutering a ! at position index, we could simply do something like the following:

if tokens[index-1] == ")":
    begin_index = braces[index-1]
    tokens[begin_index].string = "factorial("

See friendly-traceback/friendly-traceback#242 for a description.

When I attempted to use token_utils with the unnormalized_unicode transformations form ideas, it did not work correctly. I suspect that, somewhere, some normalization was applied to some strings.

The following code fragment in useful context:

t = tokens[4]
print(token_utils.untokenize([t]))

will print t.tokens[0:4] for some reason.

It is unclear to me precisely how the thing works, as it can untokenize the array tokens, while passing a list containing only one token from tokens will untokenize all elements of tokens up to that point. When iterating over tokens, untokenize seems to emit nothing if given tokens[n] where n hasn't been iterated yet…or something.

I have no idea how this thing behaves. It's doing nothing logical.

See friendly-traceback/friendly-traceback#240 for a description of the problem. friendly_traceback does not use token-utils as a separate repo but has its own version.

The idea would be to have a few tests, run with the development version in this repository, and a separate repository, using an installed version (from pypi), that could include many extended demos - in particular, an alternate version of sympy's parsing.

See friendly-traceback/friendly-traceback#241 for a description.