How to add a custom token inside a block token? about mistletoe HOT 3 OPEN

Thank you @pbodnar and @anderskaplan for pointers!

As both of you mentioned - this is currently implemented in preprocessing. We just have a regex to do that, however doesn't feel very clean and robust. That's why I wanted to use mistletoe to improve this.

I'll check if there's a nice way to get it working and post the code here if I end-up implementing this.

from mistletoe.

Hi @MilyMilo, as @anderskaplan writes, it would probably make sense to do the embedding as a pre-processing step - i.e. before calling mistletoe?

Or, I think you could alternatively try just to override the corresponding HtmlRenderer's render methods like this (schematically):

    def render_inline_code(self, token: span_token.InlineCode) -> str:
        # TODO: embed chunks inside token.children[0].content
        return super().render_inline_code(token)

    # ...

    def render_block_code(self, token: block_token.BlockCode) -> str:
        # TODO: embed chunks inside token.children[0].content (the same as above => call a shared helper method)
        return super().render_block_code(token)

What do you think?

from mistletoe.

Hi @MilyMilo, it is certainly possible to make custom tokens for fenced code blocks and for code spans with the behavior that you want, but I believe it's going to be a lot of work.

The embedding feature you want would probably be easier to build as a pre-processing step, working on the raw text input. That would also make sense from a conceptual point of view, imho.

Maybe you could use a cusomized parser which only recognizes paragraphs and links in that first step 😄

from mistletoe.