rsmenon / pygments-mathematica Goto Github PK

View Code? Open in Web Editor NEW

59.0 5.0 8.0 122 KB

Mathematica/Wolfram Language lexer and highlighter for Pygments

License: MIT License

Python 99.45% SCSS 0.55%

mathematica pygments lexer highlighting

pygments-mathematica's People

Contributors

Stargazers

Watchers

Forkers

halirutan andandandand vdt leandrocarvalho1978 alexanderz11234 dbxdev abduld rocky

pygments-mathematica's Issues

"|->" is missing as an operator in builtins.py

|-> an alternate operator form for a function. It should be included in builtins.py under the tuple for OPERATORS

Base numbers

I haven't tested it but I was looking over the code to implement the Rogue highlighter in a similar way. When I see this right, then this line is suspicious

BASE_NUMBER = r'{integer}\s*\^\^\s*({real}|{integer})'.format(integer=INTEGER, real=REAL)

Why is there whitespace allowed before and after ^^?
Does it catch the case 2^^00110011*^+3?
Does it work with characters like in 16^^abc?

Unicode characters (Greek letters) in identifiers are handled incorrectly

As far as I can tell, Unicode characters such as Greek letters are treated just like ordinary Latin letters by Mathematica. For example, a file test.m like

varλ1a = 111;
Print[varλ1a^2];

simply prints 12321. However, running pygmentize -l mathematica test.m underlines and colors the λ and also colors the 1 in the variable name varλ1a. As you can see above, the syntax highlighting here on GitHub has the same issue.

This seems wrong to me – in the example, varλ1a is simply a variable holding the value 111. As such, the whole name should be highlighted consistently as a variable name.

Degree is letter-like and not an operator

[Degree] is letter-like, not an operator,

Therefore, when we write 360° that is analogous to 2π (pun intended) it this interpreted as: 360 times "degree", and analogous to: 2 times "pi".

Note also that writing: [Degree] == Degree is valid, and this would not be the case if this symbol were not letterlike.

Referencing previous output with %, %%, %n, etc. does not work

References to previous output aren't typeset properly. I cannot find any "%" symbols anywhere in the pygments-mathematica code. Was this forgotten or omitted on purpose?
https://reference.wolfram.com/language/ref/Out.html

Backslash at the end of string causes wrong coloring

Thanks for the useful highlighter!

The minimal example is

``` wl
"\"a
```

which will be rendered as a whole string instead of string with the rest expression,

<div class="highlight"><pre id="__code_3"><span></span><code><span class="s">"\"a</span></code></pre></div>

Only after inserting some character to avoid escaping we can get the expected coloring,

``` wl
"\ "a
```

<div class="highlight"><pre id="__code_3"><span></span><code><span class="s">"\ "</span><span class="nv">a</span>
</code></pre></div>

Could you fix this issue?

Backtick in <<Package` is marked as an error

Reported by @szhorvat

Pytthon 3.7+ FutureWarning: Possible nested set at position....

Overall, I am impressed with this package and look forward to using in mathicsscript.

However on Python 3.7 and later, running pytest give:

py.test 
====================================== test session starts ======================================
platform linux -- Python 3.7.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /src/external-vcs/github/rocky/pygments-mathematica
plugins: trepan-3.0.0
collected 26 items                                                                              

tests/test_lexer.py ..........................                                            [100%]

======================================= warnings summary ========================================
../../../../../home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/nose/importer.py:12
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/nose/importer.py:12: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    from imp import find_module, load_module, acquire_lock, release_lock

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 33
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 90
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 163
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 220
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 289
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 346
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 420
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 477
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 34
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 91
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 160
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 217
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 43
    return re.compile(regex, rflags).match

tests/test_lexer.py::TestMathematicaLexer::test_comments
  /home/rocky/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pygments/lexer.py:425: FutureWarning: Possible nested set at position 100
    return re.compile(regex, rflags).match

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================ 26 passed, 15 warnings in 0.55s ================================

I will be putting in a PR to address this soon. As best as I can tell it is just adding a \ somehwere.

Highlight is wrong if a string's closing quote is at the start of a line

Hi, all. There's a highlight problem with multi-line strings: if the closing quote is at the start of the line, it is ignored and the string continues past it.

Here's a test case:

import pygments
import mathematica
print(list(pygments.lex('"string\n"variable', mathematica.MathematicaLexer())))

... which prints:

[(Token.Literal.String, '"'),
 (Token.Literal.String, 'string\n'),
 (Token.Literal.String, '"'),
 (Token.Literal.String, 'variable\n')]

As you can see, variable is marked as a string. The same doesn't happen if the closing quote is preceded by a character:

import pygments
import mathematica
print(list(pygments.lex('"string\nx"variable', mathematica.MathematicaLexer())))

This prints:

[(Token.Literal.String, '"'),
 (Token.Literal.String, 'string\nx'),
 (Token.Literal.String, '"'),
 (Token.Name.Variable, 'variable'),
 (Token.Text.Whitespace, '\n')]

(Not sure where the final '\n' whitespace is coming from in this case, but this is harmless).

Backslash in strings triggers error

This markdown

    :::wolfram
    "\\foo"

is translated to

<div class="codehilite"><pre><span></span><span class="s">&quot;</span><span class="err">\\</span><span class="s">foo&quot;</span><span class="w"></span>
</pre></div>

The \\ is marked as an error in the string.

Tilde is an operator, but is in `UNICODE_SYSTEM_UNDEFINED_SYMBOLS`

See https://reference.wolfram.com/language/ref/Infix.html

It should be removed from https://github.com/rsmenon/pygments-mathematica/blob/master/mathematica/builtins.py#L5692 and added to UNICODE_OPERATORS.

Overleaf

Did anyone get this lexer to work on Overleaf? Crucial for me...