Possibility to add a custom lexer about pygments.rb HOT 8 CLOSED

What we can do here is to teach pygments.rb to use preinstalled Pygments (if it is present) instead of the bundled one. This way if user installed Pygments plugins, they will work.

from pygments.rb.

Yeah, this would be great.

from pygments.rb.

I just had to figure this out.. It's actually already possible with some additional work. In my case it was a bit more complicated as I have a language that doesn't have a convenient existing lexer. So here's what I did:

1. Created a Pygments lexer locally for my language.
2. Created a PIP package from it.. jam_pygments
3. Installed with pip by pointing to my local git clone of that.
4. Created/registered the plugin lexer in ruby code as Pygments::Lexer.create name: 'Jam', aliases: ['jam','bjam','b2'], filenames: ['*.jam','Jamfile','Jamroot'], mimetypes: ['text/x-jam'] so that the find function will return it as needed.
5. Now follow regular use.

For you use case you only need to do steps 3-5 obviously.

HTH

from pygments.rb.

Okay, #46 is about custom styles while this issue is about custom lexers.

from pygments.rb.

Easier workaround:

--- popen.rb.orig	2021-02-17 16:13:18.000000000 +0100
+++ popen.rb	2021-02-17 16:14:02.000000000 +0100
@@ -141,11 +141,7 @@
     #
     # Returns an array of lexers.
     def lexers
-      lexer_file = File.expand_path('../../lexers', __dir__)
-      raw = File.open(lexer_file, 'rb').read
-      Marshal.load(raw)
-    rescue Errno::ENOENT
-      raise MentosError, 'Error loading lexer file. Was it created and vendored?'
+      lexers!
     end

     # Public: Get back all available lexers from mentos itself

This makes pygments.rb find lexer plugin installed in this way: https://stackoverflow.com/a/39540683/6013024

The way I found this:

1. I've checked which python interpreter is called -> it's normal python3
2. I've checked the lexer plugin is installed for this interpreter -> yes, from pygments import lexers; lexers.get_lexer_by_name('my') works
3. I've added hook to get_lexer_by_name() in gems/pygments.rb-2.0.0/vendor/ -> it's not called for "my" language
4. I've investigated how pygments.rb gets the list of lexers -> it reads files in the vendor version of pygments, ignoring plugins!

Disabling this optimization (listing files instead of asking pygments API) makes normal plugins work.

from pygments.rb.

@rbrich do I understand correctly that you install plugins into Pygments that is bundled inside pygments.rb?

from pygments.rb.

Sort of, but not really. The plugins are standalone Python packages, they don't need to depend on Pygments.

You can have this combination and it works fine:

• python3, without pygments
• a pygments plugin installed in python3's site-packages
• pygments.rb using python3 and its own installation of pygments

The only condition is that python3 is the same interpreter, in all cases. I'm using both Python and Ruby from Homebrew. Pygments.rb was installed with gem, the custom lexer was installed with python3 setup.py install. The lexer is using plugin structure as described in the SO answer.

from pygments.rb.

Oh, I see. So, bundled Pygments picks up system-wide installed plugins.

from pygments.rb.