Sphinx Pygments lexer filter extension? - python-sphinx

Question

I have a Lisp-like language I would like to highlight using Pygments in Sphinx code snippet documentation. My approach is to extend the existing CommonLispLexer to add the built-in names using a NameHighlightFilter. However, it is not working, so I must be missing something obvious. I have added the following to my conf.py:
def setup(app):
from sphinx.highlighting import lexers
from pygments.lexers import CommonLispLexer
from pygments.token import Name
from pygments.filters import NameHighlightFilter
tl_lexer = CommonLispLexer()
tl_lexer.add_filter(NameHighlightFilter(
names=['define-function', 'define-macro',
'define-variable', 'define-constant'],
tokentype=Name.Builtin,
))
app.add_lexer('tl', tl_lexer)
highlight_language = 'tl'
But the NameHighlightFilter has no effect. Code blocks are highlighted as if they were Lisp, but my new builtin names have no special highlighting.

Answer 1

The reason is that the NameHighlighFilter only converts tokens that the lexer categorizes as Token.Name, but the CommonLispLexer categorizes almost everything as Name.Variable. This is the filter function of the NameHighlightFilter, from the Pygments source code:
def filter(self, lexer, stream):
for ttype, value in stream:
if ttype is Name and value in self.names:
yield self.tokentype, value
else:
yield ttype, value
My only workaround was to write my own filter. This function gave me the look I wanted.
def filter(self, lexer, stream):
define = False
for ttype, value in stream:
if value in self.tl_toplevel_forms:
ttype = Name.Builtin
define = True
elif define and ttype == Name.Variable:
define = False
ttype = Name.Function
elif value in self.tl_special_forms:
ttype = Name.Variable
# the Common Lisp lexer highlights everything else as
# variables, which isn't the look I want. Instead
# highlight all non-special things as text.
elif ttype == Name.Variable:
ttype = Name.Text
yield ttype, value
As a note to Pygments developers, perhaps the NameHighlightFilter could take an optional argument representing the token type(s) to be converted (currently it only takes the output token type).