First, a big N.B. : I am not a Python programmer — I just know enough from reading the Pygments documentation and browsing the source of Pygments’ available lexers on Github to know how to write a Pygments lexer.

I have been working on a Jazz / SASM Saturn assembly syntax highlighter in Pygments. When I first began this endeavor, I thought the task would be extremely easy, but I soon ran into problems involving either quirks in how Pygments processes newlines and white-space, of which I don not understand the details and I have been pulling my hair out trying to get the lexer to work to no avail, hence my post here.

Jazz / SASM syntax has some white-space sensitive requirements, particularly that a label has to start in column one or two — I don’t know if this is part of the problem.

The first problem is that when a label starts in column two and is preceded by a blank line ( ie. just ‘n’ ) the label is marked as a lexing error or the whole line is marked as lexing error if a comment follows the label on the same line.

The second problem is quite strange and I have no idea what’s causing it : If a full-line comment ( which must start in column one with an asterisk ) is preceded by a blank line, then if there’s an instruction in the text that one would think would be commented out, instead it generates a lexing error for said instruction text.

For the third problem further explanation of Jazz / SASM syntax is needed : All conditional instructions must be followed by a ‘GOYES label’ or just a ‘RTNYES’ on the next line. The first sub-issue I’m encountering here is that if the conditional instruction has trailing spaces, which should be handled by the comment processing code, then Pygments marks said white-space as a lexing error. Additionally, the conditional instructions seem to be immune to whatever is causing the lexing errors associated with problem one ie. if the conditional instruction is preceded by a label on the same line, irrespective if it starts in column one or two and it’s preceded by a blank line, then Pygments seems to process the whole line without lexing errors. Lastly, the required ‘GOYES label’ or ‘RTNYES’ on the next line after the conditional is always marked as a lexing error by Pygments.

Now, I know all the above might not make a lot of sense because it is divorced from the code, so, WLOG, I’ve included the Pygments lexer code which just has two instructions : setting a register to zero and a conditional instruction that compares two registers.

from pygments.lexer import RegexLexer, bygroups
from pygments.token import *
import logging

class SasmJazzLexer(RegexLexer):
    name = 'SASM'
    aliases = ['SASM', 'Sasm', 'sasm', 'JAZZ', 'Jazz', 'jazz']
    filenames = ['*.A', '*.a']
    
    # Register fields
    fields = r'(P|WP|XS|X|S|M|B|A|W)'
    
    # Regex for label or symbol
    symbol = r'((?:[:=][(=:#0-9)][^, )]{,11})|(?:[:=]?[^(=:#0-9), ][^, ]{,11}))'
    
    
    tokens = {
        'root': [
            # Single line comment starting with '*' in column one
            (r'^*.*$', Comment.Single),
            
            # Whitespace or instructions
            (r'^( {2,})|(t)s*', Whitespace, 'possible_instructions'),
            
            # Label starting on column one or two and followed by a comment            
            (r'(^ ?)' + symbol + r'(s+*.*$)', bygroups(Whitespace, Name.Label, Comment.Single)),
            
            # Label starting in column one or two and followed by whitespace or instructions
            (r'(^ ?)' + symbol + r'(s+)', bygroups(Whitespace, Name.Label, Whitespace), 'possible_instructions'),
        ],
        'possible_instructions': [
            # Just whitespace or empty line -- emit token and bail
            (r'(s*$)|$', Whitespace, '#pop'),
            
            #
            # Register tests
            #
            
            (r'(?)([A-C])(=)([A-C])(' + r's+' + fields + r')', bygroups(Punctuation, Name.Builtin, Operator, Name.Builtin, Name.Builtin), ('possible_comment_cond', 'conditionals')),
            (r'(?)([CD])(=)([CD])(' + r's+' + fields + r')', bygroups(Punctuation, Name.Builtin, Operator, Name.Builtin, Name.Builtin), ('possible_comment_cond', 'conditionals')),
            (r'(?)(([A-D])(=)(3))(' + r's+' + fields + r')', bygroups(Punctuation, Error, Name.Builtin, Operator, Name.Builtin, Name.Builtin), ('possible_comment_cond', 'conditionals')),
            
            #
            # Arithmetic instructions
            #
            
            # r=0 fs
            # possibly followed by a comment
            (r'([A-D])(=)(0)(' + r's+' + fields + r')', bygroups(Name.Builtin, Operator, Name.Builtin, Name.Builtin), 'possible_comment'),
        ],
        'possible_comment': [
            #
            # Comment processing code
            #
            
            # Just whitespace or empty line -- emit token and bail
            (r'(s*$)|$', Whitespace, '#pop:2'),
            
            # Trailing comment
            (r's+S.*$', Comment.Single, '#pop:2')
        ],
        'possible_comment_cond': [
            #
            # Comment processing code for conditionals
            #
            
            # Just whitespace or empty line -- emit token and bail
            (r'(s*$)|$', Whitespace, '#pop'),
            
            # Trailing comment
            (r's+S.*$', Comment.Single, '#pop'),
        ],
        'conditionals': [
            #
            # GOYES and RTNYES processing code
            #
            
            (r'(^ {2,})(GOYES)(s+)' + symbol, bygroups(Whitespace, Keyword, Whitespace, Name.Label), 'possible_comment'),
            (r'(^ {2,})(RTNYES)', bygroups(Whitespace, Keyword), 'possible_comment'),

        ]
    }

In the above, if a line is not a full-line comment, then it either matches a label in column one or two which is possibly followed by an instruction ( in state ‘possible_instructions’ ) after which may follow an comment ( handled by the state ‘possible_comment’ which in this case there is no need to use an asterisk ) separated by the instruction by white-space.

For the conditional instructions, two states are pushed to the stack :

('possible_comment_cond', 'conditionals')

The ‘possible_comment_cond’ state is needed because every tuple in the ‘possible_comment’ state pops two stack levels which would pop the ‘conditionals’ state which is needed to process the ‘GOYES label’ or ‘RTNYES’ that must follow a conditional instruction.

Note that I have been generating a complete HTML document to test my lexer using

python3 -m pygments -f html -O full -o test.html -x -l pygments_sasm_lexer_5.py:SasmJazzLexer test.txt

Hopefully I’ve included enough information for this question to be understood — if I haven’t, please comment and I’ll add more details if necessary.