I’m writing a lexer and a parser for a custom programming language, with for example the following input:
set x = "hello";
The lexer would tokenize it into
ASSIGN, VARIABLE, EQUALS, STRING, END
However, if a user enters
set x = "hello;
, that is, a string with no ending quotation mark, should the lexer or the parser handle that?
On one hand, the sole responsibility of the lexer is to generate tokens, without detecting errors. On the other hand, the lexer cannot tokenize it as a STRING
because it has no ending quotation mark.