-1

So lexers are supposed to emit tokens for key structures like INDENT and DEDENT for indentation stuff, or these:

NUMBER ::= [0-9]+
ID     ::= [a-Z]+, except for keywords
IF     ::= 'if'
LPAREN ::= '('
RPAREN ::= ')'
COMMA  ::= ','
LBRACE ::= '{'
RBRACE ::= '}'
SEMICOLON ::= ';'

What does it do for template strings like in JavaScript:

`I am a ${simple} template string!`
`I am a ${complex ? `nested ${simple}` : 'basic'} template string!`

What are the lexer tokens for such strings? How does it handle the regular part of the string vs. the variable part, and how generally does the lexer parse this code to emit its tokens?

The template string has both string components, and arbitrarily deeply nested recursion of code/template/code/template/etc. How does the lexer know what to do?

Improve this question
1
  • Regular expressions can be quite a bit more intricate than the rudimentary ones you cite as examples. It comes down to creating the right ones and use them recursively. An inner $(...) would be something like "a literal $ followed by a (, followed by any number of characters not being (, followed by )". You can process that, cut it out and repeat to find outer ones. If you code the logic yourself (not using regex) you may start with outer expressions and count parentheses. Commented Aug 21, 2021 at 13:25

1 Answer 1

Reset to default
2

Two typical solutions:

  • give up on using a separate lexer. This is easy and efficient with top-down parsing approaches such as recursive descent, PEG, or parser combinators. Such an approach makes it natural to embed languages within each other, e.g. JavaScript code containing strings containing JavaScript expressions.

  • yield multiple tokens per template string. For example, the input `I am a ${simple} template string!` might be tokenized as:

    • template string start "I am a "
    • identifier simple
    • template string end " template string!"

    where the tokens for template strings would roughly be defined as follows using regex notation:

    <template string start> = /`.*?\$\{/
<template string middle> = /}.*?\$\{/
<template string end> = /}.*?`/
<template string simple> = /`.*?`/

    and the grammar for template strings might be:

    <template string> = <template string simple>
                  | <template string start> <expression> <template string rest>
<template string rest> = <template string middle> <expression> <template string rest>
                       | <template string end>

    Nevertheless, this tokenization approach is not particularly good. For example, the input {};`//foo` could be tokenized as open and closing braces, semicolon, simple template string "//foo", or as opening brace, template string end ";" followed by a comment. Thus, the correct tokenization would depend on the parser state. While this state can be discovered for some parser types, a top-down parsing approach without a separate tokenization phase is a much more natural way to deal with nested languages.

Improve this answer
3
  • How does it know that it's part of a string though? Commented Aug 21, 2021 at 7:23
  • Later the grammar would use its rules to match this up into a string expression node. Unless your language supports juxtaposition operations, it can be presumed that any combination of strings and string expressions side by side should be evaluated concatenated. If these where two strings being passed to say a function, an argument separator token would be lexed up. Overloaded juxtaposition operators would simply delay the choice of juxtaposition operator (concat vs. custom) till later in the compilation process where this can be decided on. Commented Aug 21, 2021 at 8:30
  • 2
    There's also the possibility of still having a lexer and a parser, but have the lexer call the parser or the parser call the lexer, or both. Or, have the lexer not be purely regular but partially context-free or even context-sensitive. In other words: the clear distinction between a regular lexer and a context-free parser only exists in text books, and even there, probably only in older ones. Or, put another way: compiler text books, and actual compilers are pretty different. Commented Aug 21, 2021 at 9:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.