Many Lispers will tell you that what makes Lisp special is homoiconicity, which means that the code’s syntax is represented using the same data structures as other data. For example, here’s a simple function (using Scheme syntax) for calculating the hypotenuse of a right-angled triangle with the given side lengths:

(define (hypot x y)
  (sqrt (+ (square x) (square y))))

Now, homoiconicity says that the code above is actually representable as a data structure (specifically, lists of lists) in Lisp code. Thus, consider the following lists and see how they “glue” together:

1. (define #2# #3#)
2. (hypot x y)
3. (sqrt #4#)
4. (+ #5# #6#)
5. (square x)
6. (square y)

Macros allow you to treat the source code as just that: lists of stuff. Each of those 6 “sublists” contain either pointers to other lists, or to symbols (in this example: define, hypot, x, y, sqrt, +, square).

So, how can we use homoiconicity to “pick apart” syntax and make macros? Here’s a simple example. Let’s reimplement the let macro, which we’ll call my-let. As a reminder,

(my-let ((foo 1)
         (bar 2))
  (+ foo bar))

should expand into

((lambda (foo bar)
   (+ foo bar))
 1 2)

Here’s an implementation using Scheme “explicit renaming” macros^†:

(define-syntax my-let
  (er-macro-transformer
    (lambda (form rename compare)
      (define bindings (cadr form))
      (define body (cddr form))
      `((,(rename 'lambda) ,(map car bindings)
          ,@body)
        ,@(map cadr bindings)))))

The form parameter is bound to the actual form, so for our example, it would be (my-let ((foo 1) (bar 2)) (+ foo bar)). So, let’s work through the example:

1. First, we retrieve the bindings from the form. cadr grabs the ((foo 1) (bar 2)) portion of the form.

2. Then, we retrieve the body from the form. cddr grabs the ((+ foo bar)) portion of the form. (Note that this is intended to grab all the subforms after the binding; so if the form were

   (my-let ((foo 1)
            (bar 2))
     (debug foo)
     (debug bar)
     (+ foo bar))
   

   then the body would be ((debug foo) (debug bar) (+ foo bar)).)

3. Now, we actually build the resultant lambda expression and call using the bindings and body we have collected. The backtick is called a “quasiquote”, which means to treat everything inside the quasiquote as literal datums, except the bits after the commas (“unquote”).
   • The (rename 'lambda) means to use the lambda binding in force when this macro is defined, rather than whatever lambda binding might be around when this macro is used. (This is known as hygiene.)
   • (map car bindings) returns (foo bar): the first datum in each of the bindings.
   • (map cadr bindings) returns (1 2): the second datum in each of the bindings.
   • ,@ does “splicing”, which is used for expressions that return a list: it causes the list’s elements to be pasted into the result, rather than the list itself.
4. Putting all that together, we get, as a result, the list (($lambda (foo bar) (+ foo bar)) 1 2), where $lambda here refers to the renamed lambda.

Straightforward, right? 😉 (If it’s not straightforward for you, just imagine how difficult it would be to implement a macro system for other languages.)

So, you can have macro systems for other languages, if you have a way to be able to “pick apart” source code in a non-clunky way. There are some attempts at this. For example, sweet.js does this for JavaScript.

† For seasoned Schemers reading this, I intentionally chose to use explicit renaming macros as a middle compromise between defmacros used by other Lisp dialects, and syntax-rules (which would be the standard way to implement such a macro in Scheme). I don’t want to write in other Lisp dialects, but I don’t want to alienate non-Schemers who aren’t used to syntax-rules.

For reference, here’s the my-let macro that uses syntax-rules:

(define-syntax my-let
  (syntax-rules ()
    ((my-let ((id val) ...)
       body ...)
     ((lambda (id ...)
        body ...)
      val ...))))

The corresponding syntax-case version looks very similar:

(define-syntax my-let
  (lambda (stx)
    (syntax-case stx ()
      ((_ ((id val) ...)
         body ...)
       #'((lambda (id ...)
            body ...)
          val ...)))))

The difference between the two is that everything in syntax-rules has an implicit #' applied, so you can only have pattern/template pairs in syntax-rules, hence it’s fully declarative. In contrast, in syntax-case, the bit after the pattern is actual code that, in the end, has to return a syntax object (#'(...)), but can contain other code too.

A dissenting opinion: Lisp’s homoiconicity is far less of a useful thing than most Lisp fans would have you believe.

To understand syntactic macros, it’s important to understand compilers. The job of a compiler is to turn human-readable code into executable code. From a very high-level perspective, this has two overall phases: parsing and code generation.

Parsing is the process of reading code, interpreting it according to a set of formal rules, and transforming it into a tree structure, generally known as an AST (Abstract Syntax Tree). For all the diversity among programming languages, this is one remarkable commonality: essentially every general-purpose programming language parses into a tree structure.

Code generation takes the parser’s AST as its input, and transforms it into executable code via the application of formal rules. From a performance perspective, this is a much simpler task; many high-level language compilers spend 75% or more of their time on parsing.

The thing to remember about Lisp is that it’s very, very old. Among programming languages, only FORTRAN is older than Lisp. Way back in the day, parsing (the slow part of compiling) was considered a dark and mysterious art. John McCarthy’s original papers on the theory of Lisp (back when it was just an idea that he never thought could actually be implemented as a real computer programming language) describes a somewhat more complex and expressive syntax than the modern “S-expressions everywhere for everything” notation. That came about later, when people were trying to actually implement it. Because parsing was not well-understood back then, they basically punted on it and dumbed down the syntax into a homoiconic tree structure in order to make the parser’s job utterly trivial. The end result is that you (the developer) have to do a lot of the parser’s work for it by writing the formal AST directly into your code. Homoiconicity doesn’t “make macros so much easier” as much as it makes writing everything else that much harder!

The problem with this is that, especially with dynamic typing, it’s very difficult for S-expressions to carry much semantic information around with them. When all of your syntax is the same type of thing (lists of lists), there isn’t much in the way of context provided by the syntax, and so the macro system has very little to work with.

Compiler theory has come a long way since the 1960s when Lisp was invented, and while the things it accomplished were impressive for its day, they look rather primitive now. For an example of a modern metaprogramming system, have a look at the (sadly underappreciated) Boo language. Boo is statically typed, object-oriented, and open-source, so every AST node has a type with a well-defined structure that a macro developer can read the code to. The language has a relatively simple syntax inspired by Python, with various keywords that give intrinsic semantic meaning to the tree structures built from them, and its metaprogramming has an intuitive quasiquote syntax to simplify the creation of new AST nodes.

Here’s a macro I created yesterday when I realized I was applying the same pattern to a bunch of different places in GUI code, where I would call BeginUpdate() on a UI control, perform an update in a try block, and then call EndUpdate():

macro UIUpdate(value as Expression):
    return [|
        $value.BeginUpdate()
        try:
            $(UIUpdate.Body)
        ensure:
            $value.EndUpdate()
    |]

The macro command is, in fact, a macro itself, one that takes a macro body as input and generates a class to process the macro. It uses the name of the macro as a variable that stands in for the MacroStatement AST node that represents the macro invocation. The [| … |] is a quasiquote block, generating the AST that corresponds to the code within, and inside the quasiquote block, the $ symbol provides the “unquote” facility, substituting in a node as specified.

With this, it’s possible to write:

UIUpdate myComboBox:
   LoadDataInto(myComboBox)
   myComboBox.SelectedIndex = 0

and have it expand to:

myComboBox.BeginUpdate()
try:
   LoadDataInto(myComboBox)
   myComboBox.SelectedIndex = 0
ensure:
   myComboBox.EndUpdate()

Expressing the macro this way is simpler and more intuitive than it would be in a Lisp macro, because the developer knows the structure of MacroStatement and knows how the Arguments and Body properties work, and that inherent knowledge can be used to express the concepts involved in a very intuitive way. It’s also safer, because the compiler knows the structure of MacroStatement, and if you try to code something that isn’t valid for a MacroStatement, the compiler will catch it right away and report the error instead of you not knowing until something blows up on you at runtime.

Grafting macros onto Haskell, Python, Java, Scala, etc isn’t difficult because these languages aren’t homoiconic; it’s difficult because the languages are not designed for them, and it works best when your language’s AST hierarchy is designed from the ground up to be examined and manipulated by a macro system. When you’re working with a language that was designed with metaprogramming in mind from the beginning, macros are much simpler and easier to work with!

I’m learning Scheme from the SICP and I’m getting the impression that a big part of what makes Scheme and, even more so, LISP special is the macro system.

How so? All the code in SICP is written in macro-free style. There are no macros in SICP. Only in a footnote on page 373 are macros ever mentioned.

But, since macros are expanded at compile-time

They aren’t necessarily. Lisp provides macros in both interpreters and compilers. Thus there might not be a compile-time. If you have a Lisp interpreter, macros are expanded at execution time. Since many Lisp systems have a compiler on board, one can generate code and compile it at runtime.

Let’s test that using SBCL, a Common Lisp implementation.

Let’s switch SBCL to the Interpreter:

* (setf sb-ext:*evaluator-mode* :interpret)

:INTERPRET

Now we define a macro. The macro prints something when it is called for code expanded. The generated code does not print.

* (defmacro my-and (a b)
    (print "macro my-and used")
    `(if ,a
         (if ,b t nil)
         nil))

Now let’s use the macro:

MY-AND
* (defun foo (a b) (my-and a b))

FOO

See. In above case Lisp does nothing. The macro is not expanded at definition time.

* (foo t nil)

"macro my-and used"
NIL

But at runtime, when the code is used, the macro is expanded.

* (foo t t)

"macro my-and used"
T

Again, at runtime, when the code is used, the macro is expanded.

Note that SBCL would expand only once when using a compiler. But various Lisp implementations also provide interpreters – like SBCL.

Why are macros easy in Lisp? Well, they aren’t really easy. Only in Lisps, and there are many, which have macro support built in. Since many Lisps come with extensive machinery for macros, it looks like it is easy. But macro mechanisms can be extremely complicated.

Homoiconicity makes it much much easier to implement macros. The idea that code is data and data is code makes it possible to more or less (barring accidental capture of identifiers, solved by hygienic macros) to freely substitute one for the other. Lisp and Scheme make this easier with their syntax of S-expressions which are uniformly structured and thus easy to turn into ASTs which form the basis of Syntactic Macros.

Languages without S-Expressions or Homoiconicity will run into trouble implementing Syntactic Macros though it can still certainly be done. Project Kepler is attempting to introduce them to Scala for instance.

The largest issue with syntax macros usage aside from non-homoiconicity, is the issue of arbitrarily generated syntax. They offer tremendous flexibility and power but at the price that your source code may no longer be as easy to understand or maintain.