Having worked on a few Lisp projects (Mostly Scheme, some CL) here are my thoughts.

With macros you can make code hell to maintain. But this is only when you don’t document. Frankly I think the same can be said for any language feature. When was the last time you read 500k lines of OO ravioli and went “Gee there was no docs, but this code is so understandable anyways”.

The best way to make sure everything stays readable is simply to enforce documentation. In the projects I’ve worked on the documentation had to have something like

;; sommac, Used to do foo.
;; Translation:
;;   (sommac (a b c)
;;       d e f)
;; to
;;    (loop for a from b to c
;;          do (begin d e f))

The translation isn’t exactly what the macro will do. Sometimes they were in pseudo-code. But it was a simple effective way to make sure everyone knew what was going on.

Since you have a C background it is best to first look at lisp and C macros, and see what properties they share and which they are not sharing.

In C, macros are compile-time-rules for almost textual replacement of code parts by the preprocessor. C macros have a different syntax than the C programming language, they are executed by the preprocessor and emit C code that is compiled by the C compiler. So it is a preceding step to program compilation.

Lisp is a little different here. In Lisp, every code is a tree-like (directed, acyclic graph) datastructure that is written in S-Expressions (think of S-Expressions like compact XML).

(case a
   ((10) "a is ten")
   (else "foo"))

This datastructure consists of nested list objects that offer a simple API for accessing its elements. The great thing about lisp is, that the same data structures used in a lisp program for storing lists is also used during the compilation of the Lisp code. Therefore one can write small “compiler” extensions in Lisp code that transform branches of the abstract syntax tree into other representations. Again: You use the same language that you use for programming an algorithm to transform the code.

For a moment, I would downplay the differences between a procedure and a macro call. The first real difference between a procedure and a macro is about the time when they are executed not about the nature of their abstraction in the first place.

The overall question is: How can we keep a codebase from becoming an obscure meta-langauge that is not accessible to colleagues?

Keeping programs readable

Readability of programs is a complex topic.

• First of all it comes down to the experience and expectancy of the reader. Code that looks foreign is harder to understand for a programmer than code he/she is familiar with.

• Then there is the problem of code-scope. When reading code we are quite local and cannot look at all parts of the source at the same time. Therefore code that is compact should be easier to read than related code that is distributed widely over the source file.

• However, good abstraction helps code readability since it groups operations into logical chunks. A function composes these abstracted operations is more readable than an equivalent function that concretely implements everything on its own.

• Good abstraction means separation of concerns. A function that does several unrelated things at a time is harder to understand since you have to disentangle the functionalities within the procedure.

  Consider a function write_file_and_show_results might be a bad function. Especially if it does not compose one write_file and one show_results function but directly opens the file, starts writing to it, shows results, writes some more….

The virtue of lisp is, that it is really good for implementing these good abstractions. This is often compared to “writing an own language for the problem”. Yet it is hardly different than programming in other languages, Lisp just has more drastic means for extending syntax.

Commonly if people argue against macro use or abstraction, they point out, that a direct, concrete implementation is more readable than one that involves a few abstractions. In practice, it mostly pays off to introduce some abstraction that compacts the rest of the code. It saves LoC, maintainability is increased since you only have to perform changes once than in several parts of the problem

Architectural design: Bottom up

Paul Graham has written a nice essay on how to work when writing Lisp code. http://www.paulgraham.com/progbot.html

Macro Hygiene

There is one huge problem associated with macros: Hygiene.

Take the following example of a C program to illustrate it.

#define max_defect(a,b) ((a) > (b) ? (a) : (b))

The issue at hand is, that an invocation of max_defect(i++, 3) will increase i twice. Typically this problem is solved by introducing another binding of a and b (something like _a) and evaluate the parameters to it. Then, the bound results are used to perform the actual code.

However, what happens if you then use _a as a parameter to the maximum-determining function? You have a name conflict at hand. In Lisp this problem has been solved by introducing gensyms. Generated symbols that are unique in the program.

However since this still leaves the possibility for errors, other ways to define macros have been developed, so called hygienic macros. Scheme does offer hygienic macros. They always guarantee correct lexical scoping.

Developing a common language

When presenting the idea of a “modifiable” programming language, people sometimes fear, that this might lead to a programmer building up “his/her own world” that is inaccessible to the outsider. While this is certainly possible, as long as you are not working in total isolation it will not be the immediate consequence. A lot of macros are already existing that encapsulate a certain abstraction, their naming schemes can be adapted, and there are also surprisingly readable macro constructs, such as the loop macro.

Generally the lisp communities have slightly different traditions in naming things than other programming communities. make-whatever for creating an object, with-RESOURCE for RAII like resource management, etc. Getting used to this is not an issue when reading established lisp codes and will enable you to write consisten lisp code.

Disclaimer: I have mostly worked with Scheme and Clojure, not very much with Common Lisp. Clojure is probably the most readable dialect for people who have some experience with Python and/or ruby.

Lisp macros can be used well or badly.

Simple use of macros can be similar in effect to cpp – replacing something simple looking with related boilerplate for example. It can make things prettier without changing much if used well, or it can make things awful by doing something unexpected.

The entire lisp language and environment is available to lisp macros. Sophisticated use of lisp macros can implement whole language constructs (“loop” for example is just a lisp macro), or perform complex analysis or build complex data structures; effectively becoming part of the compiler.