zeus/tools/LuaMacro/readme.md

## LuaMacro - a macro preprocessor for Lua

This is a library and driver script for preprocessing and evaluating Lua code.
Lexical macros can be defined, which may be simple C-preprocessor style macros or
macros that change their expansion depending on the context.

It is a new, rewritten version of the
[Luaforge](http://luaforge.net/projects/luamacro/) project of the same name, which
required the [token filter
patch](http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#tokenf) by Luiz Henrique de
Figueiredo. This patch allowed Lua scripts to filter the raw token stream before
the compiler stage. Within the limits imposed by the lexical filter approach this
worked pretty well.  However, the token filter patch is unlikely to ever become
part of mainline Lua, either in its original or
[revised](http://lua-users.org/lists/lua-l/2010-02/msg00325.html) form. So the most
portable option becomes precompilation, but Lua bytecode is not designed to be
platform-independent and in any case changes faster than the surface syntax of the
language. So using LuaMacro with LuaJIT would have required re-applying the patch,
and would remain within the ghetto of specialized, experimental use.

This implementation uses a [LPeg](http://www.inf.puc-rio.br/~roberto/lpeg.html)
lexical analyser originally by [Peter
Odding](http://lua-users.org/wiki/LpegRecipes) to tokenize Lua source, and builds
up a preprocessed string explicitly, which then can be loaded in the usual way.
This is not as efficient as the original, but it can be used by anyone with a Lua
interpreter, whether it is Lua 5.1, 5.2 or LuaJIT 2. An advantage of fully building
the output is that it becomes much easier to debug macros when you can actually see
the generated results. (Another example of a LPeg-based Lua macro preprocessor is
[Luma](http://luaforge.net/projects/luma/))

It is not possible to discuss macros in Lua without mentioning Fabien Fleutot's
[Metalua](metalua.luaforge.net/) which is an alternative Lua compiler which
supports syntactical macros that can work on the AST (Abstract Syntax Tree) itself
of Lua. This is clearly a technically superior way to extend Lua syntax, but again
has the disadvantage of being a direct-to-bytecode compiler. (Perhaps it's also a
matter of taste, since I find it easier to think about extending Lua on the lexical
level.)

My renewed interest in Lua lexical macros came from some discussions on the Lua
mailing list about numerically optimal Lua code using LuaJIT. We have been spoiled
by modern optimizing C/C++ compilers, where hand-optimization is often discouraged,
but LuaJIT is new and requires some assistance. For instance, unrolling short loops
can make a dramatic difference, but Lua does not provide the key concept of
constant value to assist the compiler. So a very straightforward use of a macro
preprocessor is to provide named constants in the old-fashioned C way. Very
efficient code can be generated by generalizing the idea of 'varargs' into a
statically-compiled 'tuple' type.

    tuple(3) A,B

The assigment `A = B` is expanded as:

    A_1,A_2,A_3 = B_1,B_2,B_3

I will show how the expansion can be made context-sensitive, so that the
loop-unrolling macro `do_` changes this behaviour:

    do_(i,1,3,
        A = 0.5*B
    )

expands to:

    A_1 = 0.5*B_1
    A_2 = 0.5*B_2
    A_3 = 0.5*B_3

Another use is crafting DSLs, particularly for end-user scripting. For instance,
people may be more comfortable with `forall x in t do` rather than `for _,x in
ipairs(t) do`; there is less to explain in the first form and it translates
directly to the second form. Another example comes from this common pattern:

    some_action(function()
      ...
    end)

Using the following macro:

    def_ block (function() _END_CLOSE_

we can write:

    some_action block
       ...
    end

A criticism of traditional lexical macros is that they don't respect the scoping
rules of the language itself. Bad experiences with the C preprocessor lead many to
regard them as part of the prehistory of computing. The macros described here can
be lexically scoped, and can be as 'hygenic' as necessary, since their expansion
can be finely controlled with Lua itself.

For me, a more serious charge against 'macro magic' is that it can lead to a
private dialect of the language (the original Bourne shell was written in C
'skinned' to look like Algol 68.)  This often indicates a programmer uncomfortable
with a language, who wants it to look like something more familiar. Relying on a
preprocessor may mean that programmers need to immerse themselves more in the idioms of
the new language.

That being said, macros can extend a language so that it can be more expressive for
a particular task, particularly if the users are not professional programmers.

### Basic Macro Substitution

To install LuaMacro, expand the archive and make a script or batch file that points
to `luam.lua`, for instance:

    lua /home/frodo/luamacro/luam.lua $*

(Or '%*' if on Windows.) Then put this file on your executable path.

Any Lua code loaded with `luam` goes through four distinct steps:

  * loading and defining macros
  * preprocessing
  * compilation
  * execution

The last two steps happen within Lua itself, but always occur, even though the Lua
compiler is fast enough that we mostly do not bother to save the generated bytecode.

For example, consider this `hello.lua`:

    print(HELLO)

and `hello-def.lua`:

    local macro = require 'macro'
    macro.define 'HELLO "Hello, World!"'

To run the program:

    $> luam -lhello-def hello.lua
    Hello, World!

So the module `hello-def.lua` is first loaded (compiled and executed, but not
preprocessed) and only then `hello.lua` can be preprocessed and then loaded.

Naturaly, there are easier ways to use LuaMacro, but I want to emphasize the
sequence of macro loading, preprocessing and script loading. `luam` has a `-d`
flag, meaning 'dump', which is very useful when debugging the output of the
preprocessing step:

    $> luam -d -lhello-def hello.lua
    print("Hello, World!")

`hello2.lua` is a more sensible first program:

    require_ 'hello-def'
    print(HELLO)

You cannot use the Lua `require` function at this point, since `require` is only
executed when the program starts executing and we want the macro definitions to be
available during the current compilation. `require_` is the macro version, which
loads the file at compile-time.

New with 2.5 is the default @ shortcut available when using `luam`,
so `require_` can be written `@require`.
(`@` is itself a macro, so you can redefine it if needed.)

There is also `include_/@include`, which is analogous to `#include` in `cpp`. It takes a
file path in quotes, and directly inserts the contents of the file into the current
compilation. Although tempting to use, it will not work here because again the
macro definitions will not be available at compile-time.

`hello3.lua` fits much more into the C preprocessor paradigm, which uses the `def_`
macro:

    @def HELLO "Hello, World!"
    print(HELLO)

(Like `cpp`, such macro definitions end with the line; however, there is no
equivalent of `\` to extend the definition over multiple lines.)

With 2.1, an alternative syntax `def_ (name body)` is also available, which can be
embedded inside a macro expression:

    def_ OF_ def_ (of elseif _value ==)

Or even extend over several lines:

    def_ (complain(msg,n)
      for i = 1,n do
        print msg
      end
    )

`def_` works pretty much like `#define`, for instance, `def_ SQR(x) ((x)*(x))`. A
number of C-style favourites can be defined, like `assert_` using `_STR_`, which is
a predefined macro that 'stringifies' its argument.

    def_ assert_(condn) assert(condn,_STR_(condn))

`def_` macros are _lexically scoped_:

    local X = 1
    if something then
        def_ X 42
        assert(X == 42)
    end
    assert(X == 1)

LuaMacro keeps track of Lua block structure - in particular it knows when a
particular lexical scope has just been closed.  This is how the `_END_CLOSE_`
built-in macro works

    def_ block (function() _END_CLOSE_

    my_fun block
      do_something_later()
    end

When the current scope closes with `end`, LuaMacro appends the necessary ')' to
make this syntax valid.

A common use of macros in both C and Lua is to inline optimized code for a case.
The Lua function `assert()` always evaluates its second argument, which is not
always optimal:

    def_ ASSERT(condn,expr) if condn then else error(expr) end

    ASSERT(2 == 1,"damn! ".. 2 .." is not equal to ".. 1)

If the message expression is expensive to execute, then this can give better
performance at the price of some extra code. `ASSERT` is now a statement, not a
function, however.

### Conditional Compilation

For this to work consistently, you need to use the `@` shortcut:

    @include 'test.inc'
    @def A 10
    ...

This makes macro 'preprocessor' statements stand out more. Conditional compilation
works as you would expect from C:

    -- test-cond.lua
    @if A
    print 'A defined'
    @else
    print 'A not defined'
    @end
    @if os.getenv 'P'
    print 'Env P is defined'
    @end

Now, what is `A`?  It is a Lua expression which is evaluated at _preprocessor_
time, and if it returns any value except `nil` or `false` it is true, using
the usual Lua rule. Assuming `A` is just a global variable, how can it be set?

    $ luam test-cond.lua
    A not defined
    $ luam -VA test-cond.lua
    A defined
    $ export P=1
    $ luam test-cond.lua
    A not defined
    Env P is defined

Although this looks very much like the standard C preprocessor, the implementation
is rather different - `@if` is a special macro which evaluates its argument
(everything on the rest of the line) as a _Lua expression_
and skips upto `@end` (or `@else` or `@elseif`) if that condition is false.


### Using macro.define

`macro.define` is less convenient than `def_` but much more powerful. The extended
form allows the substitution to be a _function_ which is called in-place at compile
time. These definitions must be loaded before they can be used,
either with `-l` or with `@require`.

    macro.define('DATE',function()
        return '"'..os.date('%c')..'"'
    end)

Any text which is returned will be tokenized and inserted into the output stream.
The explicit quoting here is needed to ensure that `DATE` will be replaced by the
string "04/30/11 09:57:53".  ('%c' gives you the current locale's version of the
date; for a proper version of this macro, best to use `os.date` [with more explicit
formats](http://www.lua.org/pil/22.1.html) .)

This function can also return nothing, which allows you to write macro code purely
for its _side-effects_.

Non-operator characters like `@`,`$`, etc can be used as macros. For example, say
you like shell-like notation `$HOME` for expanding environment variables in your
scripts.

    macro.define '$(x) os.getenv(_STR_(x))'

A script can now say `$(PATH)` and get the expected expansion, Make-style. But we
can do better and support `$PATH` directly:

    macro.define('$',function(get)
        local var = get:iden()
        return 'os.getenv("'..var..'")'
    end)

If a macro has no parameters, then the substitution function receives a 'getter'
object. This provides methods for extracting various token types from the input
stream. Here the `$` macro must be immediately followed by an identifier.

We can do better, and define `$` so that something like `$(pwd)` has the same
meaning as the Unix shell:

    macro.define('$',function(get)
       local t,v = get()
       if t == 'iden' then
          return 'os.getenv("'..v..'")'
       elseif t == '(' then
          local rest = get:upto ')'
          return 'os.execute("'..tostring(rest)..'")'
       end
    end)

(The getter `get` is callable, and returns the type and value of the next token.)

It is probably a silly example, but it illustrates how a macro can be overloaded
based on its lexical context. Much of the expressive power of LuaMacro comes from
allowing macros to fetch their own parameters in this way. It allows us to define
new syntax and go beyond 'pseudo-functions', which is more important for a
conventional-syntax language like Lua, rather than Lisp where everything looks like
a function anyway. These kinds of macros are called 'reader' macros in the Lisp world,
since they temporarily take over reading code.

It is entirely possible for macros to create macros; that is what `def_` does.
Consider how to add the concept of `const` declarations to Lua:

    const N,M = 10,20

Here is one solution:

    macro.define ('const',function(get)
        get() -- skip the space
        local vars = get:idens '='
        local values = get:list '\n'
        for i,name in ipairs(vars) do
            macro.assert(values[i],'each constant must be assigned!')
            macro.define_scoped(name,tostring(values[i]))
        end
    end)

The key to making these constants well-behaved is `define_scoped`, which installs a
block handler which resets the macro to its original value, which is usually `nil`.
This test script shows how the scoping works:

    require_ 'const'
    do
      const N,M = 10,20
      do
         const N = 5
         assert(N == 5)
      end
      assert(N == 10 and M == 20)
    end
    assert(N == nil and M == nil)


If we were designing a DSL intended for non-technical users, then we cannot just
say to them 'learn the language properly - go read PiL!'. It would be easier to
explain:

    forall x in {10,20,30} do

than the equivalent generic `for` loop. `forall` can be implemented fairly simply
as a macro:

    macro.define('forall',function(get)
      local var = get:iden()
      local t,v = get:next() -- will be 'in'
      local rest = tostring(get:upto 'do')
      return ('for _,%s in ipairs(%s) do'):format(var,rest)
    end)

That is, first get the loop variable, skip `in`, grab everything up to `do` and
output the corresponding `for` statement.

Useful macros can often be built using these new forms. For instance, here is a
simple list comprehension macro:

    macro.define('L(expr,select) '..
        '(function() local res = {} '..
        '  forall select do res[#res+1] = expr end '..
        'return res end)()'
    )

For example, `L(x^2,x in t)` will make a list of the squares of all elements in `t`.

Why don't we use a long string here? Because we don't wish to insert any extra line
feeds in the output.`macro.forall` defines more sophisticated `forall` statements
and list comprehension expressions, but the principle is the same - see 'tests/test-forall.lua'

There is a second argument passed to the substitution function, which is a 'putter'
object - an object for building token lists. For example, a useful shortcut for
anonymous functions:

    M.define ('\\',function(get,put)
        local args = get:idens('(')
        local body = get:list()
        return put:keyword 'function' '(' : idens(args) ')' :
            keyword 'return' : list(body) : space() : keyword 'end'
    end)

The `put` object has methods for appending particular kinds of tokens, such as
keywords and strings, and is also callable for operator tokens. These always return
the object itself, so the output can be built up with chaining.

Consider `\x,y(x+y)`: the `idens` getter grabs a comma-separated list of identifier
names upto the given token; the `list` getter grabs a general argument list. It
returns a list of token lists and by default stops at ')'.  This 'lambda' notation
was suggested by Luiz Henrique de Figueiredo as something easily parsed by any
token-filtering approach - an alternative notation `|x,y| x+y` has been
[suggested](http://lua-users.org/lists/lua-l/2009-12/msg00071.html) but is
generally impossible to implement using a lexical scanner, since it would have to
parse the function body as an expression. The `\\` macro also has the advantage
that the operator precedence is explicit: in the case of `\\(42,'answer')` it is
immediately clear that this is a function of no arguments which returns two values.

I would not necessarily suggest that lambdas are a good thing in
production code, but they _can_ be useful in iteractive exploration and within tests.

Macros with explicit parameters can define a substitution function, but this
function receives the values themselves, not the getter and putter objects. These
values are _token lists_ and must be converted into the expected types using the
token list methods:

    macro.define('test_(var,start,finish)',function(var,start,finish)
        var,start,finish = var:get_iden(),start:get_number(),finish:get_number()
        print(var,start,finish)
    end)


Since no `put` object is received, such macros need to construct their own:

        local put = M.Putter()
        ...
        return put

(They can of course still just return the substitution as text.)

### Dynamically controlling macro expansion

Consider this loop-unrolling macro:

    do_(i,1,3,
       y = y + i
    )

which will expand as

    y = y + 1
    y = y + 2
    y = y + 3

For each iteration, it needs to define a local macro `i` which expands to 1,2 and 3.

    macro.define('do_(v,s,f,stat)',function(var,start,finish,statements)
        local put = macro.Putter()
        var,start,finish = var:get_iden(),start:get_number(),finish:get_number()
        macro.push_token_stack('do_',var)
        for i = start, finish do
            -- output `set_ <var> <value> `
            put:iden 'set_':iden(var):number(i):space()
            put:tokens(statements)
        end
        -- output `undef_ <var> <value>`
        put:iden 'undef_':iden(var)
        -- output `_POP_ 'do_'`
        put:iden '_DROP_':string 'do_'
        return put
    end)

Ignoring the macro stack manipulation for a moment, it works by inserting `set_`
macro assignments into the output. That is, the raw output looks like this:

    set_ i 1
    y = y + i
    set_ i 2
    y = y + i
    set_ i 2
    y = y + i
    undef_ i
    _DROP_ 'do_'

It's important here to understand that LuaMacro does not do _recursive_
substitution. Rather, the output of macros is pushed out to the stream which is
then further substituted, etc. So we do need these little helper macros to set the
loop variable at each point.

Using the macro stack allows macros to be aware that they are expanding inside a
`do_` macro invocation.  Consider `tuple`, which is another macro which creates
macros:

    tuple(3) A,B
    A = B

which would expand as

    local A_1,A_2,A_3,B_1,B_2,B_3
    A_1,A_2,A_3 = B_1,B_2,B_3

But we would like

    do_(i,1,3,
      A = B/2
    )

to expand as

    A_1 = B_1/2
    A_2 = B_2/2
    A_2 = B_2/2

And here is the definition:

    macro.define('tuple',function(get)
        get:expecting '('
        local N = get:number()
        get:expecting ')'
        get:expecting 'space'
        local names = get:idens '\n'
        for _,name in ipairs(names) do
            macro.define(name,function(get,put)
                local loop_var = macro.value_of_macro_stack 'do_'
                if loop_var then
                    local loop_idx = tonumber(macro.get_macro_value(loop_var))
                    return put:iden (name..'_'..loop_idx)
                else
                    local out = {}
                    for i = 1,N do
                        out[i] = name..'_'..i
                    end
                    return put:idens(out)
                end
            end)
        end
    end)

The first expansion case happens if we are not within a `do_` macro; a simple list
of names is outputted.  Otherwise, we know what the loop variable is, and can
directly ask for its value.

### Operator Macros

You can of course define `@` to be a macro; a new feature allows you to add new
operator tokens:

    macro.define_tokens {'##','@-'}

which can then be used with `macro.define`, but also now with `def_`. It's now
possible to define a list comprehension syntax that reads more naturally, e.g.
`{|x^2| i=1,10}` by making `{|` into a new token.

Up to now, making a Lua operator token such as `.` into a macro was not so useful.
Such a macro may now return an extra value which indicates that the operator should
simply 'pass through' as is.  Consider defining a `with` statement:

    with A do
        .x = 1
        .y = 2
    end

I've deliberately indicated the fields using a dot (a rare case of Visual Basic
syntax being superior to Delphi).  So it is necessary to overload '.' and look at
the previous token: if it isn't a case like `name.` or `].` then we prepend the
table. Otherwise, the operator must simply _pass through_, to prevent an
uncontrolled recursion.

    M.define('with',function(get,put)
      M.define_scoped('.',function()
        local lt,lv = get:peek(-1,true) --  peek before the period...
        if lt ~= 'iden' and lt ~= ']' then
          return '_var.'
        else
          return nil,true -- pass through
        end
      end)
      local expr = get:upto 'do'
      return 'do local _var = '..tostring(expr)..'; '
    end)

Again, scoping means that this behaviour is completely local to the with-block.

A more elaborate experiment is `cskin.lua` in the tests directory. This translates
a curly-bracket form into standard Lua, and at its heart is defining '{' and '}' as
macros. You have to keep a brace stack, because these tokens still have their old
meaning and the table constructor in this example must still work, while the
trailing brace must be converted to `end`.

    if (a > b) {
       t = {a,b}
    }

### Pass-Through Macros

Normally a macro replaces the name (plus any arguments) with the substitution. It
is sometimes useful to pass the name through, but not to push the name into the
token stream - otherwise we will get an endless expansion.

    macro.define('fred',function()
      print 'fred was found'
      return nil, true
    end)

This has absolutely no effect on the preprocessed text ('fred' remains 'fred', but
has a side-effect. This happens if the substitution function returns a second
`true` value.  You can look at the immediate lexical environment with `peek`:

    macro.define('fred',function(get)
        local t,v = get:peek(1)
        if t == 'string' then
            local str = get:string()
            return 'fred_'..str
        end
        return nil,true
    end)

Pass-through macros are useful when each macro corresponds to a Lua variable; they
allow such variables to have a dual role.

An example would be Python-style lists. The [Penlight
List](http://stevedonovan.github.com/Penlight/api/modules/pl.List.html) class has
the same functionality as the built-in Python list, but does not have any
syntactical support:

    > List = require 'pl.List'
    > ls = List{10,20,20}
    > = ls:slice(1,2)
    {10,20}
    > ls:slice_assign(1,2,{10,11,20,21})
    > = ls
    {10,11,20,21,30}

It would be cool if we could add a little bit of custom syntax to make this more
natural.  What we first need is a 'macro factory' which outputs the code to create
the lists, and also suitable macros with the same names.

    -- list <var-list> [ = <init-list> ]
    M.define ('list',function(get)
        get() -- skip space
        -- 'list' acts as a 'type' followed by a variable list, which may be
        -- followed by initial values
        local values
        local vars,endt = get:idens (function(t,v)
            return t == '=' or (t == 'space' and v:find '\n')
        end)
        -- there is an initialization list
        if endt[1] == '=' then
            values,endt = get:list '\n'
        else
            values = {}
        end
        -- build up the initialization list
        for i,name in ipairs(vars) do
           M.define_scoped(name,list_check)
           values[i] = 'List('..tostring(values[i] or '')..')'
        end
        local lcal = M._interactive and '' or 'local '
        return lcal..table.concat(vars,',')..' = '..table.concat(values,',')..tostring(endt)
    end)

Note that this is a fairly re-usable pattern; it requires the type constructor
(`List` in this case) and a type-specific macro function (`list_check`). The only
tricky bit is handling the two cases, so the `idens` method finds the end using a
function, not a simple token.  `idens`, like `list`, returns the list and the token
that ended the list, so we can use `endt` to check.

    list a = {1,2,3}
    list b

becomes

    local a = List({1,2,3})
    local b = List()

unless we are in interactive mode, where `local` is not appropriate!

Each of these list macro/variables may be used in several ways:

  - directly `a` - no action!
  - `a[i]` - plain table index
  - `a[i:j]` - a list slice. Will be `a:slice(i,j)` normally, but must
  be `a:slice_assign(i,j,RHS)` if on the right-hand side of an assignment.

The substitution function checks these cases by appropriate look-ahead:

    function list_check (get,put)
        local t,v = get:peek(1)
        if t ~= '[' then return nil, true end -- pass-through; plain var reference
        get:expecting '['
        local args = get:list(']',':')
        -- it's just plain table access
        if #args == 1 then return '['..tostring(args[1])..']',true end

        -- two items separated by a colon; use sensible defaults
        M.assert(#args == 2, "slice has two arguments!")
        local start,finish = tostring(args[1]),tostring(args[2])
        if start == '' then start = '1' end
        if finish == '' then finish = '-1' end

        -- look ahead to see if we're on the left hand side of an assignment
        if get:peek(1) == '=' then
           get:next() -- skip '='
           local rest,eoln = get:upto '\n'
           rest,eoln = tostring(rest),tostring(eoln)
           return (':slice_assign(%s,%s,%s)%s'):format(start,finish,rest,eoln),true
        else
            return (':slice(%s,%s)'):format(start,finish),true
        end
    end

This can be used interactively, like so (it requires the Penlight list library.)

    $> luam  -llist -i
    Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
    Lua Macro 2.3.0 Copyright (C) 2007-2011 Steve Donovan
    > list a = {'one','two'}
    > = a:map(\x(x:sub(1,1)))
    {o,t}
    > a:append 'three'
    > a:append 'four'
    > = a
    {one,two,three,four}
    > = a[2:3]
    {two,three}
    > = a[2:2] = {'zwei','twee'}
    {one,zwei,twee,three,four}
    > = a[1:2]..{'five'}
    {one,zwei,five}

### Preprocessing C

With the 2.2 release, LuaMacro can preprocess C files, by the inclusion of a C LPeg
lexer based on work by Peter Odding. This may seem a semi-insane pursuit, given
that C already has a preprocessor, (which is widely considered a misfeature.)
However, the macros we are talking about are clever, they can maintain state, and
can be scoped lexically.

One of the irritating things about C is the need to maintain separate include
files. It would be better if we could write a module like this:


    // dll.c
    #include "dll.h"

    export {
        typedef struct {
            int ival;
        } MyStruct;
    }

    export int one(MyStruct *ms) {
        return ms->ival + 1
    }

    export int two(MyStruct *ms) {
        return 2*ms->ival;
    }

and have the preprocessor generate an apppropriate header file:


    #ifndef DLL_H
    #define DLL_H
    typedef struct {
            int ival;
        } MyStruct;

    int one(MyStruct *ms) ;
    int two(MyStruct *ms) ;
    #endif

The macro `export` is straightforward:


    M.define('export',function(get)
        local t,v = get:next()
        local decl,out
        if v == '{' then
            decl = tostring(get:upto '}')
            decl = M.substitute_tostring(decl)
            f:write(decl,'\n')
        else
            decl = v .. ' ' .. tostring(get:upto '{')
            decl = M.substitute_tostring(decl)
            f:write(decl,';\n')
            out = decl .. '{'
        end
        return out
    end)

It looks ahead and if it finds a `{}` block it writes the block as text to a file
stream; otherwise writes out the function signature.  `get:upto '}'` will do the
right thing here since it keeps track of brace level.  To allow any other macro
expansions to take place, `substitute_tostring` is directly called.

`tests/cexport.lua` shows how this idea can be extended, so that the generated
header is only updated when it changes.

To preprocess C with `luam`, you need to specify the `-C` flag:

    luam -C -lcexport -o dll.c dll.lc

Have a look at [lc](modules/macro.lc.html) which defines a simplified way to write
Lua bindings in C. Here is `tests/str.l.c`:

    // preprocess using luam -C -llc -o str.c str.l.c
    #include <string.h>

    module "str" {

      def at (Str s, Int i = 0) {
        lua_pushlstring(L,&s[i-1],1);
        return 1;
      }

      def upto (Str s, Str delim = " ") {
        lua_pushinteger(L, strcspn(s,delim) + 1);
        return 1;
      }

    }

The result looks like this:

    // preprocess using luam -C -llc -o str.c str.l.c
    #line 2 "str.lc"
    #include <string.h>

    #include <lua.h>
    #include <lauxlib.h>
    #include <lualib.h>
    #ifdef WIN32
    #define EXPORT __declspec(dllexport)
    #else
    #define EXPORT
    #endif
    typedef const char *Str;
    typedef const char *StrNil;
    typedef int Int;
    typedef double Number;
    typedef int Boolean;


      #line 6 "str.lc"
    static int l_at(lua_State *L) {
        const char *s = luaL_checklstring(L,1,NULL);
        int i = luaL_optinteger(L,2,0);

        #line 7 "str.lc"

        lua_pushlstring(L,&s[i-1],1);
        return 1;
      }

      static int l_upto(lua_State *L) {
        const char *s = luaL_checklstring(L,1,NULL);
        const char *delim = luaL_optlstring(L,2," ",NULL);

        #line 12 "str.lc"

        lua_pushinteger(L, strcspn(s,delim) + 1);
        return 1;
      }

    static const luaL_reg str_funs[] = {
           {"at",l_at},
       {"upto",l_upto},
        {NULL,NULL}
    };

    EXPORT int luaopen_str (lua_State *L) {
        luaL_register (L,"str",str_funs);

        return 1;
    }

Note the line directives; this makes working with macro-ized C code much easier
when the inevitable compile and run-time errors occur. `lc` takes away some
of the more irritating bookkeeping needed in writing C extensions
(here I only have to mention function names once)

`lc` was used for the [winapi](https://github.com/stevedonovan/winapi) project to
preprocess [this
file](https://github.com/stevedonovan/winapi/blob/master/winapi.l.c)
into [standard C](https://github.com/stevedonovan/winapi/blob/master/winapi.c).

This used an extended version of `lc` which handled the largely superficial
differences between the Lua 5.1 and 5.2 API.

(The curious thing is that `winapi` is my only project where I've leant on
LuaMacro, and it's all in C.)

### A Simple Test Framework

LuaMacro comes with yet another simple test framework - I apologize for this in
advance, because there are already quite enough. But consider it a demonstration
of how a little macro sugar can make tests more readable, even if you are
uncomfortable with them in production code (see `tests/test-test.lua`)

    require_ 'assert'
    assert_ 1 == 1
    assert_ "hello" matches "^hell"
    assert_ x.a throws 'attempt to index global'

The last line is more interesting, since it's transparently wrapping
the offending expression in an anonymous function. The expanded output looks
like this:

    T_ = require 'macro.lib.test'
    T_.assert_eq(1 ,1)
    T_.assert_match("hello" ,"^hell")
    T_.assert_match(T_.pcall_no(function() return x.a  end),'attempt to index global')

(This is a generally useful pattern - use macros to provide a thin layer of sugar
over the underlying library. The `macro.assert` module is only 75 lines long, with
comments - its job is to format code to make using the implementation easier.)

Remember that the predefined meaning of @ is to convert `@name` into `name_`. So we
could just as easily say `@assert 1 == 1` and so forth.

Lua functions often return multiple values or tables:

    two = \(40,2)
    table2 = \({40,2})
    @assert two() == (40,2)
    @assert table2() == {40,2}

For a proper grown-up Lua testing framework
that uses LuaMacro, see [Specl](http://gvvaughan.github.io/specl).


### Implementation

It is not usually necessary to understand the underlying representation of token
lists, but I present it here as a guide to understanding the code.

#### Token Lists

The token list representation of the expression `x+1` is:

    {{'iden','x'},{'+','+'},{'number','1'}}

which is the form returned by the LPeg lexical analyser. Please note that there are
also 'space' and 'comment' tokens in the stream, which is a big difference from the
token-filter standard.

The `TokenList` type defines `__tostring` and some helper methods for these lists.

The following macro is an example of the lower-level coding needed without the
usual helpers:

    local macro = require 'macro'
    macro.define('qw',function(get,put)
      local append = table.insert
      local t,v = get()
      local res = {{'{','{'}}
      t,v = get:next()
      while t ~= ')' do
        if t ~= ',' then
          append(res,{'string','"'..v..'"'})
          append(res,{',',','})
        end
        t,v = get:next()
      end
      append(res,{'}','}'})
      return res
    end)

We're using the getter `next` method to skip any whitespace, but building up the
substitution without a putter, just manipulating the raw token list.  `qw` takes a
plain list of words, separated by spaces (and maybe commas) and makes it into a
list of strings. That is,

    qw(one two three)

becomes

    {'one','two','three'}

#### Program Structure

The main loop of `macro.substitute` (towards end of `macro.lua`) summarizes the
operation of LuaMacro:

There are two macro tables, `imacro` for classic name macros, and `smacro` for
operator style macros. They contain macro tables, which must have a `subst` field
containing the substitution and may have a `parms` field, which means that they
must be followed by their arguments in parentheses.

A keywords table is chiefly used to track block scope, e.g.
`do`,`if`,`function`,etc means 'increase block level' and `end`,`until` means
'decrease block level'. At this point, any defined block handlers for this level
will be evaluated and removed. These may insert tokens into the stream, like
macros. This is how something like `_END_CLOSE_` is implemented: the `end` causes
the block level to decrease, which fires a block handler which passes `end` through
and inserts a closing `)`.

Any keyword may also have an associated keyword handler, which works rather like a
macro substitution, except that the keyword itself is always passed through first.
(Allowing keywords as regular macros would generally be a bad idea because of the
recursive substitution problem.)

The macro `subst` field may be a token list or a function. if it is a function then
that function is called, with the parameters as token lists if the macro defined
formal parameters, or with getter and setter objects if not. If the result is text
then it is parsed into a token list.