The Early History of Metaprogramming

Throughout the history of computer science, various languages have evolved to support different forms of metaprogramming. One of the most ancient is the LISP family of languages, in which the program itself was considered a piece of data and well-known LISP macros were able to be used to extend the languages from within. Other languages relied on deep reflection capabilities to handle such tasks either during compile time or during runtime.

Outside of the LISP family, C and its preprocessor became a tool of choice to generate code from a boilerplate template. Beyond classical function-like macros, the technique known as X-macros was a very interesting one. An X-macro is, in fact, a header file containing a list of similar macro invocations—often called the components—which can be included multiple times. Each inclusion is prefixed by the redefinition of said macro to generate different code fragments for the same list of components. A classic example is structure serialization, wherein the X-macros will enumerate structure members, first during definition and then during serialization:

// in components.h
PROCESS(float, x      )
PROCESS(float, y      )
PROCESS(float, z      )
PROCESS(float, weight )

// in particle.c
typedef struct
{
  #define PROCESS(type, member) type member;
  #include "components.h"
  #undef PROCESS
} particle_t;

void save(particle_t const* p, unsigned char* data)
{
  #define PROCESS(type, member)                   
  memmove(data, &(p->member), sizeof(p->member)); 
  data += sizeof(p->member);                      
  /**/

  #include "components.h"

  #undef PROCESS
}

X-macros are a well-tested, pure C-style solution. Like a lot of C-based solutions, they work quite well and deliver the performance we expect. We could debate the elegance of this solution, but consider that a very similar yet more automated system is available through the Boost.Preprocessor vertical repetition mechanism, based on self-referencing macros.¹

Enter C++ Templates

Then came C++ and its generic types and functions implemented through the template mechanism. Templates were originally a very simple feature, allowing code to be parameterized by types and integral constants in such a way that more generic code can emerge from a collection of existing variants of a given piece of code. It was quickly discovered that by supporting partial specialization, compile-time equivalents of recursion or conditional statements were feasible. After a short while, Erwin Unruh came up with a very interesting program² that builds the list of every prime number between 1 and an arbitrary limit. Quite mundane, isn’t it? Except that this enumeration was done through warnings at compile time.

Let’s take a moment to ponder the scope of this discovery. It meant that we could turn templates into a very crude and syntactically impractical functional language, which later would actually be proven by Todd Veldhuizen to be Turing-complete. If your computer science courses need to be refreshed, this basically meant that, given the necessary effort, any functions computable by a Turing machine (i.e., a computer) could be turned into a compile-time equivalent by using C++ templates. The era of C++ template metaprogramming was coming.

C++ template metaprogramming is a technique based on the use (and abuse) of C++ template properties to perform arbitrary computations at compile time. Even if templates are Turing-complete, we barely need a fraction of this computational power. A classic roster of applications of C++ template metaprogramming includes the following:

• Complex constant computations
• Programmatic type constructions
• Code fragment generation and replication

Those applications are usually backed up by some libraries, like Boost.MPL or Boost.Fusion, and a set of patterns including tag dispatching, recursive inheritance, and SFINAE.³ All of those components thrived in the C++03 ecosystem and have been used by a large number of other libraries and applications.

The main goal of those components was to provide compile-time constructs with an STL look and feel. Boost.MPL is designed around compile-time containers, algorithms, and iterators. In the same way, Boost.Fusion provides algorithms to work both at compile time and at runtime on tuple-like structures.

For some reason, even with those familiar interfaces, metaprogramming tools continued to be used by experts and were often overlooked and considered unnecessarily complex. The compilation time of metaprograms was also often criticized as hindering a normal, runtime-based development process.

Most of the critiques you may have heard about template metaprogramming stem from this limitation—which no longer applies, as we will see in the rest of this report.

Checking Integer Type

Some programs or libraries obfuscate the underlying integer type of a variable. Having a compilation error if your assumption is wrong is a great way to prevent difficult-to-track errors:

static_assert(std::is_same<obfuscated_int,
                           std::uint32_t>::value, 
              "invalid integer detected!");

If your obfuscated integer isn’t an unsigned 32-bit integer, your program will not compile and the message “invalid integer detected” will be printed.

You might not care about the precise type of the integer—maybe just the size is important. This check is very easy to write:

static_assert(sizeof(obfuscated_int) == 4, 
    "invalid integer size detected!");

Checking the Memory Model

Is an integer the size of a pointer? Are you compiling on a 32-bit or 64-bit platform? You can have a compile-time check for this:

static_assert(sizeof(void *) == 8, "expected 64-bit platform");

In this case, the program will not compile if the targeted platform isn’t 64-bit. This is a nice way to detect invalid compiler/platform usage.

We can, however, do better than that and build a value based on the platform without using macros. Why not use macros? A metaprogram can be much more advanced than a macro, and the error output is generally more precise (i.e., you will get the line where you have the error, whereas with preprocessor macros this is often not the case).

Let’s assume that your program has a read buffer. You might want the value of this read buffer to be different if you are compiling on a 32-bit platform or a 64-bit platform because on 32-bit platforms you have less than 3 GB of user space available.

The following program will define a 100 MB buffer value on 32-bit platforms and 1 GB on 64-bit platforms:

static const std::uint64_t default_buffer_size = 
  std::conditional<sizeof(void *) == 8,
    std::integral_constant<std::uint64_t, 100 * 1024 * 1024>,
    std::integral_constant<std::uint64_t, 1024 * 1024 * 1024>
>::type::value;

Here’s what the equivalent in macros would be:

#ifdef IS_MY_PLATFORM_64
static const std::uint64_t default_buffer_size
    = 100 * 1024 * 1024;
#else
static const std::uint64_t default_buffer_size
    = 1024 * 1024 * 1024;
#endif

The macros will silently set the wrong value if you have a typo in the macro value, if you forget a header, or if an exotic platform on which you compile doesn’t have the value properly defined.

Also, it is often very difficult to come up with good macros to detect the correct platform (although Boost.Predef has now greatly reduced the complexity of the task).