A SAS program can be any combination of the following elements:

When you submit a SAS program, the code is copied to a memory location called the input stack. The presence of text in the input stack triggers a component called the word scanner to begin its work.

The word scanner has two major functions. First, it pulls the raw text from the input stack character by character and transforms it into tokens. Second, it sends tokens for processing to the compiler and macro processor. A program is then separated into components called tokens. There are four types of tokens: name, number, special, and literal.

To build a token, the word scanner extracts characters until it reaches a delimiter, or until the next character does not meet the rules of the current token. A delimiter is any whitespace character such as a space, tab, or end-of-line character.

Name tokens consist of a maximum of 32 characters, must begin with a letter or underscore, and can include only letter, digit, and underscore characters.

Number tokens define a SAS floating-point numeric value. They can consist of a digit, decimal point, leading sign, and exponent indicator (e or E). Date, time, and datetime specifications also become number tokens (for example: '29APR2019'd, '14:05:32.1't, '29APR2019 14:05:32.1'dt).

Literal tokens consist of a string of any characters enclosed in single or double quotation marks. They can contain up to 32,767 characters and are handled as a single unit.

Special tokens are made up of any character or group of characters that have special meaning in the SAS language. Examples include * / + - ; ( ) . & %

Knowing how tokenization works helps you understand how the various parts of SAS and the macro processor work together. Understanding differences in timing between macro processing and SAS code compilation and execution is especially important.

Between the input stack and the compiler, SAS programs are tokenized into smaller pieces.

The following example illustrates how the input stack, word scanner, and compiler work together.

title "MPG City Over 25";
proc print data=sashelp.cars noobs;
   var Make Model Type MPG_City MPG_Highway MSRP;
   where MPG_City>25;
run;

When the code is copied to the input stack, the word scanner retrieves one character at a time until it reaches the first delimiter, a blank. When TITLE is recognized as a name token, the word scanner tags it and passes it to the compiler.

The word scanner tags the double quotation mark as the start of a literal token.

It then retrieves, tokenizes, and holds additional text until it retrieves another double quotation mark. It passes the text as a single literal token to the compiler, and then tokenization continues. The semicolon is a special token, and the end-of-line character is a delimiter.

The semicolon is sent to the compiler, ending the TITLE statement. The compiler checks the syntax, and because TITLE is a global statement, it is executed immediately. The tokenization process continues with the PROC PRINT step. The compiler performs a syntax check at the end of each statement.

The code executes when it encounters a step boundary, in this case the RUN statement.

The macro facility includes a macro processor that is responsible for handling all macro language elements. Certain token sequences, known as macro triggers, alert the word scanner that the subsequent code should be sent to the macro processor.

The word scanner recognizes the following token sequences as macro triggers:

When a macro trigger is detected, the word scanner passes it to the macro processor for evaluation. Here is the sequence that the macro processor follows:

For macro variables, the processor does one of the following:

The word scanner then resumes processing tokens from the input stack.