15.1.1 Predictor and predicted variables

Suppose we want to predict someone's weight from their height. In this case, weight is the predicted variable and height is the predictor. Or, suppose we want to predict high school grade point average (GPA) from Scholastic Aptitude Test (SAT) score and family income. In this case, GPA is the predicted variable, while SAT score and income are predictor variables. Or, suppose we want to predict the blood pressure of patients who either take drug A, or take drug B, or take a placebo, or merely wait. In this case, the predicted variable is blood pressure, and treatment category is the predictor.

The key mathematical difference between predictor and predicted variables is that the likelihood function expresses the probability of values of the predicted variable as a function ofvalues of the predictor variable. The likelihood function does not describe the probabilities ofvalues of the predictor variable. The value of the predictor variable comes from outside the system being modeled, whereas the value of the predicted variable depends on the value of the predictor variable.

Because the predicted variable depends on the predictor variable, at least mathematically in the likelihood function if not causally in the real world, the predicted variable can also be called the “dependent” variable. The predictor variables are sometimes called “independent” variables. The key conceptual difference between independent and dependent variables is that the value of the dependent variable depends on the value of the independent variable. The term “independent” can be confusing because it can be used strictly or loosely. In experimental settings, the variables that are actually manipulated and set by the experimenter are the independent variables. In this context ofexperimental manipulation, the values of the independent variables truly are (in principle, at least) independent of the values of other variables, because the experimenter has intervened to arbitrarily set the values of the independent variables. But sometimes a non-manipulated variable is also referred to as “independent,” merely as a way to indicate that it is being used as a predictor variable.

Among non-manipulated variables, the roles of predicted and predictor are arbitrary, determined only by the interpretation of the analysis. Consider, for example, people's weights and heights. We could be interested in predicting a person?s weight from his or her height, or we could be interested in predicting a person?s height from his or her weight. Prediction is merely a mathematical dependency, not necessarily a description of underlying causal relationship. Although height and weight tend to co-vary across people, the two variables are not directly causally related. When a person slouches, thereby getting shorter, she does not lose weight. And when a person drinks a glass of water, thereby weighing more, she does not get taller.

Just as “prediction” does not imply causation, “prediction” also does not imply any temporal relation between the variables. For example, we may want to predict a person?s sex, male or female, from his or her height. Because males tend to be taller than females, this prediction can be made with better-than-chance accuracy. But a person?s sex is not caused by his or her height, nor does a person?s sex occur only after their height is measured. Thus, we can “predict” a person?s sex from his or her height, but this does not mean that the person?s sex occurred later in time than his or her height.

In summary, all manipulated independent variables are predictor variables, not predicted. Some dependent variables can take on the role of predictor variables, if desired. All predicted variables are dependent variables. The likelihood function specifies the probability of values of the predicted variables as a function of the values of the predictor variables.

Why we care: We care about these distinctions between predicted and predictor variables because the likelihood function is a mathematical description of the dependency of the predicted variable on the predictor variable. The first thing we have to do in statistical inference is identify what variables we are interested in predicting, on the basis of what predictors. As you should recall from Section 2.3, p. 25, the first step of Bayesian data analysis is to identify the data relevant to the analysis, and which variables are predictors and which variable is predicted.

15.1.2 Scale types: metric, ordinal, nominal, and count

Items can be measured on different scales. For example, the participants in a foot race can be measured either by the time they took to run the race, or by their placing in the race (first, second, third, etc.), or by the name of the team they represent. These three measurements are examples of metric, ordinal, and nominal scales, respectively (Stevens, 1946).

Examples of metric scales include response time (i.e., latency or duration), temperature, height, and weight. Those are actually cases of a specific type of metric scale called a ratio scale, because they have a natural zero point on the scale. The zero point on the scale corresponds to there being a complete absence of the stuff being measured. For example, when the duration is zero, there has been no time elapsed, and when the weight is zero, there is no downward force. Because these scales have a natural zero point, it is meaningful to talk about ratios of amounts being measured, and that is why they are called ratio scales. For example, it is meaningful to say that taking 2 min to solve a problem is twice as long as taking 1 min to solve the problem. On the other hand, the scale of historical time has no known absolute zero. We cannot say, for example, that there is twice as much time in January 2nd as there is time in January 1st. We can refer to the duration since some arbitrary reference point, but we cannot talk about the absolute amount of time in any given moment. Scales that have no natural zero are called interval scales because all we know about them is the amount of stuff in an interval on the scale, not the amount of stuff at a point on the scale. Despite the conceptual difference between ratio and interval scales, I will lump them together into the category of metric scales.

A special case of metric-scaled data is count data, also called frequency data. For example, the number of cars that pass through an intersection during an hour is a count. The number of poll respondents who say they belong to a particular political party is a count. Count data can only have values that are nonnegative integers. Distances between counts have meaning, and therefore the data are metric, but because the data cannot be negative and are not continuous, they are treated with different mathematical forms than continuous, real-valued metric data.

Examples of ordinal scales include placing in a race, or rating of degree of agreement. When we are told that, in a race, Jane came in first, Jill came in second, and Jasmine came in third, we only know the order. We do not know whether Jane beat Jill by a nose or by a mile. There is no distance or metric information in an ordinal scale. As another example, many polls have ordinal response scales: Indicate how much you agree with this statement: “Bayesian statistical inference is better than null hypothesis significance testing,” with 5 = strongly agree, 4 = mildly agree, 3 = neither agree nor disagree, 2 = mildly disagree, and 1 = strongly disagree. Notice that there is no metric information in the response scale, because we cannot say the difference between ratings of 5 and 4 is the same amount of difference as between ratings of 4 and 3.

Examples of nominal, a.k.a. categorical, scales include political party affiliation, the face of a rolled die, and the result of a flipped coin. For nominal scales, there is neither distance between categories nor order between categories. For example, suppose we measure the political party affiliation of a person. The categories of the scale might be Green, Democrat, Republican, Libertarian, and Other (in the United States, with different parties in different countries). While some political theories might suggest that the parties fall on an underlying liberal-conservative scale, there is no such scale in the actual categorical values themselves. In the actual categorical labels there is neither distance nor ordering. As another example of values on a nominal scale, consider eye color and hair color, as in Table 4.1, p. 90. Both eye color and hair color are nominal variables because they have neither distance between levels nor ordering of levels. The cells of Table 4.1 show count data that may be predicted from the nominal color predictors, as will be explored in Chapter 24.

In summary, if two items have different nominal values, all we know is that the two items are different (and what categories they are in). On the other hand, if two items have different ordinal values, we know that the two items are different and we know which one is “larger” than the other, but not how much larger. If two items have different metric values, then we know that they are different, which one is larger, and how much larger.

Why we care: We care about the scale type because the likelihood function must specify a probability distribution on the appropriate scale. If the scale has two nominal values, then a Bernoulli likelihood function may be appropriate. If the scale is metric, then a normal distribution may be appropriate as a probability distribution to describe the data. Whenever we are choosing a model for data, we must answer the question, What kind of scale are we dealing with? As you should recall from Section 2.3, p. 25, the first step of Bayesian data analysis includes identifying the measurement scales of the predictor and predicted variables.

In the following sections, we will first consider the case of a metric-predicted variable with metric predictors. In that context of all metric variables, we will develop the concepts of linear functions and interactions. Once those concepts are established for metric predictors, the notions will be extended to nominal predictors.

15.2.1 Linear function of a single metric predictor

Suppose we have identified one variable to be predicted, which we?ll call y, and one variable to be the predictor, which we?ll call x. Suppose we have determined that both variables are metric. The next issue we need to address is how to model a relationship between x and y. This is now Step 2 of Bayesian data analysis from Section 2.3, p. 25. There are many possible dependencies of y on x, and the particular form of the dependency is determined by the specific meanings and nature of the variables. But in general, across all possible domains, what is the most basic or simplistic dependency of y on x that we mightconsider? The usual answer to this question is: a linear relationship. A linear function is the generic, “vanilla,” off-the-shelf dependency that is used in statistical models.

Linear functions preserve proportionality (relative to an appropriate baseline). If you double the input, then you double the output. If the cost of a candy bar is a linear function of its weight, then when the weight is reduced 10%, the cost should be reduced 10%. If automobile speed is a linear function of fuel injection to the engine, then when you press the gas pedal 20% further, the car should go 20% faster. Nonlinear functions do not preserve proportionality. For example, in actuality, car speed is not a linear function of the amount of fuel consumed. At higher and higher speeds, it takes proportionally more and more gas to make the car go faster. Despite the fact that many real-world dependencies are nonlinear, most are at least approximately linear over moderate ranges of the variables. For example, If you have twice the wall area, it takes approximately twice the amount of paint. It is also the case that linear relationships are intuitively prominent (Brehmer, 1974; P.J. Hoffman, Earle, & Slovic, 1981; Kalish, Griffiths, & Lewandowsky, 2007). Linear relationships are the easiest to think about: Turn the steering wheel twice as far, and we believe that the car should turn twice as sharp. Turn the volume knob 50% higher, and we believe that the loudness should increase 50%.

The general mathematical form for a linear function of a single variable is

$y={\beta }_0+{\beta }_{1}x$

When values of x and y that satisfy Equation 15.1 are plotted, they form a line. Examples are shown in Figure 15.1. The value of parameter β₀ is called the y-intercept because it is where the line intersects the y-axis when x = 0. The left panel of Figure 15.1 shows two lines with different y-intercepts. The value of parameter β₁ is called the slope because it indicates how much y increases when x increase by 1. The right panel of Figure 15.1 shows two lines with the same intercept but different slopes.

In strict mathematical terminology, the type of transformation in Equation 15.1 is called affine. When β ≠ 0, the transformation does not preserve proportionality. For example, consider y = 10 + 2x. When x is doubled from x = 1 to x = 2, y increases from y = 12 to y = 14, which is not doubling y. Nevertheless, the rate of increase in y is the same for all values of x: Whenever x increases by 1, y increases by β₁. Moreover, Equation 15.1 can have either y or x shifted so that proportionality is achieved. If y is simply shifted by β₀ and the shifted value is called y^∗, then proportionality is achieved: y^∗ = y—β₀ = β₁x. Or, if x is shifted by — β₀/β₁ and the shifted value is called x^∗, then proportionality is achieved: y=β₁(x-β₀/β₁)=β₁x^∗. This form of the equation is called x-intercept form or x-threshold form, because —β₀/β₁ is where the line crosses the x-axis, and is the threshold between negative and positive values of y.

Summary of why we care. The likelihood function includes the form of the dependency of y on x. When y and x are metric variables, the simplest form of dependency, both mathematically and intuitively, is one that preserves proportionality. The mathematical expression of this relation is a so-called linear function. The usual mathematical expression of a line is the y-intercept form, but sometimes a more intuitive expression is the x threshold form. Linear functions form the core of the GLM.

15.2.2 Additive combination of metric predictors

If we have more than one predictor variable, what function should we use to combine the influences of all the predictor variables? If we want the combination to be linear in each of the predictor variables, then there is just one answer: Addition. In other words, if we want an increase in one predictor variable to predict the same increase in the predicted variable for any value of the other predictor variables, then the predictions of the individual predictor variables must be added.

In general, a linear combination of K predictor variables has the form

$\begin{array}{c}y={\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_{K^{x}K}\\ ={\beta }_0+{\displaystyle \sum _{k=1}^K{\beta }_{k^{x}k}}\hfill \end{array}$

Figure 15.2 shows examples of linear functions of two variables, x₁ and x₂. The graphs show y plotted only over a the domain with 0 ≤ x₁ ≤ 10 and x₂ ≤ 10. It is important to realize that the plane extends from minus to plus infinity, and the graphs only show a small region. Notice in the upper left panel, where y = 0 + 1x₁ + 0x₂,the plane tilts upward in the x₁ direction, but the plane is horizontal in the x₂ direction. For example, when x₁= 10 then y = 10 regardless of the value of x₂ The opposite is true in the upper right panel, where y = 0+x₁ + 2x₂. In this case, the plane tilts upward in the x₂ direction, but the plane is horizontal in the x₁ direction. The lower left panel shows the two influences added: y = 0+x₁ + 2x₂. Notice that the slope in the x₂ direction is steeper than in the x₁ direction. Most importantly, notice that the slope in the x₂ direction is the same at any specific value of x₁. For example, when x₁ is fixed at 0, then y rises from y = 0to y = 20 when x₂ goes from x₂ = 0to x₂ = 10. When x₁ is fixed at 10, then y again rises 20 units, from y = 10 to y = 30.

Summary of section: When the influence of every individual predictor is unchanged by changing the values of other predictors, then the influences are additive. The combined influence of two or more predictors can be additive even if the individual influences are nonlinear. But if the individual influences are linear, and the combined influence is additive, then the overall combined influence is also linear. The formula of Equation 15.2 is one expression of a linear model, which forms the core of the GLM.

15.2.3 Nonadditive interaction of metric predictors

The combined influence of two predictors does not have to be additive. Consider, for example, a person?s self-rating of happiness, predicted from his or her overall health and annual income. It?s likely that if a person?s health is very poor, then the person is not happy, regardless of his or her income. And if the person has zero income, then the person is probably not happy, regardless of his or her health. But if the person is both healthy and rich, then the person has a higher probability of being happy (despite celebrated counter-examples in the popular media).

A graph of this sort of nonadditive interaction between predictors appears in the upper left panel of Figure 15.3. The vertical axis, labeled y, is happiness. The horizontal axes, x₁ and x₂, are health and income. Notice that if either x₁= 0 orx₂= 0, then y= 0. But if both x₁ > 0and x₂x₂ 0, then y> 0. The specific form of interaction plotted here is multiplicative: y = 0+0x₁+0x₂+0.2x₁x₂. For comparison, the upper right panel of Figure 15.3 shows a non-interactive (i.e., additive) combination of x₁ and x₂. Notice that the graph of the interaction has a twist or curvature in it, but the graph of the additive combination is flat.

The lower left panel of Figure 15.3 shows a multiplicative interaction in which the individual predictors increase the outcome, but the combined variables decrease the outcome. A real-world example of this occurs with some drugs: Individually, each of two drugs might improve symptoms, but when taken together, the two drugs might interact and cause a decline in health. As another example, consider lighter-than-air travel in the form of ballooning. The levity of a balloon can be increased by fire, as in hot air balloons. And the levity of a balloon can be increased by hydrogen, as in many early twentieth century blimps and dirigibles. But the levity of a balloon is dramatically decreased by the combination of fire and hydrogen.

The lower right panel of Figure 15.3 shows a multiplicative interaction in which the direction of influence of one variable depends on the magnitude of the other variable. Notice that when x₂=0, then an increase in the x₁ variable leads to a decline in y. But whenx₂, then an increase in the x₂ variable leads to an increase in y. Again, the graph of the interaction shows a twist or curvature; the surface is not flat.

There is a subtlety in the use of the term “linear” that can sometimes cause confusion in this context. The interactions shown in Figure 15.3 are not linear on the two predictors x₁ and x₂. But if the product of the two predictors, x₁x₂, is thought of as a third predictor, then the model is linear on the three predictors, because the predicted value of y is a weighted additive combination of the three predictors. This reconceptualization can be useful for implementing nonlinear interactions in software for linear models, but we will not be making that semantic leap to a third predictor, and instead we will think of a nonadditive combination of two predictors.

A nonadditive interaction of predictors does not have to be multiplicative. Other types of interaction are possible. The type of interaction is motivated by idiosyncratic theories for different variables in different application domains. Consider, for example, predicting the magnitude of gravitational force between two objects from three predictor variables: the mass of object one, the mass of object two, and the distance between the objects. The force is proportional to the multiplicative product of the two masses divided by the squared distance between them.

After the predictor variables are combined, they need to be mapped to the predicted variable. This mathematical mapping is called the (inverse) link function, and is denoted by f() in the following equation:

$y=f\text{?}(\text{lin}(x))$

Until now, we have been assuming that the link function is merely the identity function, f(lin(x)) = lin(x). For example, in Equation 15.9, y equals the linear combination of the predictors; there is no transformation of the linear combination before mapping the result to y.

Before describing different link functions, it is important to make some clarifications ofterminology and corresponding concepts. First, the function f() in Equation 15.11 is called the inverse link function, because the link function is traditionally thought of as transforming the value y into a form that can be linked to the linear model. That is, the link function goes from y to the predictors, not from the predictors to y. Imay occasionally abuse convention and refer to either f() or f⁻¹() as “the” link function, and rely on context to disambiguate which direction of linkage is intended. The reason for this terminological sadism is that the arrows in hierarchical diagrams of Bayesian models will flow from the linear combination toward the data, and therefore it is natural for the functions to map toward the predicted values, as in Equation 15.11. But repeatedly referring to this function as the “inverse” link would strain my patience and violate my aesthetic sensibilities. Second, the value y that results from the link function f(lin(x)) is not a data value per se. Instead,f(lin(x)) is the value of a parameter that expresses a central tendency of the data, usually their mean. Therefore the function f() in Equation 15.11 is sometimes called the mean function (instead of the inverse link function), and is written μ = f() instead of y = f(). I will use this notation temporarily for some summary tables below, but then abandon it because f() does not always refer to the mean of the predicted data.

There are situations in which a non-identity link function is appropriate. Consider, for example, predicting response time as a function of amount of caffeine consumed. Response time decreases as caffeine dosage increases (e.g., Smit & Rogers, 2000, Fig. 1; although most of the decrease is produced by even a small dose). Therefore a linear prediction of RT (y)from dosage(x) would have a negative slope. This negative slope on a linear function implies that for a very large dosage of caffeine, response time would become negative, which is impossible unless caffeine causes precognition (i.e., foreseeing events before they occur). Therefore a linear function cannot be used for extrapolation to large doses, and we might instead want to use a link function that asymptotes above zero, such as an exponential function with y = exp + (β₀+β₁x). In the next sections we will consider some frequently used link functions.

15.3.1.1 The logistic function

A frequently used link function is the logistic:

$y=\text{logistic}\left(x\right)=1/\left(1+\exp \left(-x\right)\right)$

  (15.12)

Notice the negative sign in front of the x. The value y of the logistic function ranges between 0 and 1. The logistic is nearly 0 when x is large negative, and is nearly 1 when x is large positive. In our applications, x is a linear combinations of predictors. For a single metric predictor, the logistic can be written:

$y=\text{logistic}\left(x;\text{?}{\beta }_0,\text{?}{\beta }_1\right)=1/\left(1+\exp \left(-\left({\beta }_0+{\beta }_{1}x\right)\right)\right)$

  (15.13)

The logistic function is often most conveniently parameterized in a form that shows the gain γ (Greek letter gamma) and threshold θ:

$y=\text{logistic}\left(x;\text{?}\gamma ,\text{?}\theta \right)=1/\left(1+\exp \left(-\gamma \left(x-\theta \right)\right)\right)$

  (15.14)

Examples of Equation 15.14 are shown in Figure 15.6. Notice that the threshold is the point on the x-axisfor which y = 0.5. The gain indicates how steeply the logistic rises through that point.

Figure 15.7 shows examples of a logistic of two predictor variables. Above each panel is the equation for the corresponding graph. The equations are parameterized in normalized-threshold form: $y=\text{logistic}\left(\gamma \left({\displaystyle {\sum }_k{w}_k}{x}_k-\theta \right)\right)$, with ${\left({\displaystyle {\sum }_k{w}_k^2}\right)}^{1/2}=1$. Notice, in particular, that the coefficients of x₁ and x₂ in the plotted equations do indeed have Euclidean length of 1.0. For example, in the upper right panel, (0.71² + 0.71²)^1/2 = 1.0, except for rounding error. If you start with the linear part parameterized as ${\beta }_0+{\displaystyle {\sum }_{k=1}^K{\beta }_k{x}_k}$ you can convert to the equivalent normalized-threshold form as follows: Let $\gamma ={\left({\displaystyle {\sum }_{k=1}^K{\beta }_k^2}\right)}^{1/2},\text{?}\theta =-{\beta }_0/\gamma ,$,and $W_k={\beta }_k/\gamma$.

The coefficients of the x variables determine the orientation of the logistical “cliff.” For example, compare the two top panels in Figure 15.7, which differ only in the coefficients, not in gain or threshold. In the top left panel, the coefficients are W₁ = 0 and W₂ = 1, and the cliff rises in the x₂ direction. In the top right panel, the coefficients are W₁ = 0.71 and W₂ = 0.71, and the cliff rises in the positive diagonal direction.

The threshold determines the position of the logistical cliff. In other words, the threshold determines the x values for which y = 0.5. For example, compare the two left panels of Figure 15.7. The coefficients are the same, but the thresholds (and gains) are different. In the upper left panel, the threshold is zero, and therefore the mid-level of the cliff is over x₂ = 0. In the lower left panel, the threshold is −3, and therefore the mid-level of the cliff is over x₂ = −3.

The gain determines the steepness of the logistical cliff. Again compare the two left panels of Figure 15.7. The gain of the upper left is 1, whereas the gain of the lower left is 2.

Terminology: The logit function. The inverse of the logistic function is called the logit function. For 0 < p < 1, logit(p) = log (p/(1 − p)). It is easy to show(try it!) that logit(logistic(x)) = x, which is to say that the logit is indeed the inverse of the logistic. Some authors, and programmers, prefer to express the connection between predictors and predicted in the opposite direction, by first transforming the predicted variable to match the linear model. In other words, you may see the link expressed either of these ways:

$\begin{array}{lll}\hfill y& =\hfill & \text{logistic}\left(\text{lin}\left(x\right)\right)\hfill \\ \hfill \text{logit}\left(y\right)& =\hfill & \text{lin}\left(x\right)\hfill \end{array}$

  (15.15)

The two expressions achieve the same result, mathematically. The difference between them is merely a matter of emphasis. In the first expression, the combination of predictors is transformed so it maps onto y. In the second expression, y is transformed onto a new scale, and that transformed value is modeled as a linear combination of predictors. I find that the logistic formulation is usually more intuitive than the logit formulation, but we will see in Chapter 21 that the logit formulation will be useful for interpreting the β coefficients.

15.3.1.2 The cumulative normal function

Another frequently used link function is the cumulative normal distribution. It is qualitatively very similar to the logistic function. Modelers will use the logistic or the cumulative normal depending on mathematical convenience or ease of interpretation. For example, when we consider ordinal predicted variables (in Chapter 23), it will be natural to model the responses in terms of a continuous underlying variable that has normally distributed variability, which leads to using the cumulative normal as a model of response probabilities.

The cumulative normal is denoted Φ(x; μ, σ, where x is a real number and where μ and σ are parameter values, called the mean and standard deviation of the normal distribution. The parameter μ governs the point at which the cumulative normal, Φ(x), equals 0.5. In other words, μ plays the same role as the threshold in the logistic. The parameter σ governs the steepness of the cumulative normal function at x = μ, but inversely, such that a smaller value of σ corresponds to a steeper cumulative normal. A graph of a cumulative standardized normal appears in Figure 15.8.

Terminology: The inverse of the cumulative normal is called the probit function. (“Probit” stands for “probability unit”; Bliss, 1934). The probit function maps a value p,for $0.0\text{?}\le \text{?}p\text{?}\le 1.0$, onto the infinite real line, and a graph of the probit function looks very much like the logit function. You may see the link expressed either of these ways:

$\begin{array}{c}y=\Phi (\mathrm{lin}(x))\\ \text{probit}\left(\gamma \right)=\text{lin}(x)\end{array}$

Traditionally, the transformation of y (in this case, the probit function) is called the link function, and the transformation of the linear combination of x (in this case, the Φ function) is called the inverse link function. As mentioned before, I abuse the traditional terminology and call either one a link function, relying on context to disambiguate. In later applications, we will be using the $ function, not the probit.

15.3.2 From predicted central tendency to noisy data

In the real world, there is always variation in y that we cannot predict from x. This unpredictable “noise” in y might be deterministically caused by sundry factors we have neither measured nor controlled, or the noise might be caused by inherent non-determinism in y. It does not matter either way because in practice the best we can do is predict the probability that y will have any particular value, dependent upon x.Therefore we use the deterministic value predicted by Equation 15.11 as the predicted tendency of y as a function of the predictors. We do not predict that y is exactly f(lin(x)) because we would surely be wrong. Instead, we predict that y tends to be near f(lin(x)).

To make this notion of probabilistic tendency precise, we need to specify a probability distribution for y that depends on f(lin(x)). To keep the notation tractable, first define μ = f(lin(x)). The value μ represents the central tendency of the predicted y values, which might or might not be the mean. With this notation, we then denote the probability distribution of y as some to-be-specified probability density function, abbreviated as “pdf”:

$y~\text{pdf}\left(\mu ,\text{?}\left[\text{scale, shape, etc.}\right]\right)$

As indicated by the bracketed terms after μ, the pdf might have various additional parameters that control the distribution?s scale (i.e., standard deviation), shape, etc.

The form of the pdf depends on the measurement scale of the predicted variable. If the predicted variable is metric and can extend infinitely in both positive and negative directions, then a typical pdf for describing the noise in the data is a normal distribution. A normal distribution has a mean parameter μ and a standard deviation parameter σ, so we would write y ~ normal(μ, σ) with μ = f(lin(x)). In particular, if the link function is the identity, then we have a case of conventional linear regression. Examples of linear regression are shown in Figure 15.9. The upper panel of Figure 15.9 shows a case in which there is a single metric predictor, and the lower panel shows a case with two metric predictors. For both cases, the black dots indicate data that are normally distributed around a linear function of the predictors.

If the predicted variable is dichotomous, with y ∈ {0, 1}, then a typical pdf is the Bernoulli distribution, y ~ Bernoulli(μ), where μ = f(lin(x)) is the probability that y = 1. In other words, μ is playing the role of the parameter that was called θ in Equation 5.10, p. 109. Because μ must be between 0 and 1, the link function must convert lin(x) to a value between 0 and 1. A typical link function for this purpose is the logistic function, and when the predictor variables are metric, we have a case of conventional logistic regression. An example of logistic regression is shown in Figure 15.10. The black dots indicate data that are Bernoulli distributed, at 0 or 1, with probability indicated by the logistic surface.

Table 15.2 shows a summary of typical link functions and pdfs for various types of predicted variables. The previous paragraphs provided examples for the first two rows of Table 15.2, which correspond to predicted variables that are metric or dichotomous. The remaining rows of Table 15.2 are for predicted variables that are nominal, ordinal, or count. Each of these cases will be explained at length in later chapters. The point now is for you to notice that in each case, there is a linear combination of the predictors converted by an inverse link function in the right column to a predicted central tendency μ, which is then used in a pdf in the middle column.

Table 15.2

For the generalized linear model: typical noise distributions and inverse link functions for describing various scale types of the predicted variable y

Scale type of predicted y	Typical noise distribution y ~ pdf (μ, [parameters])	Typical inverse link function μ = f(lin(x), [parameters])
Metric	y ~ normal(μ, σ)	μ = lin(x)
Dichotomous	y ~ bernoulli(μ)	μ = logistic (lin(x))
Nominal	y ~ categorical(…, μk, …)	${\mu }_k=\frac{\exp \left({\text{lin}}_{\text{k}}\left(x\right)\right)}{{\displaystyle {\sum }_c\exp \left(li{n}_c\left(x\right)\right)}}$
Ordinal	y ~ categorical(…, μk, …)	μk=Φ((θk-lin(x))/σ)-Φ((θk-1-lin(x))/σ)
Count	y ~ poisson(μ)	μ = exp (lin(x))

The value μ is a central tendency of the predicted data (not necessarily the mean). The predictor variable is x, and lin(x) is a linear function ofx, such as those shown in Table 15.1.

The forms shown in Table 15.2 are merely typical and not necessary. For example, if metric data are skewed or kurtotic (i.e., have heavy tails) then a non-normal pdf could be used that better describes the noise in the data. Furthermore, you might have realized that the case of a dichotomous scale is subsumed by the case of a nominal scale. But the dichotomous case is encountered often, and its model is generalized to the nominal scale, so the dichotomous case is treated separately.

15.4.1 Cases of the GLM

Table 15.3 displays the various cases of the GLM that are considered in this book, along with the chapters that address them. Each cell of Table 15.3 corresponds to a combination of lin(x) from Table 15.1 and link with pdf from Table 15.2. The chapter for each combination provides conceptual explanation of the descriptive mathematical model and of how to do Bayesian estimation of the parameters in the model.

The first row of Table 15.3 lists cases for which the predicted variable is metric. Moving from left to right within this row, the first two columns are for when there are one or two groups of data. Classical NHST would apply a single-group or two-group t test to these cases. This case is described in its Bayesian setting in Chapter 16. Moving to the next column, there is a single metric predictor. This corresponds to so-called “simple linear regression,” and is explored in Chapter 17. Moving rightward to the next column, we come to the scenario involving two or more metric predictors, which corresponds to “multiple linear regression” and is explored in Chapter 18. The next two columns involve nominal predictors, instead of metric predictors, with the penultimate column devoted to a single predictor and the final column devoted to two or more predictors. The last two columns correspond to what NHST calls “one-way ANOVA” and “multifactor ANOVA.” Bayesian approaches are explained in Chapters 19 and 20. For both the Bayesian approaches to linear regression and the Bayesian approaches to ANOVA, we will use hierarchical models that are not used in classical approaches.

In the second row of Table 15.3, the predicted variable is dichotomous. This simplest of data scales was used to develop all the foundational concepts of Bayesian data analysis in Chapters 6-9. When the predictors are more elaborate, and especially when the predictors are metric, this situation is referred to as “logistic regression” because of the logistic (inverse) link function. It is discussed in Chapter 21.

The next two rows of Table 15.3 are for nominal and ordinal scales on the predicted variable. Both of these will use a categorical noise distribution, but with probabilities computed through different link functions. For nominal predicted values, Chapter 22 will explain how logistic regression is generalized to address multiple categories. For ordinal predicted values, Chapter 23 will explain how an underlying metric scale is mapped to an ordinal scale using thresholds on a cumulative normal distribution.

Finally, the bottom row of Table 15.3 refers to a predicted variable that measures counts. We have previously seen this sort of data in the counts of eye color and hair color in Table 4.1, p. 90. For this situation, we will consider a new pdf called the Poisson distribution, which requires a positive-valued mean parameter. After all, counts are nonnegative. Because the mean must be positive, the inverse link function must provide a positive value. A natural function that satisfies that requirement is the exponential, as shown in Table 15.2. Because the inverse link function is the exponential, the link function is the logarithm, and therefore these are called “log-linear models.” Chapter 24 provides an introduction.

How to be your own statistical consultant: The key point to understand is that each cell of Table 15.3 corresponds to a combination of lin(x) from Table 15.1 and link with pdf from Table 15.2. This organization is well known to every statistical consultant. When a client brings an application to a consultant, one of the first things the consultant does is find out from the client which data are supposed to be predictors and which data are supposed to be predicted, and the measurement scales of the data. Very often the situation falls into one of the cells of Table 15.3, and then the standard models can be applied. A benefit of Bayesian analysis is that it is easy to create nonstandard variations of the models, as appropriate for specific situations. For example, if the data have many outliers, it is trivial to use a heavy-tailed pdf for the noise distribution. If there are lots of data for each individual, and there are many individuals, perhaps in groups, it is usually straight forward to create hierarchical structure on the parameters of the GLM. When you are considering how to analyze data, your first task is to be your own consultant and find out which data are predictors, which are predicted, and what measurement scales they are. Then you can determine whether one of the cases of the GLM applies to your situation. This also constitutes the first steps of Bayesian data analysis, described back in Section 2.3, p. 25.