Choosing Between GLM and GLMM: Core Decision Points
This article explains when to use a generalized linear model (GLM) versus a generalized linear mixed model (GLMM). It covers how the two models relate to ordinary linear and linear mixed models, their mathematical components, two worked examples, conditional versus marginal interpretation, and a step-by-step workflow for checking dependence, distributional assumptions, overdispersion, group size, and prediction goals.
1. Where the Problem Starts: When Normality and Independence Break Down
Classical linear regression rests on two default assumptions: the response is normally distributed, and observations are independent of one another. Both often fail in practice:
- Count responses: daily new case counts in a region, or hourly requests to an API, take non-negative integer values, and the variance often changes with the mean.
- Proportion responses: a sample positivity rate, or a page click-through rate, is confined to the interval from 0 to 1.
- Positive continuous responses: catch per unit time, session duration, or transaction amount are typically right-skewed, with variance that grows as the mean grows.
- Correlated observations: departments in the same hospital, repeated measures on the same subject, or adjacent dates in the same region are naturally similar within groups.
Statistics developed two extensions for these settings:
- Generalized linear models (GLMs) keep the framework of a linear predictor plus a parametric distribution. A link function relates the mean of the response to the linear predictor, so the response need not be normal. A GLM mainly addresses a non-normal response.
- Generalized linear mixed models (GLMMs) add random effects on top of a GLM, explicitly describing correlation induced by groups, individuals, or repeated measures. A GLMM mainly addresses non-independence.
The relationship is inclusion and extension, not mutual exclusion. Which model to use depends on two questions: what distribution the response follows, and whether observations are independent.
2. Family Tree: A Symmetric Structure of Four Models
Placing GLM and GLMM back in the broader model family makes the relationships clearer:
| Observations independent | Grouped or repeated correlation | |
|---|---|---|
| Normal response, identity link | Ordinary linear model (LM) | Linear mixed model (LMM) |
| Non-normal response, nonlinear link | Generalized linear model (GLM) | Generalized linear mixed model (GLMM) |
The table can be summarized in one idea:
- A GLMM stands to an LMM as a GLM stands to an LM: the former adds random effects to the latter.
- A GLMM stands to a GLM as an LMM stands to an LM: the former relaxes the independence assumption.
Any two models that sit next to each other horizontally or vertically differ by only one step. Once the response is not normal, move to the “generalized” row; then decide whether observations are independent, which places the model in the left column (GLM) or the right column (GLMM).
When the random-effect structure is empty, a GLMM reduces to a GLM. A GLM is therefore the special case of a GLMM in which within-group correlation is zero.
3. The Mathematical Skeleton of a GLM: Three Components
A GLM reuses the basic formula of a linear model. It can be viewed as a transformation of the mean of the response:
Linear model: E(Y_i) = x_i^T β
Generalized linear model: g(E(Y_i)) = g(μ_i) = x_i^T β
The right-hand side is identical to that of a linear model; the change is entirely on the left. A GLM has three components.
3.1 Random component: an exponential-family response
Distributional assumption: given the covariates, the Y_i are independent and each follows an exponential-family distribution. The density or probability mass function can be written as:
f(y_i; θ_i, φ) = exp{ [y_i·θ_i − b(θ_i)] / a(φ) + c(y_i, φ) }
Here θ_i is the natural parameter, which can vary by observation; φ is the scale (dispersion) parameter; and b(·), a(·), and c(·) are known functions.
- Mean and variance: it follows that E(Y_i) = b’(θ_i) and Var(Y_i) = a(φ)·b’’(θ_i). Once the distribution is chosen, mean and variance are tied together by a fixed relationship, written Var(Y) = φ·V(μ), where V(μ) is the variance function. This is the key difference from “normal plus identity” linear regression: the variance need not be constant.
The exponential family covers many distributions used in modeling, including the normal, Poisson, binomial, and gamma distributions, each matching a typical scenario in the table below.
3.2 Systematic component: the linear predictor
ηi = x_i^T β = β_0 + β_1·x{i1} + … + βp·x{ip}
Here x_i is the i-th row of the model matrix X, and β is the parameter vector to be estimated. “Linear” means a linear combination in the parameters β, not that the fitted curve itself is a straight line.
3.3 Link component: the link function
g is a smooth, monotone (usually strictly monotone), and differentiable link function that maps the mean μ_i to the linear predictor η_i. Common pairings are:
| Response setting | Typical distribution | Common link | Variance structure |
|---|---|---|---|
| Continuous, roughly symmetric | Normal | Identity, η = μ | Constant |
| Counts (0, 1, 2, …) | Poisson | Log, η = log(μ) | Var ∝ μ |
| Successes and failures | Binomial | logit, probit, complementary log-log | Var ∝ μ(1−μ) |
| Strictly positive, right-skewed | Gamma | Reciprocal, η = 1/μ; or log | Var ∝ μ² |
Special case: with an identity link and a normal response, g(μ) = μ = x^T β, so the ordinary linear model is recovered as a special case of a GLM.
Estimation: GLMs are usually fit by maximum likelihood. In iteratively reweighted least squares (IRLS), the weights at each step depend on the variance function and the derivative of the link. In essence, the algorithm repeatedly solves a weighted linear regression.
4. Where the “Linearity” Hides: Two Worked Examples
Example 1: Exponential growth early in an epidemic
Early in an outbreak, when most of the population is still susceptible and transmission chains have not yet been interrupted, new case counts often grow approximately exponentially:
μ_i = γ · exp(δ · t_i)
Here μ_i is the expected number of new cases on day t_i, γ > 0 is the initial level, and δ > 0 is the growth rate. The model is nonlinear in the parameters, but taking logs of both sides gives:
log(μ_i) = log(γ) + δ · t_i = β_0 + β_1 · t_i
Parameterization: set β_0 = log(γ) and β_1 = δ, so the intercept is the log of the initial level and the slope is the growth rate.
Model specification:
- The response is the daily count of new cases, so a Poisson distribution is a reasonable choice.
- Use a log link.
- The linear predictor is β_0 + t_i·β_1.
Then μ_i = exp(β_0 + β_1·t_i) recovers the original model. Exponential growth is still a curve, but it is linear in the parameters and the linear predictor, so standard GLM estimation and inference apply.
Practical notes:
An offset can fix a known doubling time. If the epidemiological mechanism implies a fixed doubling time τ (for example, a fixed generation interval), set β_1 = log(2)/τ and rewrite:
log(μ_i) = log(2)/τ · t_i + β_0
Treat the known term as an offset and estimate only the intercept, which improves efficiency.
- Poisson variance equals the mean. The Poisson assumption Var = μ is relatively plausible early in an outbreak. As the outbreak develops, clustered transmission, underreporting, or interventions can produce overdispersion, in which case a negative binomial distribution may be considered.
- Does independence hold? The original specification assumes daily counts are independent, but adjacent dates on the same epidemic curve are usually autocorrelated. Data from multiple places or institutions also have a clear grouping structure. That is the opening for moving from a GLM to a GLMM (see Section 6.2).
Example 2: Saturation in a predator’s capture rate
A predator’s capture rate y_i rises with prey density x_i, but handling time (pursuit, ingestion, and digestion) imposes an upper limit, so the rate cannot grow without bound. A common Type II functional response is:
μ_i = α · x_i / (h + x_i)
Here α is the maximum capture rate, the asymptote as density tends to infinity, and h is the prey density at which the capture rate reaches α/2, the half-saturation density.
Shape intuition: rewriting the right-hand side as μ_i = α / (1 + h/x_i) makes the shape clear: as x_i → 0, μ_i → 0; as x_i → ∞, μ_i → α; and the curve is monotone increasing.
Model specification:
- Parameters appear in both the numerator and the denominator, so they enter the model nonlinearly.
- With a reciprocal link (η = 1/μ) and 1/x_i as the covariate in the linear predictor, 1/μ_i = β_0 + β_1 · (1/x_i).
- Then β_0 = 1/α and β_1 = h/α, so the parameters are recovered in linear form.
Back-transformation: after fitting, α = 1/β_0 and h = β_1/β_0. Standard errors can be approximated by the delta method:
- Var(α̂) ≈ Var(β̂_0) / β_0⁴
- Var(ĥ) ≈ [Var(β̂_1) + h²·Var(β̂_0) − 2h·Cov(β̂_0, β̂_1)] / β_0²
Distribution choice: a capture rate is often treated as a continuous positive quantity whose spread is roughly proportional to the mean, so the standard deviation σ satisfies σ ∝ μ. A gamma distribution has variance φ·μ², so the ratio of standard deviation to mean is constant, which matches that empirical pattern. A gamma response with a reciprocal link is therefore a complete specification.
Other choices:
- If the capture rate is recorded as “captures / number of trials” with a known denominator, a binomial distribution is more natural. A log link can then be used with an offset of log(number of trials).
- If the proportion lies strictly in (0, 1), beta regression is another option.
- Under a reciprocal link, η → ∞ as μ → 0. When x_i = 0, the covariate 1/x_i is undefined, so those observations must be dropped or the link changed.
What the two examples share
The linear predictor is not limited to a straight line. Any structure that can be written in a linear model can be copied into the linear predictor:
- categorical variables and dummy variables;
- interactions;
- polynomial terms;
- piecewise linear terms and splines;
- offsets fixed at known values.
The real choices are only two: the link function and the response distribution.
5. The Mathematical Skeleton of a GLMM: Random Effects in the Linear Predictor
5.1 Conditional modeling: a three-layer structure
Using the common random-intercept case, index observations by group j (for example, region, hospital, or subject) and by i within the group:
- Conditional distribution: given the random effect b_j, the within-group observations Y_ij are independent and follow an exponential-family distribution.
- Conditional mean: g(μ_ij) = x_ij^T β + z_ij^T b_j, where Z is the random-effect design matrix. For a random intercept, z_ij = 1.
- Random-effect distribution: b_j ~ N(0, G). For a random intercept, b_j ~ N(0, σ_u²).
Compared with a GLM, the linear predictor has an extra term b_j. Compared with an LMM, g(·) is no longer the identity and the conditional distribution is no longer normal. A GLMM sits at the intersection of the two.
5.2 Inference target: conditional or marginal interpretation
If the interest is only the population-average marginal mean, then
E(Y_ij) = E_b[ g^{-1}(x_ij^T β + b_j) ]
Identity link: g^{-1} is linear, so expectation and integration can be exchanged, and E(Y_ij) = x_ij^T β, exactly as in a GLM.
Log link: g^{-1}(η) = exp(η) and E[exp(b_j)] = exp(σ_u²/2), so the marginal mean is
E(Y_ij) = exp(x_ij^T β + σ_u²/2)
The slope β is the same; only the intercept differs by the factor exp(σ_u²/2). The conditional and marginal models therefore give the same relative growth rate, but different baseline levels.
Logit link: g^{-1} is an S-shaped curve. Conditional effects are generally larger than marginal effects, and the attenuation grows as the random-effect variance grows.
Two consequences follow:
- In a GLMM, β is a conditional effect given the group: the change associated with a one-unit change in a covariate within the same group.
- In a GLM, β is a marginal effect in the population-average sense.
Numerical closeness under a log link does not mean the two interpretations can be used interchangeably.
5.3 Random intercepts and random slopes
| Structure | Assumption | Meaning |
|---|---|---|
| Random intercept only | Groups differ in baseline, with a common slope | Within-group correlation is constant: any two records in the same group are equally correlated |
| Random intercept plus random slope | Groups differ in both baseline and trend, and the two may be correlated | Treatment effects may differ across groups, while still partially borrowing strength from the population |
Ignoring a random slope can underestimate the standard error of a treatment effect and inflate significance.
5.4 Fitting methods and computational cost
The marginal likelihood of a GLMM is:
L(β, G) = ∏_j ∫ ∏_i f(y_ij | b_j) · f(b_j) db_j
Because the link is nonlinear, this integral usually has no closed form and must be handled by numerical integration or approximation. Common approaches include:
- Laplace approximation: approximate the integrand by a Gaussian. It is fast, suitable for counts and similar data, and can be extended when the random-effect dimension is moderate to high.
- Adaptive Gauss–Hermite quadrature: quadrature points are placed around the mode of the conditional posterior. Accuracy is higher, but cost grows exponentially with the dimension of the random effects.
- Penalized quasi-likelihood (PQL): repeatedly fit a weighted linear mixed model with a mixed-model solver. Computation is light, but bias is a concern for binary responses such as Bernoulli data, very small clusters, or very low counts.
- Bayesian methods (for example, MCMC): sampling yields a full posterior. This is flexible for complex random-effect structures and small samples, but the computational cost is substantially higher.
6. GLM or GLMM: A Layered Decision Process
Given a dataset, the choice can be made in the following order.
6.1 Step 1: Identify the data structure and within-group correlation
Is the data grouped?
Look for a natural grouping, for example:
- repeated measurements on the same patient;
- multiple departments in the same hospital;
- adjacent dates in the same region;
- multiple trials on the same subject;
- repeated counts on the same individual.
Strength of within-group correlation
If grouping exists, assess how strong the within-group correlation is. For a random-intercept model, the intraclass correlation (ICC) is:
ICC = σ_u² / (σ_u² + σ_e²)
On the latent scale of a logit link, the ICC can be approximated as:
ICC ≈ σ_u² / (σ_u² + π²/3)
where π²/3 is the variance of the standard logistic distribution.
- If the ICC is near 0, within-group correlation is weak and a GLM can be tried first.
- If the ICC is clearly above 0, observations within a group are not independent. GLM standard errors will be too small and significance will be overstated.
Method choice
- If the ICC is small, or the goal is only robust inference for fixed effects, a GLM with cluster-robust or sandwich variance estimates can be used without an explicit model for within-group correlation.
- If the ICC is substantial, there are many groups, and the grouping structure itself is of interest, a GLMM is more appropriate.
6.2 Step 2: Revisit the two original examples
Example 1 revisited: from GLM to GLMM
The original specification assumes epidemic curves in different places are independent. In data from multiple regions, institutions, or hospitals, baseline levels, testing capacity, and intervention intensity all vary by place:
- If the only target is an overall average growth rate, region indicators can be added to the linear predictor and paired with cluster-robust standard errors. A GLM is then sufficient.
A GLMM is warranted if the goals are to:
- estimate an overall growth trend;
- describe whether each region sits above or below that overall level;
- predict for a new region.
A multilevel GLMM can be written as:
log(μijk) = β_0 + β_1·t_ijk + u{0j} + u{1j}·t_ijk + u{0k(j)}
where:
- u_{0j} is the random intercept for region j, capturing baseline differences;
- u_{1j} is the random slope for region j, capturing differences in growth rate;
- u_{0k(j)} is the random intercept for institution k nested in region j.
Partial-effect adjustment: the random intercepts are themselves error terms. They absorb unobserved heterogeneity across regions, and the resulting estimates are partial effects.
Overdispersion: if each region–date cell also contains counts from several institutions, an institution-level random intercept plus an observation-level random term distributed as N(0, σ_ε²) yields a model equivalent to a negative binomial. The conditional variance is:
Var(Y|b) = μ + [exp(σ_ε²) − 1]·μ²
That is, beyond the Poisson term μ there is an extra overdispersion term that grows with μ².
Example 2 revisited: from GLM to GLMM
If the experiment involves multiple predator individuals, multiple tanks, or multiple batches, each individual’s maximum capture rate α and half-saturation density h may differ:
- Random intercept only: equivalent to a group-level shift in the linear predictor, reflecting differences in baseline capture ability.
- Random intercept plus random slope: equivalent to letting individuals differ in both α and h, reflecting differences in the upper limit and the inflection.
A GLMM should be used in that case as well.
6.3 Step 3: Check the distributional assumption and overdispersion
After fitting a GLM, check:
- Pearson statistic / degrees of freedom: is it clearly greater than 1?
- Deviance / degrees of freedom: is it clearly greater than 1?
- Residuals versus fitted values: is there systematic curvature?
If overdispersion is clear, consider the following in order:
- Change the distribution: Poisson to negative binomial, or normal to gamma or lognormal.
- Add an observation-level random effect, which produces a GLMM.
- Check whether an important grouping structure or random slope was omitted.
6.4 Step 4: Consider the number of groups and exchangeability
| Number of groups | Suggestion |
|---|---|
| Very few groups (for example, fewer than 5–6) | The random-effect variance is unstable. Prefer fixed effects, or a Bayesian model with a prior. |
| Many groups, and the groups are a sample from a larger population | A GLMM is more appropriate. Shrinkage of the random effects can produce more stable estimates. |
| The groups are the entire set of units of interest (for example, every province in a country, with no wish to extrapolate) | Treat them as fixed effects. |
6.5 Step 5: Start from the prediction target
- Extrapolation to a new group: if predictions are needed for a region or individual that never appeared in the training data, the random effect cannot be estimated directly. It must be integrated out to obtain a marginal prediction. The marginal interpretation of the GLMM then matters.
- Interpretation for existing groups only: if the analysis concerns only specific groups in the training data, either fixed effects or the conditional interpretation of a GLMM can be used.
- Comparing groups: if the question is whether between-group differences are substantial, compare random-effect variances, use a variance-component test, or inspect a Bayesian posterior.
6.6 Step 6: Sample size and the design effect
For grouped data, a sample-size calculation that treats observations as independent underestimates the sample size required. A common adjustment is:
DE = 1 + (m̄ − 1)·ρ
where m̄ is the average group size and ρ is the ICC.
- With ICC = 0.1 and 30 observations per group, DE = 1 + 29×0.1 = 3.9, so the effective sample size is only about one quarter of the nominal sample size.
- With ICC = 0.01, DE ≈ 1.29, and the impact is smaller.
At the design stage, the expected within-group correlation should be anticipated before deciding whether a GLMM is needed.
7. A Practical Workflow and Illustrative Pseudocode
7.1 Standard workflow
- Explore the data: plot the response distribution and within-group mean–variance relationships, and check the proportion of zeros and the grouping structure.
- Choose the response distribution: counts → Poisson or negative binomial; proportions → binomial or beta; positive right-skewed values → gamma.
- Choose the link: counts → log; proportions → logit; gamma → reciprocal or log.
- Assess within-group correlation: when grouping exists, use the ICC, within-group mean–variance plots, and repeated-measures plots.
- Fit a GLM first, then a GLMM: compare AIC, residual diagnostics, and estimated random-effect variances.
- Run a sensitivity analysis: change the distribution and the random-effect structure and see whether the conclusions hold.
Illustrative pseudocode (structure only; not the syntax of any specific tool):
# Section 6.2, Example 1: exponential growth + Poisson# Basic formlog(mu_i) = beta0 + beta1 * t_i# With regional grouping, as a mixed model:log(mu_ijk) = beta0 + beta1 * t_ijk + u_{0j} + u_{1j} * t_ijk + u_{0k(j)}u_{0j} ~ N(0, sigma_u0^2)u_{1j} ~ N(0, sigma_u1^2)u_{0k(j)} ~ N(0, sigma_v0^2)y_ijk ~ Poisson(mu_ijk)# Section 6.3, Example 2: reciprocal link + gamma# Basic formeta_i = 1 / mu_i = beta0 + beta1 * (1 / x_i)alpha = 1 / beta0h = beta1 / beta0y_i ~ Gamma(shape = 1 / phi, scale = phi * mu_i)# With individual grouping, as a mixed model:eta_ij = beta0 + beta1 * (1 / x_ij) + u_{0j} + u_{1j} * (1 / x_ij)u_{0j}, u_{1j} ~ N(0, G)y_ij ~ Gamma(shape = 1 / phi, scale = phi * mu_ij)
7.2 Model comparison
| Comparison | What can be compared | Notes |
|---|---|---|
| GLM vs. GLM (same data) | AIC, BIC, deviance | Deviance is not directly comparable across distribution families; use AIC. |
| GLMM vs. GLMM (same data) | AIC, BIC, conditional AIC | Different random-effect structures can be compared when the fixed effects are the same. |
| GLM vs. GLMM | Direct AIC comparison is not recommended | Marginal interpretations differ, and the marginal likelihoods of a GLM and a GLMM are usually not on the same scale. |
Boundary issue in likelihood-ratio tests: when testing whether a random-effect variance is zero, the null hypothesis lies on the boundary of the parameter space. The test statistic no longer follows a standard chi-square distribution. Use a mixture chi-square distribution, or compare AIC directly.
8. A Diagnostic Checklist and Common Mistakes
8.1 Diagnostic checklist
- Residuals versus fitted values: look for systematic curvature. For a Poisson model, residual variance should generally increase with the mean.
- QQ plots: build simulation envelopes for the distribution in use (Poisson, gamma, or normal) and check whether residuals match the assumption.
- Overdispersion measures: whether Pearson χ² / degrees of freedom and deviance / degrees of freedom are clearly greater than 1.
- Random-effect diagnostics: whether random effects are approximately normal and whether variance components are substantial.
- Influential points: the effect of high-leverage points and large residuals on the estimates.
- Convergence: whether the GLMM fit converged, and whether a singular fit occurred.
8.2 Common mistakes
- Ignoring within-group correlation: fitting a GLM while ignoring grouping underestimates standard errors and inflates Type I error.
- Giving every observation its own random intercept: one random effect per observation is an unconstrained model and fixed effects cannot be estimated.
- Using a GLMM with too few groups: the random-effect variance is unstable. Prefer fixed effects or a Bayesian approach.
- Choosing the distribution casually: leaving Poisson overdispersion untreated, or using a gamma distribution on data that contain zeros (the gamma distribution does not support 0).
- Confusing conditional and marginal effects: under a log link the slopes match, but the intercepts differ by exp(σ_u²/2). Under a logit link the gap between conditional and marginal effects is larger.
- Misusing offsets: an offset must be fixed with coefficient 1; its coefficient should not be estimated. An offset belongs in the linear predictor, not on the response itself.
- Ignoring the design effect in sample-size calculations: treating observations as independent leaves the study underpowered.
- Overcomplicating the random-effect structure: adding random slopes and random intercepts together when there are few groups and sparse data, which leads to non-convergence.
9. Conclusion
Moving from a GLM to a GLMM is a move from “independent observations plus a parametric distribution” to “conditional independence plus random effects plus a parametric distribution”:
- A GLM addresses a response that is not normal and a mean that is nonlinearly related to the linear predictor.
- A GLMM addresses observations that are not independent and heterogeneity across groups or individuals.
The two central questions are:
- Are the observations independent?
- Is it necessary to model between-group heterogeneity explicitly, and to extrapolate or use shrinkage?
After those questions, choose the response distribution, the link function, and the random-effect structure. Residual checks, overdispersion checks, and sensitivity analysis then confirm whether the model is reliable.
