Hypothesis tests and model selection

Slide 1

1

• Likelihood Ratio Tests
• And
• Model Selection

Slide 2: Model selection — Why?

• Understand which explanatory variables are important
• Find evidence of relationship(s) between variable(s) and response
• Quantify the effects of explanatory variables on the response
• Use model to predict response values
• Predict response at another location or time

Slide 3: Inference, Estimation, Prediction

• Understand which explanatory variables are important
• Find evidence of relationship(s) between variable(s) and response
• Quantify the effects of explanatory variables on the response
• Use model to predict response values
• Predict response at another location or time

Slide 4: Model selection — Why?

• Q: Why not just include all explanatory variables?
• A: They will (almost certainly) be correlated.
• Correlation between explanatory variables…
• makes it hard to identify important effects;
• causes bias in effect sizes/parameter estimates;
• leads to overfitting, and poor out-of-sample prediction.

Slide 5: Fit?

• Compare “fit” of different models
• Define “fit” — (maximum) likelihood?
• Increases when adding more variables
• Need to penalise fit somehow (to avoid overfitting)
• Choice between different approaches reflects choice of penalty
• sometimes just strength of penalty
• Three ways of comparing likelihoods…

Slide 6: Definitions: AIC, BIC, LRT

• AIC — Akaike’s Information Criterion
• AIC = −2 log ℓ(θ̂) + 2p
• AICC — corrected for small samples
• AICC = AIC + 2p(p+1) / (n − p − 1)
• BIC — Bayesian Information Criterion
• BIC = −2 log ℓ(θ̂) + p log n
• LRT — Likelihood Ratio Test
• −2 log ℓ(θ̂₁) + 2 log ℓ(θ̂₂) ~ χ² with df = p₂ − p₁

Slide 7: Hypothesis Tests to Compare Models (not parameters)

7

• One can use the likelihood ratio test to compare two models when
• both are based on the same data
• one of the models is “nested” within the other
• That is, one model is a special case with a parameter equal to a specific value (usually zero) from the more general model.

Slide 8: Nested Models

8

• The following models are nested:
• y = Time + Depth * Region
• y = Depth * Region

• The following models are not nested:
• y = Time + Depth * Region
• y = Temp + Depth * Region

Slide 9: Hypothesis Tests for Models

• Model A nested within more general model B can be compared as follows:
• H0: Model A fits the data as well as Model B
• (with fewer parameters)
• HA: Model A does not fit the data as well as Model B
• (you need those extra parameters)
• Fit models A and B and record:
• the negative log-likelihood at the best fit
• the number of parameters in models A and B, pA and pB.

9

Slide 10: Hypothesis Tests for Models

10

Compute the likelihood ratio

( −ln LA − (−ln LB) )

If using lm or glm objects in R: lrtest(full_model, reduced_model)

Slide 11: When is this useful rather than just interpreting coefficients?

Interpret coefficients when you're asking about individual parameters. Use a likelihood ratio test (LRT) when you're asking about a model or a collection of parameters.

glm(count ~ temperature + elevation, family = poisson)

No Need for LRT, just interpret coefficients

glm(count ~ treatment, family = poisson)

Suppose treatment has 5 levels. How do you know if the model is better with treatment? Use LRT

glm(count ~ treatment*temperature, family = poisson)

How do you know if the interaction term makes the model better? Use LRT

Slide 12: Usually the simpler model sets some parameters to zero, but null can be nonzero

• You monitor an intersection for cars that run the red light. Based on 50 days of observation, you detect 30 days on which “light running” took place. You repeat the “experiment” after a red light camera is installed and now observe 10 days on which “running” occurred.
• How do we know if there is a significant difference in the rate of running red lights after a camera is installed?

Slide 13: Example-I

13

• Let us define P1 as the probability of running the light before the
• camera was installed and P2 the probability after the camera
• The Binomial likelihood function (ignoring the normalizing constant):
• Model A: The probability of running the light has not changed
• (P1=P2; The more specific model; you only need 1 parameter)
• Model B: P1 may differ from P2.
• (The more general model – more parameters)

L = P130 (1 − P1)20 P210 (1 − P2)40

Slide 14: Example-II

Summary for this example

• Model A: 0.4 0.4 67.30
• Model B: 0.6 0.2 58.67

• The difference between the negative log likelihoods
• 67.3 - 58.7 = 8.6
• Twice this difference is 17.2
• Is this significant given the difference in parameters is 1?

P̂1

P̂2

−ln L

Slide 15: Example-II

• The difference between the negative log likelihoods is 8.6 – is
• this significant given the difference in parameters is 1?
• 2*8.6=17.2, compare to chisq distribution with one d.f

• REJECT THE NULL
• That the one parameter model fits that data. Much better to use two parameters, indicating that the probability of running the red light has changed!

Slide 16

• Likelihood ratio test can typically only be applied to nested models
• “Information theoretic” approaches (e.g. AIC, BIC) can be used to compare non-nested models.
• Models with smaller values of AIC or BIC are “better”, or “have more support” or “are more parsimonious”, but we’ll dig into what this really means for AIC and BIC.

Slide 17: Information Theoretic Criteria for Model Selection

Look familiar?

Compare to chi.sq distribution with degrees of freedom = # parameters

AIC and BIC are meaningless on their own, but models with lower values are more parsimonious (i.e. supported). They balance better fit with more parameters. This is exactly what the likelihood ratio test does.

LRT = −2 loge ( ℒs(θˆ) / ℒg(θˆ) )

• AICi = −2 log Li + 2 pi
• BICi = −2 log Li + pi log n

Slide 18

Compare to chi.sq distribution with degrees of freedom = # parameters

compare two models with k parameters and with k+1

Information Theoretic Criteria for Model Selection

LRT = −2 loge ( ℒs(θˆ) / ℒg(θˆ) )

Slide 19

Convention within>2 AIC units are “considered competitive”, and this is approximately the same as frequentist model selection using LRT

If 1 parameter separate the model. AIC likes bigger models

“the extra parameter is worth it”

Slide 20

What if 3 parameters separate the model? AIC likes bigger models

(reject the null that the parameters equal 0)

2 deltaAIC still equivalent to p = 0.05

Slide 21: Most common mistake in ecology!

• Interpretation of relative AIC is not symmetric
• More complex model only “significantly better” is it’s AIC score is >2 units smaller.
• Otherwise simpler and complex model are competitive
• Penalty for adding parameter is 2 units and fit improves with complexity, so models with random data as predictor will be within 2 AIC. The complex model has not support even if within 2 AIC units
• The 2 AIC convention is about whether a more complex model with lower AIC is better enough relative to simpler model (in the sense that the parameters are informative)

AICi = −2 log Li + 2 pi

Slide 22

• 3700 Citations!
• 72% of all papers incorrectly interpret uninformative parameters!

Slide 23: The Cult of AIC

Burnham & Anderson (2002)

Slide 24: Two key questions

• “Always use AIC for model selection or model comparison”:
• Is this instruction correct?
• Why is it wrong?

From statisticians Mark Brewer and Adam Butler

Slide 25: AIC and p-values

• From Murtaugh Ecology (2014), Fig. 2:
• Direct comparison
• Two models, differ by one parameter

Slide 26

• P < 0.05 is an arbitrary threshold
• But ⍙AIC < 2 is as well
• AIC doesn’t suffer from Type I error or problems with autocorrelation
• This is just wrong, as we will see by simulation. Making inference by AIC has MUCH MORE Type I error
• P-values don’t tell us how important an effect is – i.e. whether the effect is biologically significant. Use effect sizes and confidence intervals instead
• AIC doesn’t tell you that either. And I notice big tables full of models with no effect sizes or R2, sometimes without even an intercept only model. What if all the models are terrible?
• We always need effect sizes for inference. These are the most important part of the science!
• But let’s remember how confidence intervals and p-values are related

Why does the cult of AIC hate p-values?

Slide 27: Confidence intervals and p-values

-3 -2 -1 0 1 2 3

Estimate and 95% CI. What is p?

Estimate and 99% CI. What is p?

Effect sizes and p tell you the same thing as effect sizes and CIs!

Slide 28: P-values should be interpreted on a continuum. Not a step function at 0.05. AIC should be interpreted in the same way!

deltaAIC=2

deltaAIC=7

Slide 29: So why use AIC?

• It is easy to assess non-nested models of highly variable structure.
• It is good for selecting models for prediction
• Useful when you have lots of potentially correlated predictors
• But it is widely used for inference, resulting in more Type I error
• Often used incorrectly, resulting in more error (more on this later)

Slide 30

Why did AIC take over wildlife science?

• Aggressive marketing of the method by Burnham and Anderson, who were in Fisheries and Wildlife and “wrote the book”
• People adopt new statistical methods, then find the holes, and then the pendulum swings back
• It feels rigorous because you have to run so many models
• People like ranking things

Slide 31: AIC in practice…

Which is the top model? Which models are competitive?

Slide 32: AIC in practice

• For small sample sizes, AIC is biased and so people use AICc. AICc is treated as “corrected AIC” and converges to AIC for large n.
• Model weights. Each model i of R is weighted based on its deltaAIC.
• Variables included with lots of model weight considered important.
• Model weight used for averaging across models for prediction
• Both deltaAIC and weights depend on model set

Why would I hate this table as the only presentation of results?

AICc = AIC + (2k2 + 2k) / (n − k − 1)

wi = exp(−0.5 Δi) / Σr=1 R exp(−0.5 Δr)

Slide 33: Most common mistake in ecology!

• Interpretation of relative AIC is not symmetric
• If a more complex model has higher AIC but less than 2 units, it is NOT competitive
• Penalty for adding parameter is 2 units and fit improves with complexity, so models with random data as predictor will be within 2 AIC.
• The 2 AIC convention is about whether a more complex model with lower AIC is better enough relative to simpler model

AICi = −2 log Li + 2 pi

Slide 34

• 3700 Citations!
• 72% of all papers incorrectly interpret uninformative parameters!

Slide 35: Best recent paper as a resource to stop doing dumb stuff with AIC

(No text on this slide.)

Slide 36: B & A recipe for modelling

• Procedure for model selection:
• Decide on scientific questions to answer
• Define small candidate set of models
• Fit models, calculate AIC, find “best” model
• Use threshold on ΔAIC to determine set of “plausible” models
• Calculate model weights via ΔAIC
• (Assess variable importance using weights)

Slide 37: Burnham and Anderson did not advocate model dredging (fitting all/many combinations of predictors)

• If you look at many models, you will have many Type I errors leading to “overfitting”.
• But did ecologists listen to B&A and construct sensible model sets?
• Hell no
• Do ecologists even think about whether the purpose of modeling is inference or prediction?
• Also no

“An investigator with, say, 10 explanatory variables cannot expect to learn much from his data and a multiple linear regression analysis unless there is some substantial supporting science that can be used to help narrow the number of models to consider. In this example, there would be 1,024 models (many more if transformations or interaction terms were allowed), and over-fitting would surely be a serious risk. The analysis, by whatever method, should probably be considered exploratory and the results used to design further data gathering leading to a more confirmatory analysis, based on some a priori considerations” (Burnham and Anderson 2002:85)

Slide 38

(No text on this slide.)

Slide 39: Then what about BIC? Is it just a similar alternative?

• The penalty in BIC increases with the sample size
• AIC retains more complex models than BIC or p-values
• AIC chooses the model that gives the best prediction. It is efficient for prediction.
• BIC chooses the “correct” model if it is in the set.

• AICi = −2 log Li + 2 pi
• BICi = −2 log Li + pi log n

Slide 40: AIC vs BIC

• “Theoretically, AIC is optimal in the sense of minimizing the mean square error of predictions, whereas a competing metric, the Bayesian Information Criterion (BIC; Schwarz 1978), is consistent – that is, it will tend to select the ‘true’ model as the sample size increases to infinity.”
• AIC is asymptotically equivalent to leave-one-out crossvalidation (prediction), and BIC is most similar to inference (p-values)

Slide 41

Slide 42: Crossvalidation: Split data, train model, compare predictions to actual using test data. Repeat.

• k-fold crossvalidation
• Leave-one-out (n-fold crossvalidation)

Train

Test

Dataset

Slide 43: Recommended Readings

• Aho et al. 2014 “Model selection for ecologists: The worldviews of AIC and BIC”. Ecology
• Shmueli 2010 “To explain or to predict”. Statistical Science
• Murtaugh 2007 “Simplicity and complexity in ecological data analysis”. Ecology
• Arnold 2010 “Uninformative Parameters and Model Selection Using Akaike's Information Criterion” JWM
• ”Murtaugh 2014 “In defense of P-values”. Ecology
• Fieberg and Johnson 2015 “MMI: Multimodel Inference or Models With Management Implications?”. Journal of Wildlife Management
• Sutherland et al. 2023 “Practical advice on variable selection and reporting using Akaike information criterion. Proceedings B
• Bolker 2024 “Multimodel Approaches Are Not the Best Way to Understand Multifactorial Systems”. Entropy

Slide 44

Slide 45: Don’t take my word for it…

One way to avoid some of the problems associated with model selection is to “just say no” to data-driven model selection—in other words, fit a single pre-specified model, one in which all explanatory variables are chosen prior to looking at their relationship with the response variable, and then use this model for inference (Harrell 2001, Babyak 2004, Whittingham et al. 2006, Guidice et al. 2012).

Slide 46

“If researchers want to accurately estimate the strength of multiple competing ecological processes along with reliable confidence intervals, their best hope is to use full (maximal) statistical models after making principled, a priori decisions about which predictors to include. “

Don’t take my word for it…

Left as a preprint for years until Bolker finally submitted it to a journal after we chatted about this problem

Slide 47: Don’t take my word for it…

The associated sums of AIC model weights recommended to assess relative importance of individual predictors are really a measure of relative importance of models, with little information about contributions by individual predictors compared to other measures of relative importance based on effects size or variance reduction.

The standardized estimates or equivalently the t statistics on unstandardized estimates also can be used to provide more informative measures of relative importance than sums of AIC weights. Finally, I illustrate how seriously compromised statistical interpretations and predictions can be for all three of these flawed practices by critiquing their use in a recent species distribution modeling technique developed for predicting Greater Sage‐Grouse (Centrocercus urophasianus) distribution in Colorado, USA. These model averaging issues are common in other ecological literature and ought to be discontinued if we are to make effective scientific contributions to ecological knowledge and conservation of natural resources.