Let’s say I run a procedure where I fit every possible model given some set of covariates and I select the model with the minimum AIC. I know that if my selection criteria was based on minimizing p-values, the p-values of the selected model would be misleading. But what if my selection criteria was AIC alone? To what extent would this bias the p-values?
I had assumed the effect on p-values would be negligible, but came across this paper, which proves the following:
P values are intimately linked to confidence intervals and to
differences in Akaike’s information criterion (ΔAIC), two metrics that
have been advocated as replacements for the P value.
If this is true, does it imply that p-values are misleading after automatic selection based on AIC? To what extent will they be biased, and what determines this?