#StackBounty: #lasso #regularization #bias #asymptotics #unbiased-estimator Asymptotic bias of LASSO vs. none of SCAD

Bounty: 50

I am reading a paper which says that LASSO is asymptotically biased while SCAD is not. I take asymptotic (un)biasedness to concern the slope estimators from LASSO and SCAD as the sample size goes to infinity, but I am not sure. I wonder what exactly these statements mean, under which assumptions they hold and whether these assumptions are realistic.

In my understanding, LASSO is asymptotically biased given fixed regularization intensity $lambda$. However, realistically $lambda$ would not be kept fixed as the sample size grows but would rather be reduced (e.g. this would be the case if one used LOOCV for selecting $lambda$, something that is fairly common), reducing the bias accordingly. Taking this to the limit, it appears LASSO would not be asymptotically biased.

On the other hand, if we look at some typical pictures illustrating LASSO and SCAD estimators, they often consider them as functions of the slope coefficient. There, we see that LASSO is asympt. biased while SCAD is asympt. unbiased when the slope coefficient (rather than the sample size) goes to infinity. (See below.) Again, I think $lambda$ is fixed here.

enter image description here

So I am confused, and hence my question.

Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.