# #StackBounty: #bayesian #bootstrap #posterior How well does weighted likelihood bootstrap approximate the Bayesian posterior?

### Bounty: 50

$$DeclareMathOperator*{argmax}{arg,max}$$Given a set of $$N$$ i.i.d. observations $$X=left{x_1, ldots, x_Nright}$$, we train a model $$p(x|boldsymbol{theta})$$ by maximizing marginal log-likelihood $$log p(X mid boldsymbol{theta})$$. A full posterior $$p(boldsymbol{theta}|X)$$ over model parameters $$boldsymbol{theta}$$ can be approximated as a Gaussian distribution using Laplace method.

In the case that the Gaussian distribution gives a poor approximation of $$p(boldsymbol{theta}|X)$$, Newton and Raftery (1994) proposed weighted likelihood bootstrap (WLB) as a way to simulate approximately from a posterior distribution. Extending Bayesian bootstrap (BB) of Rubin (1981), this method generates BB samples $$tilde{X}=(X,boldsymbol{pi})$$ by repeatedly drawing sampling weights $$boldsymbol{pi}$$ from a uniform Dirichlet distribution and maximizes a weighted likelihood to calculate $$boldsymbol{theta}_{text{MWLE}}$$.

$$begin{equation} boldsymbol{theta}{text{MWLE}}=argmax{boldsymbol{theta}}sum_{n=1}^{N} pi_nlog p(x_n|boldsymbol{theta}). end{equation}$$

So the algorithm can be summarized as

• Draw a posterior sample $$boldsymbol{pi}sim p(boldsymbol{pi}|X)=mathcal{D}ir(1,dots,1)$$.
• Calculate $$theta_{text{MWLE}}$$ from weighted sample $$tilde{X}=(X, boldsymbol{pi})$$

Newton and Raftery (1994) state that

In the generic weighting scheme, the WLB is first order correct under
quite general conditions.

1. I was wondering what exactly does this mean and what does first order refer to? How well does this approximation $$p(boldsymbol{theta}|X)$$?

Later authors state that

Inaccuracies can be removed by using the WLB as a source of samples in
the sampling-importance resampling (SIR) algorithm.

1. I was not sure what this exactly means. Could someone point what step in my algorithm exactly should I change?

Get this bounty!!!

This site uses Akismet to reduce spam. Learn how your comment data is processed.