*Bounty: 50*

*Bounty: 50*

$DeclareMathOperator*{argmax}{arg,max}$Given a set of $N$ i.i.d. observations $X=left{x_1, ldots, x_Nright}$, we train a model $p(x|boldsymbol{theta})$ by maximizing marginal log-likelihood $log p(X mid boldsymbol{theta})$. A full posterior $p(boldsymbol{theta}|X)$ over model parameters $boldsymbol{theta}$ can be approximated as a Gaussian distribution using Laplace method.

In the case that the Gaussian distribution gives a poor approximation of $p(boldsymbol{theta}|X)$, Newton and Raftery (1994) proposed weighted likelihood bootstrap (WLB) as a way to simulate approximately from a posterior distribution. Extending Bayesian bootstrap (BB) of Rubin (1981), this method generates BB samples $tilde{X}=(X,boldsymbol{pi})$ by repeatedly drawing sampling weights $boldsymbol{pi}$ from a uniform Dirichlet distribution and maximizes a weighted likelihood to calculate $boldsymbol{theta}_{text{MWLE}}$.

begin{equation} boldsymbol{theta}*{text{MWLE}}=argmax*{boldsymbol{theta}}sum_{n=1}^{N} pi_nlog p(x_n|boldsymbol{theta}).

end{equation}

So the algorithm can be summarized as

- Draw a posterior sample $boldsymbol{pi}sim p(boldsymbol{pi}|X)=mathcal{D}ir(1,dots,1)$.
- Calculate $theta_{text{MWLE}}$ from weighted sample $tilde{X}=(X, boldsymbol{pi})$

Newton and Raftery (1994) state that

In the generic weighting scheme, the WLB is first order correct under

quite general conditions.

- I was wondering what exactly does this mean and what does first order refer to? How well does this approximation $p(boldsymbol{theta}|X)$?

Later authors state that

Inaccuracies can be removed by using the WLB as a source of samples in

the sampling-importance resampling (SIR) algorithm.

- I was not sure what this exactly means. Could someone point what step in my algorithm exactly should I change?