## #StackBounty: #bayesian #confidence-interval #frequentist #credible-interval When does a confidence interval "make sense" but…

### Bounty: 50

It is often the case that a confidence interval with 95% coverage is very similar to a credible interval that contains 95% of the posterior density. This happens when the prior is uniform or near uniform in the latter case. Thus a confidence interval can often be used to approximate a credible interval and vice versa. Importantly, we can conclude from this that the much maligned misinterpretation of a confidence interval as a credible interval has little to no practical importance for many simple use cases.

There are a number of examples out there of cases where this does not happen, however they all seem to be cherrypicked by proponents of Bayesian stats in an attempt to prove there is something wrong with the frequentist approach. In these examples, we see the confidence interval contains impossible values, etc which is supposed to show that they are nonsense.

I don’t want to go back over those examples, or a philosophical discussion of Bayesian vs Frequentist.

I am just looking for examples of the opposite. Are there any cases where the confidence and credible intervals are substantially different, and the interval provided by the confidence procedure is clearly superior?

Get this bounty!!!

## Context

This is somewhat similar to this question, but I do not think it is an exact duplicate.

When you look for how instructions on how to perform a bootstrap hypothesis test, it is usually stated that it is fine to use the empirical distribution for confidence intervals but that you need to correctly bootstrap from the distribution under the null hypothesis to get a p-value. As an example, see the accepted answer to this question. A general search on the internet mostly seems to turn up similar answers.

The reason for not using a p-value based on the empirical distribution is that most of the time we do not have translation invariance.

## Example

Let me give a short example. We have a coin and we want to do an one-sided test to see if the frequency of heads is larger than 0.5

We perform $$n = 20$$ trials and get $$k = 14$$ heads. The true p-value for this test would be $$p = 0.058$$.

On the other hand if we bootstrap our 14 out of 20 heads, we effectively sample from the binomial distribution with $$n = 20$$ and $$p = frac{14}{20}=0.7$$. Shifting this distribution by subtracting 0.2 we will get a barely significant result when testing our observed value of 0.7 against the obtained empirical distribution.

In this case the discrepancy is very small, but it gets larger when the success rate we test against gets close to 1.

## Question

Now let me come to the real point of my question: the very same defect also holds for confidence intervals. In fact, if a confidence interval has the stated confidence level $$alpha$$ then the confidence interval not containing the parameter under the null hypothesis is equivalent to rejecting the null hypothesis at a significance level of $$1- alpha$$.

Why is it that the confidence intervals based upon the empirical distribution are widely accepted and the p-value not?

Is there a deeper reason or are people just not as conservative with confidence intervals?

In this answer Peter Dalgaard gives an answer that seems to agree with my argument. He says:

There’s nothing particularly wrong about this line of reasoning, or at
least not (much) worse than the calculation of CI.

Where is the (much) coming from? It implies that generating p-values that way is slightly worse, but does not elaborate on the point.

## Final thoughts

Also in An Introduction to the Bootstrap by Efron and Tibshirani they dedicate a lot of space to the confidence intervals but not to p-values unless they are generated under a proper null hypothesis distribution, with the exception of one throwaway line about the general equivalence of confidence intervals and p-values in the chapter about permutation testing.

Let us also come back to the first question I linked. I agree with the answer by Michael Chernick, but again he also argues that both confidence intervals and p-values based on the empirical bootstrap distribution are equally unreliable in some scenarios. It does not explain why you find many people telling you that the intervals are ok, but the p-values are not.

Get this bounty!!!

### Bounty: 50

Suppose $$Xsim N_3(0,Sigma)$$, where $$Sigma=begin{pmatrix}1&rho&rho^2\rho&1&rho\rho^2&rho&1end{pmatrix}$$.

On the basis of one observation $$x=(x_1,x_2,x_3)’$$, I have to obtain a confidence interval for $$rho$$ with confidence coefficient $$1-alpha$$.

We know that $$X’Sigma^{-1}Xsim chi^2_3$$.

So expanding the quadratic form, I get

$$x’Sigma^{-1}x=frac{1}{1-rho^2}left[x_1^2+(1+rho^2)x_2^2+x_3^2-2rho(x_1x_2+x_2x_3)right]$$

To use this as a pivot for a two-sided C.I with confidence level $$1-alpha$$, I setup $$chi^2_{1-alpha/2,3}le x’Sigma^{-1}xle chi^2_{alpha/2,3}$$

I get two inequalities of the form $$g_1(rho)le 0$$ and $$g_2(rho)ge 0$$, where

$$g_1(rho)=(x_2^2+chi^2_{alpha/2,3})rho^2-2(x_1x_2+x_2x_3)rho+x_1^2+x_2^2+x_3^2-chi^2_{alpha/2,3}$$

and $$g_2(rho)=(x_2^2+chi^2_{1-alpha/2,3})rho^2-2(x_1x_2+x_2x_3)rho+x_1^2+x_2^2+x_3^2-chi^2_{1-alpha/2,3}$$

Am I right in considering a both-sided C.I.? After solving the quadratics in $$rho$$, I am guessing that the resulting C.I would be quite complicated.

Another suitable pivot is $$frac{mathbf1′ x}{sqrt{mathbf1’Sigma mathbf 1}}sim N(0,1)quad,,,mathbf1=(1,1,1)’$$

With $$bar x=frac{1}{3}sum x_i$$, this is same as saying $$frac{3bar x}{sqrt{3+4rho+2rho^2}}sim N(0,1)$$

Using this, I start with the inequality $$left|frac{3bar x}{sqrt{3+4rho+2rho^2}}right|le z_{alpha/2}$$

Therefore, $$frac{9bar x^2}{3+4rho+2rho^2}le z^2_{alpha/2}implies 2(rho+1)^2+1ge frac{9bar x^2}{z^2_{alpha/2}}$$

That is, $$rhoge sqrt{frac{9bar x^2}{2z^2_{alpha/2}}-frac{1}{2}}-1$$

Since the question asks for any confidence interval, there are a number of options available here. I could have also squared the standard normal pivot to get a similar answer in terms of $$chi^2_1$$ fractiles. I am quite sure that both methods I used are valid but I am not certain whether the resulting C.I. is a valid one. I am also interested in other ways to find a confidence interval here.

Get this bounty!!!

### Bounty: 50

Suppose $$Xsim N_3(0,Sigma)$$, where $$Sigma=begin{pmatrix}1&rho&rho^2\rho&1&rho\rho^2&rho&1end{pmatrix}$$.

On the basis of one observation $$x=(x_1,x_2,x_3)’$$, I have to obtain a confidence interval for $$rho$$ with confidence coefficient $$1-alpha$$.

We know that $$X’Sigma^{-1}Xsim chi^2_3$$.

So expanding the quadratic form, I get

$$x’Sigma^{-1}x=frac{1}{1-rho^2}left[x_1^2+(1+rho^2)x_2^2+x_3^2-2rho(x_1x_2+x_2x_3)right]$$

To use this as a pivot for a two-sided C.I with confidence level $$1-alpha$$, I setup $$chi^2_{1-alpha/2,3}le x’Sigma^{-1}xle chi^2_{alpha/2,3}$$

I get two inequalities of the form $$g_1(rho)le 0$$ and $$g_2(rho)ge 0$$, where

$$g_1(rho)=(x_2^2+chi^2_{alpha/2,3})rho^2-2(x_1x_2+x_2x_3)rho+x_1^2+x_2^2+x_3^2-chi^2_{alpha/2,3}$$

and $$g_2(rho)=(x_2^2+chi^2_{1-alpha/2,3})rho^2-2(x_1x_2+x_2x_3)rho+x_1^2+x_2^2+x_3^2-chi^2_{1-alpha/2,3}$$

Am I right in considering a both-sided C.I.? After solving the quadratics in $$rho$$, I am guessing that the resulting C.I would be quite complicated.

Another suitable pivot is $$frac{mathbf1′ x}{sqrt{mathbf1’Sigma mathbf 1}}sim N(0,1)quad,,,mathbf1=(1,1,1)’$$

With $$bar x=frac{1}{3}sum x_i$$, this is same as saying $$frac{3bar x}{sqrt{3+4rho+2rho^2}}sim N(0,1)$$

Using this, I start with the inequality $$left|frac{3bar x}{sqrt{3+4rho+2rho^2}}right|le z_{alpha/2}$$

Therefore, $$frac{9bar x^2}{3+4rho+2rho^2}le z^2_{alpha/2}implies 2(rho+1)^2+1ge frac{9bar x^2}{z^2_{alpha/2}}$$

That is, $$rhoge sqrt{frac{9bar x^2}{2z^2_{alpha/2}}-frac{1}{2}}-1$$

Since the question asks for any confidence interval, there are a number of options available here. I could have also squared the standard normal pivot to get a similar answer in terms of $$chi^2_1$$ fractiles. I am quite sure that both methods I used are valid but I am not certain whether the resulting C.I. is a valid one. I am also interested in other ways to find a confidence interval here.

Get this bounty!!!

### Bounty: 50

Suppose $$Xsim N_3(0,Sigma)$$, where $$Sigma=begin{pmatrix}1&rho&rho^2\rho&1&rho\rho^2&rho&1end{pmatrix}$$.

On the basis of one observation $$x=(x_1,x_2,x_3)’$$, I have to obtain a confidence interval for $$rho$$ with confidence coefficient $$1-alpha$$.

We know that $$X’Sigma^{-1}Xsim chi^2_3$$.

So expanding the quadratic form, I get

$$x’Sigma^{-1}x=frac{1}{1-rho^2}left[x_1^2+(1+rho^2)x_2^2+x_3^2-2rho(x_1x_2+x_2x_3)right]$$

To use this as a pivot for a two-sided C.I with confidence level $$1-alpha$$, I setup $$chi^2_{1-alpha/2,3}le x’Sigma^{-1}xle chi^2_{alpha/2,3}$$

I get two inequalities of the form $$g_1(rho)le 0$$ and $$g_2(rho)ge 0$$, where

$$g_1(rho)=(x_2^2+chi^2_{alpha/2,3})rho^2-2(x_1x_2+x_2x_3)rho+x_1^2+x_2^2+x_3^2-chi^2_{alpha/2,3}$$

and $$g_2(rho)=(x_2^2+chi^2_{1-alpha/2,3})rho^2-2(x_1x_2+x_2x_3)rho+x_1^2+x_2^2+x_3^2-chi^2_{1-alpha/2,3}$$

Am I right in considering a both-sided C.I.? After solving the quadratics in $$rho$$, I am guessing that the resulting C.I would be quite complicated.

Another suitable pivot is $$frac{mathbf1′ x}{sqrt{mathbf1’Sigma mathbf 1}}sim N(0,1)quad,,,mathbf1=(1,1,1)’$$

With $$bar x=frac{1}{3}sum x_i$$, this is same as saying $$frac{3bar x}{sqrt{3+4rho+2rho^2}}sim N(0,1)$$

Using this, I start with the inequality $$left|frac{3bar x}{sqrt{3+4rho+2rho^2}}right|le z_{alpha/2}$$

Therefore, $$frac{9bar x^2}{3+4rho+2rho^2}le z^2_{alpha/2}implies 2(rho+1)^2+1ge frac{9bar x^2}{z^2_{alpha/2}}$$

That is, $$rhoge sqrt{frac{9bar x^2}{2z^2_{alpha/2}}-frac{1}{2}}-1$$

Since the question asks for any confidence interval, there are a number of options available here. I could have also squared the standard normal pivot to get a similar answer in terms of $$chi^2_1$$ fractiles. I am quite sure that both methods I used are valid but I am not certain whether the resulting C.I. is a valid one. I am also interested in other ways to find a confidence interval here.

Get this bounty!!!

### Bounty: 50

Suppose $$Xsim N_3(0,Sigma)$$, where $$Sigma=begin{pmatrix}1&rho&rho^2\rho&1&rho\rho^2&rho&1end{pmatrix}$$.

On the basis of one observation $$x=(x_1,x_2,x_3)’$$, I have to obtain a confidence interval for $$rho$$ with confidence coefficient $$1-alpha$$.

We know that $$X’Sigma^{-1}Xsim chi^2_3$$.

So expanding the quadratic form, I get

$$x’Sigma^{-1}x=frac{1}{1-rho^2}left[x_1^2+(1+rho^2)x_2^2+x_3^2-2rho(x_1x_2+x_2x_3)right]$$

To use this as a pivot for a two-sided C.I with confidence level $$1-alpha$$, I setup $$chi^2_{1-alpha/2,3}le x’Sigma^{-1}xle chi^2_{alpha/2,3}$$

I get two inequalities of the form $$g_1(rho)le 0$$ and $$g_2(rho)ge 0$$, where

$$g_1(rho)=(x_2^2+chi^2_{alpha/2,3})rho^2-2(x_1x_2+x_2x_3)rho+x_1^2+x_2^2+x_3^2-chi^2_{alpha/2,3}$$

and $$g_2(rho)=(x_2^2+chi^2_{1-alpha/2,3})rho^2-2(x_1x_2+x_2x_3)rho+x_1^2+x_2^2+x_3^2-chi^2_{1-alpha/2,3}$$

Am I right in considering a both-sided C.I.? After solving the quadratics in $$rho$$, I am guessing that the resulting C.I would be quite complicated.

Another suitable pivot is $$frac{mathbf1′ x}{sqrt{mathbf1’Sigma mathbf 1}}sim N(0,1)quad,,,mathbf1=(1,1,1)’$$

With $$bar x=frac{1}{3}sum x_i$$, this is same as saying $$frac{3bar x}{sqrt{3+4rho+2rho^2}}sim N(0,1)$$

Using this, I start with the inequality $$left|frac{3bar x}{sqrt{3+4rho+2rho^2}}right|le z_{alpha/2}$$

Therefore, $$frac{9bar x^2}{3+4rho+2rho^2}le z^2_{alpha/2}implies 2(rho+1)^2+1ge frac{9bar x^2}{z^2_{alpha/2}}$$

That is, $$rhoge sqrt{frac{9bar x^2}{2z^2_{alpha/2}}-frac{1}{2}}-1$$

Since the question asks for any confidence interval, there are a number of options available here. I could have also squared the standard normal pivot to get a similar answer in terms of $$chi^2_1$$ fractiles. I am quite sure that both methods I used are valid but I am not certain whether the resulting C.I. is a valid one. I am also interested in other ways to find a confidence interval here.

Get this bounty!!!