#StackBounty: #confidence-interval #weighted-mean Better confidence intervals for weighted average

Bounty: 200

Suppose I have a large sequence of size $M$ which contains $K$ unique items, where item $k$ occurs with unknown probability $pi_k$. I can choose to measure its quality, $x_k$, which is constant for a given item $k$.

My goal is to estimate the average quality (i.e., the true weighted average as well as CI around it):

$$frac{1}{K}sum_{k=1}^K pi_k x_k$$

One plan is to get a uniform sample of items $J$ from this sequence, and compute the average over each sampled item (since item $k$ is sampled with probability $pi_k$):

$$frac{1}{|J|} sum_{j in J} x_j$$

and estimate the variance of the estimator using the usual CLT-based approach.

Suppose, however, it’s also easy to compute the total number of times each item occurs, $(n_1, …, n_K)$. Can I use this information to produce estimates with smaller confidence intervals?

Not to bias the potential answers, but I feel like it should be possible to do, since I will have more information about $pi$, and therefore should be able to do some sort of variance reduction technique.

Also, to work through a specific example, I’ve been using the following distribution which mimics my actual usecase.

import numpy as np

# Suppose we K unique items
K=10000
freq = np.array([K/(i+100) for i in range(K)])
true_pi = freq / sum(freq)
true_x = np.array([.8 - .4*i/K for i in range(K)])


Get this bounty!!!

#StackBounty: #confidence-interval #maximum-likelihood #quantiles #uncertainty #error-propagation Uncertainty propagation for the solut…

Bounty: 50

I have a dataset and I use Maximum Likelihood Estimation to estimate the values of the parameters of a weibull distribution. The MLE theory provides with theoretical Confidence Intervals (asymptotical, or for $n$ samples).

Then, I use the fitted Weibull distribution in an expression which is currently optimised numerically :

$Y(t_0) = h(t_0) . int_{0}^{t_0} S(t) dt + S(t_0)$

Where $t_0$ is unknown and $h$ and $S$ are the hazard function and the survival function of the distribution, and therefore are functions of the parameters.

I would like to propagate uncertainty on the fitted weibull parameters to estimate confidence intervales or quantiles for Y(t_0), how could I do that (numerically or analytically) ?
Thanks !


Get this bounty!!!

#StackBounty: #confidence-interval #poisson-distribution #bernoulli-process confidence intervals for the Poisson process (lambda) samp…

Bounty: 50

Say, I have a Poisson process which was measured $N$ times, and each measurement produced $k_i$ value.
Also, $k_i$ are events that I have to detect and my detection probability is $p$. In fact, I detect $widetilde{k}_i$ which correlates with $k_i$ via $p$.

Knowing, the number of the detected events $widetilde{k}_i$, the constant probability $p$, and the constant measurement time $T$, how I find the confidence interval for estimation of Poisson’s $lambda$?


Get this bounty!!!

#StackBounty: #confidence-interval #poisson-distribution #bernoulli-process confidence intervals for the Poisson process (lambda) samp…

Bounty: 50

Say, I have a Poisson process which was measured $N$ times, and each measurement produced $k_i$ value.
Also, $k_i$ are events that I have to detect and my detection probability is $p$. In fact, I detect $widetilde{k}_i$ which correlates with $k_i$ via $p$.

Knowing, the number of the detected events $widetilde{k}_i$, the constant probability $p$, and the constant measurement time $T$, how I find the confidence interval for estimation of Poisson’s $lambda$?


Get this bounty!!!

#StackBounty: #confidence-interval #p-value #model #model-comparison p value for difference in model outcomes

Bounty: 50

I’ve run two different linear mixed effects models on the same data and got two different estimates for the gradient of the longitudinal variable. e.g.

model 1 has estimate 30 with standard error 5.
model 2 has estimate 40 with standard error 4.

I’m interested in calculating a p value for the probability that the models are different, from the estimate and standard error. How do I do this? I’m aware that checking for overlap in the 95% confidence intervals is a bad idea, and that overlapping 83% CIs are a better test, but would like to be able to quantify this with a p value.


Get this bounty!!!

#StackBounty: #confidence-interval #standard-error #fitting #measurement-error #error-propagation Correct way to combine 95% confidence…

Bounty: 50

I am looking for someone to just confirm / double-check something for me with regards to errors on measurements.

Let’s say I am trying to determine the slope of a relationship by varying one quantity and measuring another, and then I plot the graph and do a least-squares fit straight line to the data (graph on the left). Then I repeat this procedure twice more, to get the middle and right-most graphs.
enter image description here

Each fit routune will typically give me back a slope and the corresponding 95% confidence interval, so that I obtain $(m_1pmDelta m_1), (m_2pmDelta m_2)$ and $(m_3pmDelta m_3)$. Now I know that the underlying quantity which determines $m$ in each case is the same, so I should be able to quote a best estimate for the slope as their mean

$$
bar{m} = frac{m_1+m_2+m_3}{3}. tag{1}
$$

My question is about the appropriate way to quote the error. We know that for a function $f(x,y)$ with errors in $x$ and $y$ given by $Delta x$ and $Delta y$, respectively, the error on $f$ is given by

$$
Delta f = sqrt{ (Delta x)^2 bigg(frac{partial f}{partial x}bigg)^2 + (Delta y)^2 bigg(frac{partial f}{partial y}bigg)^2 } tag{2}
$$

So I would think I can determine the error in $bar{m}$ to be

$$
begin{align}
Delta bar{m} &= sqrt{ (Delta m_1)^2 bigg(frac{partial bar{m}}{partial m_1}bigg)^2 + (Delta m_2)^2 bigg(frac{partial bar{m}}{partial m_2}bigg)^2 + (Delta m_3)^2 bigg(frac{partial bar{m}}{partial m_3}bigg)^2} tag{3} \
&= frac{1}{3} sqrt{ (Delta m_1)^2 + (Delta m_2)^2 + (Delta m_3)^2 } tag{4}
end{align}
$$

First question, is this correct?

Second question, is it okay to propagate 95% confidence intervals in this way? Should I simply quote now the result as $bar{m} pm Delta bar{m}$ and just explain that $Delta bar{m}$ is the combined 95% confidence interval, or should I convert the 95% number from the fits into standard errors (through the factor of 1.96)?

Thanks in advance,

(I am for now assuming Gaussian errors everywhere.)


EDIT

It was suggested in the comments that I first implement weighting in the averaging step before worrying about the errors. This should help to give more weight to slopes which have tighter confidence intervals (and vice versa).

According to this link, the weighted version of the mean would be given by
$$
bar{m}_textrm{w} = frac{sum_i w_i m_i}{sum_iw_i}, hspace{1cm} textrm{where} hspace{0.5cm} w_i = frac{1}{sigma_i^2}tag{5}
$$

and $sigma_i$ is the variance of each slope. Therefore, in my case with the three example slopes, it should be
$$
bar{m}_textrm{w} = frac{m_1/sigma_1^2 + m_2/sigma_2^2 + m_3/sigma_3^2}{1/sigma_1^2 + 1/sigma_2^2 + 1/sigma_3^2}. tag{6}
$$

The variance on the weighted mean slope is given at the above link again by
$$
begin{align}
textrm{Var}(bar{m}_textrm{w}) &= frac{sum_iw_i^2sigma_i^2}{big( sum_iw_ibig)^2}tag{7}\
&= frac{1/sigma_1^2 + 1/sigma_2^2 + 1/sigma_3^2}{big(1/sigma_1^2 + 1/sigma_2^2 + 1/sigma_3^2big)^2}tag{8}\
&= big(1/sigma_1^2 + 1/sigma_2^2 + 1/sigma_3^2big)^{-1}.tag{9}
end{align}
$$

So now my main question remains – these are variances, so should we convert the 95% confidence intervals $Delta m_i$ returned by a fitting algorthm somehow into a variance?

Maybe for a concrete example we could imagine the following values were returned from the fitting routine:
$$
begin{align}
m_1 &= 5.5; (4.9, 6.1)rightarrow Delta m_1 = 0.6\
m_2 &= 5.5; (5.3, 5.7)rightarrow Delta m_2 = 0.2\
m_3 &= 5.2; (4.5, 5.9)rightarrow Delta m_3 = 0.7
end{align}
$$

where the values in brackets represent the 95% confidence intervals. How should the estimate of the slope be reported, including errors?


Get this bounty!!!

#StackBounty: #r #confidence-interval #growth-model #logistic-curve Confidence Intervals for the Parameters of a Logistic Growth Curve

Bounty: 100

I’m using the growthcurver library to estimate the parameters from some bacterial growth curves. It fits a logistic equation of the following form:

enter image description here

This returns values for K, N0 and r in that equation, along with standard errors. I would like, ideally, to turn those into 95% confidence intervals.

Are these in the sort of standard estimate +/- 1.96*se form, or do they need any transformation (ala the coefficients and standard errors in logistic regression model). My suspicion is no, but I want to verify that.

So for example, with an r of 1.119 and an se of 0.015, is the 95% confidence interval (1.09,1.15)?


Get this bounty!!!

#StackBounty: #confidence-interval #cross-validation #modeling #standard-error #uncertainty K-fold cross validation based standard erro…

Bounty: 50

I have an expensive model (or class of models). My baseline approach to quantify uncertainty re the model parameters are hessian based standard errors, and I use k-fold cross validation for model comparison / validation. While a full bootstrap would be pleasant as a more robust uncertainty quantification, this is quite expensive. I think I should also be able to develop expectations for the variance of the leave-k out estimates, to at least get a rough sense of where the hessian based standard error estimates are not performing well. I wonder if someone knows how to do this, or can point to work that does this? Something like an approximate jackknife?


Get this bounty!!!

#StackBounty: #confidence-interval #python #standard-error #cohens-kappa Calculating the Standard Error and Confidence Interval for Coh…

Bounty: 50

I need to evaluate the performance of a machine learning application. One of the evaluation metrics chosen is Cohen’s Quadratic Kappa. I found this Python tutorial on how to calculate Cohen’s Quadratic Kappa. What is missing, however, is how to calculate the confidence interval.

Let’s walk through my example (I use a smaller data set for the sake of simplicity). I use NumPy and Scipy Stats for this purpose:

from math import sqrt
import numpy as np
from scipy.stats import norm

This is my confusion matrix:

# x: actuals, y: predictions
confusion_matrix = np.array([
    [9, 5, 2, 0, 0, 0],
    [4, 7, 1, 0, 0, 0],
    [1, 2, 4, 0, 1, 0],
    [0, 1, 1, 5, 1, 0],
    [0, 0, 0, 1, 2, 1],
    [0, 0, 0, 0, 0, 1],
], dtype=np.int)
rows = confusion_matrix.shape[0]
cols = confusion_matrix.shape[1]

I calculate a weight matrix and histograms:

weights = np.zeros((rows, cols))
for r in range(rows):
    for c in range(cols):
        weights[r, c] = float(((r-c)**2)/(rows*cols))
hist_actual = np.sum(confusion_matrix, axis=0)
hist_prediction = np.sum(confusion_matrix, axis=1)

The expected prediction quality by mere chance is calculated as follows:

expected = np.outer(hist_actual, hist_prediction)

This matrix, and the actual confusion matrix, are normalized:

expected_norm = expected / expected.sum()
confusion_matrix_norm = confusion_matrix / confusion_matrix.sum()

Now I calculate the numerator (actual observed agreement) and the denominator (expected agreement by chance):

for r in range(rows):
    for c in range(cols):
        numerator += weights[r, c] * confusion_matrix_norm[r, c]
        denominator += weights[r, c] * expected_norm[r, c]

Cohen’s Kappa can now be calculated as:

weighted_kappa = (1 - (numerator/denominator))

Which gives me a result of 0.817.

Now to my question: I need to calculate the standard error, in order to calculate the confidence interval. Here’s my approach:

#            p(1-p)
# sek = sqrt -------
#            n(1-e)²
#
# p: numerator (actual observed agreement)
# e: denominator (expected agreement by chance)
# n: total number of predictions
total = hist_actual.sum()
sek = sqrt((numerator * (1 - numerator)) / (total * (1 - denominator) ** 2))

Can I use the total number of predictions, even though I calculate with a normalized numerator and denominator? This would result in a standard error of kappa of 0.023.

The 95% confidence interval then is just straightforward:

alpha = 0.95
margin = (1 - alpha) / 2  # two-tailed test
x = norm.ppf(1 - margin)
lower = weighted_kappa - x * sek
upper = weighted_kappa + x * sek

Which gives an interval of [0.772;0.861].


Get this bounty!!!

#StackBounty: #mathematical-statistics #confidence-interval #maximum-likelihood #asymptotics How can I obtain an asymptotic $1-alpha$ …

Bounty: 50

Let $X sim Gamma(alpha,1)$ and $Y|X=x sim Exp(frac{1}{theta x}), alpha >1$ and $theta >0$ are unknown. Let $tau=E(Y)$. Suppose that based on the random sample $Y_1,…,Y_n$, we have MLEs, $hat{alpha}$ and $hat{theta}$. Use these MLEs to develop an asymptotic $1-alpha$ confidence interval for $tau$.

my work:

First, I need to find $tau=E(Y)=E(frac{1}{theta x})=frac{1}{theta}E(frac{1}{x})$. We use a transformation of $T=frac{1}{X}$, where $f_T(t)=frac{1}{Gamma(alpha)t^{alpha+1}}e^{-1/t},t>0$. However, I am having trouble evaluating $E(T)=int^infty_0frac{1}{Gamma(alpha)t^{alpha}}e^{-1/t}dt$.

Assuming we have $tau$, we can get the asymptotic $1-alpha$ CI by using the asymptotic property of MLE. We know that $hat{alpha}sim AN(alpha,frac{1}{ni(alpha)})$ and $hat{theta} sim AN(theta,frac{1}{ni(theta)})$. However, I am failing to see how I can obtain the asymptotic CI for $tau$.


Get this bounty!!!