#StackBounty: #false-discovery-rate Understanding the Benjamini-Hochberg method proof

Bounty: 100

I’m trying to understand the proof in Benjamini & Hochberg’s 1995 paper, specifically the Lemma in the appendix, as the rest of the proof is short and follows it.

I got stuck somewhere after equation (5)—where it says: "Thus all $m_0+j_0$ hypotheses are rejected"—why is that? By which procedure? Procedure (1) (=BH)? Or by using the cutoff declared earlier? (largest j satisfying $p_j le frac{m_0+j}{m+1}q^*$—which I understand is defined only for the False Null) This would only be true if we indeed fix the cutoff value, which is defined only on the False Null (i.e. discoveries) and simply reject any p-value below this. But I don’t immediately see how this is true if we use procedure (1)…

Also the first inequality of equation (6) seems to me wrong—it should be equality, as $p”$ is defined to be the value it’s replaced with…

In any case after that I completely get lost until equation (8). I have no idea how they arrive that there must be a $kle m_0+j-1$ for which $p_{(k)}le{k/(m+1)}q^*$—what is that $j$? Why $-1$?

From (8) onward it’s understood.

Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1), 289–300.


Get this bounty!!!

#StackBounty: #anova #multiple-comparisons #false-discovery-rate Multiple Comparisons Correction for Multiple ANOVA models – Include on…

Bounty: 50

I have a a biological dataset, in which I am fitting the following partial mixed ANOVA model (apologizes for the Matlab syntax):

Baseline_Bio_Data,Post_Bio_Data ~ Treatment_Group*Response_Group + Age + Sex + Baseline_Clinical Scores

Baseline_Bio_Data and Post_Bio_Data are my dependent variables, taken before and following a treatment.

Treatment_Group is a 2 level categorical variable (participants underwent one of two treatments with no randomization)

Response_Group is a 2 level categorial variable (those who responded to the treatment and those who did not)

And Age, Sex, and Baseline_Clinical Scores are covariates I’d like to control for.

Now, this leads to 6 between-level effects (treatment group, response_group, baseline_clinical scores, age, sex, and the interaction term) and 7 within-level effects (time and its interactions with the between-level variables). This leads to 13 estimates in total.

Now, I have to apply the same model to 100 different datasets in total (with each dataset representing a different part of the brain). In my field, the use of False Discovery Rate (FDR) for multiple comparisons correction is common.

My question is: if I am only interested in the effects that contain time, treatment group, and response group (totalling to 7 estimates), and not interested in my covariates, can I only correct only for those estimates I am interested in?

I initially went ahead and corrected for all estimates (applying FDR to a 13 estimate x 100 brain region = 1300 matrix of p-values). Would it be wrong to exclude the estimates of no interest from this matrix?

Thank you in advance.


Get this bounty!!!

#StackBounty: #least-squares #negative-binomial #false-discovery-rate Appropriate action to take once I've run 30 regressions with …

Bounty: 50

I run 30 OLS and GLM regressions using the same 9 IV/variables/features, but each with a slightly tweaked DV/target/label. More or less the same same variables are significant each time, but DV-tweaks produces interesting variations regarding which IVs are significant and which are not. I have captured it all in a table, and feel the IVs that keep surfacing time after time (say 25 out of 30 regressions) are better predictors that those that come up only once or twice (as significant). However, I feel I might be accused of fooling myself by running so many regressions. Should I be using to sort of correction or penalty? How is this done?

Note 1: I’m teaching myself statistics, and have rather a few gaps in my knowledge.

Note 2: I use OLS for all the versions of the Target which are continuous. I use a “Negative Binomial” for the others, because it is count data, and overdispersed.

Note 3: I look at the number of protests in municipalities (i.e. count), but then also at protests/capita ; protests*size of protests / capita ; violent protests only / capita (all per municipality), and so on.

Note 4: When the IVs are significant, they are properly so – p values less than 0.001.

Is this unease of mine something to do with “false discovery rates”? Am I way off course?

Speak, oh wise ones.


Get this bounty!!!

#StackBounty: #least-squares #negative-binomial #false-discovery-rate Appropriate action to take once I've run 30 regressions with …

Bounty: 50

I run 30 OLS and GLM regressions using the same 9 IV/variables/features, but each with a slightly tweaked DV/target/label. More or less the same same variables are significant each time, but DV-tweaks produces interesting variations regarding which IVs are significant and which are not. I have captured it all in a table, and feel the IVs that keep surfacing time after time (say 25 out of 30 regressions) are better predictors that those that come up only once or twice (as significant). However, I feel I might be accused of fooling myself by running so many regressions. Should I be using to sort of correction or penalty? How is this done?

Note 1: I’m teaching myself statistics, and have rather a few gaps in my knowledge.

Note 2: I use OLS for all the versions of the Target which are continuous. I use a “Negative Binomial” for the others, because it is count data, and overdispersed.

Note 3: I look at the number of protests in municipalities (i.e. count), but then also at protests/capita ; protests*size of protests / capita ; violent protests only / capita (all per municipality), and so on.

Note 4: When the IVs are significant, they are properly so – p values less than 0.001.

Is this unease of mine something to do with “false discovery rates”? Am I way off course?

Speak, oh wise ones.


Get this bounty!!!

#StackBounty: #least-squares #negative-binomial #false-discovery-rate Appropriate action to take once I've run 30 regressions with …

Bounty: 50

I run 30 OLS and GLM regressions using the same 9 IV/variables/features, but each with a slightly tweaked DV/target/label. More or less the same same variables are significant each time, but DV-tweaks produces interesting variations regarding which IVs are significant and which are not. I have captured it all in a table, and feel the IVs that keep surfacing time after time (say 25 out of 30 regressions) are better predictors that those that come up only once or twice (as significant). However, I feel I might be accused of fooling myself by running so many regressions. Should I be using to sort of correction or penalty? How is this done?

Note 1: I’m teaching myself statistics, and have rather a few gaps in my knowledge.

Note 2: I use OLS for all the versions of the Target which are continuous. I use a “Negative Binomial” for the others, because it is count data, and overdispersed.

Note 3: I look at the number of protests in municipalities (i.e. count), but then also at protests/capita ; protests*size of protests / capita ; violent protests only / capita (all per municipality), and so on.

Note 4: When the IVs are significant, they are properly so – p values less than 0.001.

Is this unease of mine something to do with “false discovery rates”? Am I way off course?

Speak, oh wise ones.


Get this bounty!!!

#StackBounty: #least-squares #negative-binomial #false-discovery-rate Appropriate action to take once I've run 30 regressions with …

Bounty: 50

I run 30 OLS and GLM regressions using the same 9 IV/variables/features, but each with a slightly tweaked DV/target/label. More or less the same same variables are significant each time, but DV-tweaks produces interesting variations regarding which IVs are significant and which are not. I have captured it all in a table, and feel the IVs that keep surfacing time after time (say 25 out of 30 regressions) are better predictors that those that come up only once or twice (as significant). However, I feel I might be accused of fooling myself by running so many regressions. Should I be using to sort of correction or penalty? How is this done?

Note 1: I’m teaching myself statistics, and have rather a few gaps in my knowledge.

Note 2: I use OLS for all the versions of the Target which are continuous. I use a “Negative Binomial” for the others, because it is count data, and overdispersed.

Note 3: I look at the number of protests in municipalities (i.e. count), but then also at protests/capita ; protests*size of protests / capita ; violent protests only / capita (all per municipality), and so on.

Note 4: When the IVs are significant, they are properly so – p values less than 0.001.

Is this unease of mine something to do with “false discovery rates”? Am I way off course?

Speak, oh wise ones.


Get this bounty!!!

#StackBounty: #least-squares #negative-binomial #false-discovery-rate Appropriate action to take once I've run 30 regressions with …

Bounty: 50

I run 30 OLS and GLM regressions using the same 9 IV/variables/features, but each with a slightly tweaked DV/target/label. More or less the same same variables are significant each time, but DV-tweaks produces interesting variations regarding which IVs are significant and which are not. I have captured it all in a table, and feel the IVs that keep surfacing time after time (say 25 out of 30 regressions) are better predictors that those that come up only once or twice (as significant). However, I feel I might be accused of fooling myself by running so many regressions. Should I be using to sort of correction or penalty? How is this done?

Note 1: I’m teaching myself statistics, and have rather a few gaps in my knowledge.

Note 2: I use OLS for all the versions of the Target which are continuous. I use a “Negative Binomial” for the others, because it is count data, and overdispersed.

Note 3: I look at the number of protests in municipalities (i.e. count), but then also at protests/capita ; protests*size of protests / capita ; violent protests only / capita (all per municipality), and so on.

Note 4: When the IVs are significant, they are properly so – p values less than 0.001.

Is this unease of mine something to do with “false discovery rates”? Am I way off course?

Speak, oh wise ones.


Get this bounty!!!

#StackBounty: #least-squares #negative-binomial #false-discovery-rate Appropriate action to take once I've run 30 regressions with …

Bounty: 50

I run 30 OLS and GLM regressions using the same 9 IV/variables/features, but each with a slightly tweaked DV/target/label. More or less the same same variables are significant each time, but DV-tweaks produces interesting variations regarding which IVs are significant and which are not. I have captured it all in a table, and feel the IVs that keep surfacing time after time (say 25 out of 30 regressions) are better predictors that those that come up only once or twice (as significant). However, I feel I might be accused of fooling myself by running so many regressions. Should I be using to sort of correction or penalty? How is this done?

Note 1: I’m teaching myself statistics, and have rather a few gaps in my knowledge.

Note 2: I use OLS for all the versions of the Target which are continuous. I use a “Negative Binomial” for the others, because it is count data, and overdispersed.

Note 3: I look at the number of protests in municipalities (i.e. count), but then also at protests/capita ; protests*size of protests / capita ; violent protests only / capita (all per municipality), and so on.

Note 4: When the IVs are significant, they are properly so – p values less than 0.001.

Is this unease of mine something to do with “false discovery rates”? Am I way off course?

Speak, oh wise ones.


Get this bounty!!!

#StackBounty: #least-squares #negative-binomial #false-discovery-rate Appropriate action to take once I've run 30 regressions with …

Bounty: 50

I run 30 OLS and GLM regressions using the same 9 IV/variables/features, but each with a slightly tweaked DV/target/label. More or less the same same variables are significant each time, but DV-tweaks produces interesting variations regarding which IVs are significant and which are not. I have captured it all in a table, and feel the IVs that keep surfacing time after time (say 25 out of 30 regressions) are better predictors that those that come up only once or twice (as significant). However, I feel I might be accused of fooling myself by running so many regressions. Should I be using to sort of correction or penalty? How is this done?

Note 1: I’m teaching myself statistics, and have rather a few gaps in my knowledge.

Note 2: I use OLS for all the versions of the Target which are continuous. I use a “Negative Binomial” for the others, because it is count data, and overdispersed.

Note 3: I look at the number of protests in municipalities (i.e. count), but then also at protests/capita ; protests*size of protests / capita ; violent protests only / capita (all per municipality), and so on.

Note 4: When the IVs are significant, they are properly so – p values less than 0.001.

Is this unease of mine something to do with “false discovery rates”? Am I way off course?

Speak, oh wise ones.


Get this bounty!!!