#StackBounty: #hypothesis-testing #post-hoc #friedman-test How must the null hypothesis be rejected for Nemenyi test to be allowed?

Bounty: 50

Suppose we have algorithms $A_k$ and $B_k$, $1leq kleq n$, and $alpha = 0.05$.

Friedman test for the algorithms $A_k$ shows that the null hypothesis that all $A_k$s perform equally well, can be rejected. Suppose $p_A = 10^{-5}llalpha$.

Friedman test for the algorithms $B_k$ shows that the null hypothesis that all $B_k$s perform equally well, cannot be rejected. Suppose $p_B = 0.4$.

My problem is that the $p$-value $p_{AB}$ from the Friedman test for $A_k$s and $B_k$s is greater than $alpha$, hence I cannot apply Nemenyi post-hoc test if I blindly follow the procedure

  • perform Friedman on algorithms
  • if $H_0$ is rejected, proceed to Nemenyi on the same algorithms

Question: Are the results from Nemenyi test for all algorithms still valid?

I am not sure, but I would say yes because

  • I have shown that already not all $A_k$s have the same performance (and the results would still be significant with $2p_A$, i.e., if I used Bonferroni correction for multiple-hypothesis testing)
  • Nemenyi for $A_k$s and $B_k$s found two groups of algorithms, i.e., some differences.


Get this bounty!!!