#StackBounty: #statistical-significance #anova #nonparametric #multiple-comparisons #wilcoxon-mann-whitney Multiple comparisons between…

Bounty: 50

My experiment comprised two groups, control (N=25) and experimental (N=26). Each participant belonged to one group. Their performance has been tested three times (on a ratio scale from 0-60), once before a training course, and twice after. My spreadsheet contains the following variables:

UserID, Group, T1, T2, T3

First of all, I wanted to check whether there are any significant differences between T1, T2, and T3 inside a group, and therefore analyzed the two groups independently. I planned to use non-parametric tests beforehand as I was not sure how the sets would look like. Density graphs showed that almost all of the sets do not approximate a normal distribution, so I stick to my decision. As my sets (T1, T2, T3) were paired, I used the Friedman pre-hoc test to determine whether there is any significant difference followed by the Dunn-Bonferroni test to determine where the differences lie (as offered by SPSS). This worked fine.

Question: Now I want to carry out a between-group comparison. More precisely, I want to check whether one group performs better than the other on T1, T2, or T3.
This leads to three tests:

Control T1 vs Experimental T1
Control T2 vs Experimental T2
Control T3 vs Experimental T3

This time, I have to run 3 tests with 2 unpaired sets each. I know that I can run three Mann-Whitney-U tests and adjust the p-value at the end (post-hoc). This procedure tells me where the differences actually lie. But, is there a pre-hoc test to determine whether there are any differences? If so, how can I perform it under SPSS or R?

I know this question has been asked quite a few times (e.g., here Is there an equivalent to Kruskal Wallis one-way test for a two-way model? and here https://www.researchgate.net/post/Is_there_a_nonparametric_test_equivalent_to_a_2x3_ANOVA), but I am not exactly sure whether 1) such a “complex” operation is doable using one hypothesis test only and 2) which of the proposed methods are suitable in my case.

  • Ioannis K

UPDATE

I provide my data for better clarification:

Test A:

Control T1       = [14, 14, 14, 10, 22, 20, 31, 21, 28, 10, 12, 22, 33, 28, 15, 8, 7, 16, 18, 22, 25, 33, 2, 24, 18, 26, 27, 29, 27, 9] (Mean = 19.5)
Experimental T1  = [25, 28, 15, 20, 34, 20, 27, 17, 22, 16, 7, 16, 9, 16, 23, 8, 30, 20, 17, 14, 15, 22, 21, 23, 20, 18, 12, 17]        (Mean = 19)

Test B:

Control T2       = [14, 10, 14, 21, 27, 22, 16, 2, 7, 11, 25, 20, 20, 9, -1, 17, 20, 18, 19, 10, 21, 17, 19, 20, 26] (Mean = 16.16)
Experimental T2  = [22, 25, 12, 20, 30, 17, 24, 17, 16, 17, 7, 5, 4, 14, 21, 14, 23, 17, 8, 15, 22, 22, 24, 13, 12]  (Mean = 16.84)

Test C:

Control T3       = [0, -4, 0, 1, -4, 1, -12, -8, -5, -11, -8, -8, 5, 1, -8, 1, -2, -7, -14, 8, -3, -9, -8, -9, -1] (Mean = -4.16)
Experimental T3  = [-3, -3, -3, 0, -4, -3, -3, 0, -6, 1, 0, -11, -5, -2, -2, 6, -7, 0, -6, 0, 0, 1, 1, 1, -5]      (Mean = -2.12)

T3 has been performed after a three-month hiatus. Therefore I expected a little drop.

Now I want to check whether it holds A = B = C or mu(Control T1) = mu(Experimental T1) AND mu(Control T2) = mu(Experimental T2) AND mu(Control T3) = mu(Experimental T3).
So I performed three Mann-Whitney-U tests (two-tailed, 5%):

Test                        U           z           p (raw)     p adj (holm)
A: ControlT1-ExperimentalT1 U=441       0.327220    0.743502    1
B: ControlT2-ExperimentalT2 U=301       -0.223595   0.823073    1
C: ControlT3-ExperimentalT3 U=234.50    -1.522182   0.127963    0.3912776

The differences are minimal and not statistically significant. Both groups performed the same at any given T.

Now the question is, whether theres an ANOVA-ish test for my non-parametric 2×3 matrix of sets. I know there is no significant difference given any T..If I would have known this in advance, then I wouldnt run three post-hoc tests to find where differences exactly lie that are actually not there.


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.