#StackBounty: #interaction #mediation Why do mediation and moderation analysis give substantively different results?

Bounty: 50

I have a study where I pair two people up and have them play a behavioral game together. I measure some change score before and after the game. I create a four-level factor variable called treatment that has the following values: MM, MF, FM, FF which describe your biological sex, and the biological sex of your partner.

Let’s say I also have some index moderator_idx where I believe, for people in the MF and FM category, high levels of moderator_idx will be associated with high levels of the outcome. So I model the following:

> m1 <- lm_robust(outcome ~ treatment / moderator_idx -1,
              cluster = team_id,
              se = "stata",
              data = data_full)



                             Estimate  Std. Error   t value    Pr(>|t|)      CI Lower     CI Upper  DF
treatmentFF                 9.6698108  4.31703853  2.239918 0.025776158   1.176747161 18.162874409 323
treatmentFM                -2.4975363  6.76639155 -0.369109 0.712288286 -15.809299377 10.814226686 323
treatmentMF                -6.5241575  5.42255272 -1.203152 0.229798459 -17.192138536  4.143823553 323
treatmentMM               -20.0332461 13.66149199 -1.466403 0.143511539 -46.909985777  6.843493496 323
treatmentFF:moderator_idx  -0.1041088  0.05135792 -2.027123 0.043470833  -0.205147088 -0.003070541 323
treatmentFM:moderator_idx   0.1580438  0.07965394  1.984131 0.048087366   0.001337787  0.314749868 323
treatmentMF:moderator_idx   0.1845383  0.06667904  2.767562 0.005973407   0.053358306  0.315718388 323
treatmentMM:moderator_idx   0.2405057  0.15315809  1.570310 0.117322055  -0.060807677  0.541819060 323

Examining only the interaction effects (e.g. treatmentFM:moderator_idx), I get a result that I suspected: for people in different-sex conditions (FM, MF), the effect of an increase in the moderator is associated with significant increases in the outcome of interest.

However, I can re-cast this analysis as a mediation analysis like so (where different_sex is a dummy variable set to 1 if you’re paired with someone in the opposite sex, and sex is your own biological sex). Note that this is equivalent to a four-level treatment factor above…

med.fit <- lm(moderator_idx ~ different_sex * sex, data = data_full)
out.fit <- lm(outgroup_feelings_diff ~ sex * different_sex * moderator_idx, data = data_full)
med.out <- mediation::mediate(med.fit, out.fit, treat = "different_sex", mediator = "moderator_idx", robustSE = TRUE, sims = 1000)
summary(med.out)

Quasi-Bayesian Confidence Intervals

                         Estimate 95% CI Lower 95% CI Upper p-value    
ACME (control)            -0.0484      -0.5168         0.37    0.84    
ACME (treated)            -0.6666      -1.7546         0.20    0.14    
ADE (control)             11.2765       7.7503        14.58  <2e-16 ***
ADE (treated)             10.6583       7.2103        13.88  <2e-16 ***
Total Effect              10.6099       7.2444        13.87  <2e-16 ***
Prop. Mediated (control)  -0.0032      -0.0525         0.04    0.84    
Prop. Mediated (treated)  -0.0620      -0.1759         0.02    0.14    
ACME (average)            -0.3575      -0.9851         0.13    0.15    
ADE (average)             10.9674       7.5714        14.19  <2e-16 ***
Prop. Mediated (average)  -0.0326      -0.1000         0.01    0.15    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

And I get that ACME of the treated is non-significant. My question is: How can I get in the first analysis large and significant effects of moderation, but non-significant effects of mediation. What are the substantive differences between the two results, and which should I trust?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.