#StackBounty: #forecasting #model-evaluation #kolmogorov-smirnov-test #calibration #pit Assess calibration of a density forecast by Kol…

Bounty: 50

According to Elliott & Timmermann "Economic Forecasting" (2016) p. 429-430,

Calibration requires that if a density forecast assigns a certain probability to an event,
then the event should occur with the stated probability over successive observations.
<…>
For any <…> event, $A$, if the associated density forecast $int_A p_Y (y|z)(y) dy = p$, calibration requires that $P(y_{t+1} in A)$ is indeed equal to $p$, conditional on the same information.

I wonder how one could assess calibration of a density forecast. I think Kolmogorov-Smirnov test applied on the probability integral transform (PIT) of realized values vs. the theoretical Uniform[0,1] distribution could be used for that. The PIT would be based on the distribution that is implied by the density forecast. However, use of the test is not mentioned in the textbook (it says Most attempts to examine calibration lead to informal rather than formal hypothesis tests and goes on to discuss some difficulties with assessing calibration), so I am probably missing something.

Q: Does Kolmogorov-Smirnov test applied on the probability integral transform (PIT) of realized values vs. the theoretical Uniform[0,1] distribution assess calibration of a density forecast?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.