The situation is as following:
I have N data points which represent past measurements. Then I have a new single measurement and I’d like to know the difference of this new measurement based on the past measurements. This measurement is known to not be accurate and thus the N data points are used to establish of how accurate it is because we’re always measuring the same value and thus the measurement should always output the same value but it doesn’t do that. I want to know whether the value has actually changed or not and by how much – that is – with some degree of certainty. I figured the easiest way to do this would be to calculate average and standard deviation of the N data points and assume a normal distribution and then see whether the new measurement falls within a certain threshold (i.e. the new measurement must fall into the <5% range). The question is how do I extend this to multiple objects being measured. The accuracy of the measurement depends on the object being measured. What I can easily do is as mentioned is for each new single measurement of an object calculate how likely this happens by chance but how do I use this to make a statement about a group of objects? I was thinking of doing this: for every object I calculate the difference between the new single measurement and the average of the N past measurements and then I multiply this with the probability that it is a significant change.
I think an example should demonstrate this better:
I have N measurements and av=24.0,sd=2.0 and the new single measurement is 26.5 and according to the normal distribution P(X >= 26.5) is 10.5% so the probability that this isn’t by chance is 0.895. The difference from the average multiplied by this would be 2.5*0.895 = 2.24 so I conclude that for this object the “effective” difference is 2.24. Then I have another object and the “effective” difference is 2.1 and the average of all differences would thus be (2.24+2.1)/2 = 2.17 and the standard deviation 0.01.
Prior Measurements for A: [22, 24, 26] => AV=24,SD=2.0 Prior Measurements for B: [40, 40, 40] => AV=40,SD=0.0 Prior Measurements for C: [25, 30, 33] => AV=29.3,SD=4.04 Measurement for A: 26.5, p=0.895 Measurement for B: 38, p=1.0 Measurement for C: 31, p=0.663 Effective differences: A: 2.5*0.895 = 2.24 B: -2*1.0 = -2.0 C: 1.7*0.663 = 1.13 Average of effective differences: (2.24 - 2.0 + 1.13)/3 = 1.37.
Is this a useful method?