Why it can be better to wait and see after new and exciting results

It can be hard not to attach expectations to the value of the first report or two that come out after a big new policy or program change on how things are going. Although a change in either a positive or negative direction are often and easily interpreted as a signal that things are going well or poorly right away, it’s important to give some time for adjustment.

A common phenomenon in data is something called regression to the mean. It’s a fancy way of saying that things will even out over time. The practical implication of this is that abnormally high or low numbers will generally be followed by more moderate numbers, which is why it’s best not to call progress over time a trend until you get several data points suggesting otherwise. It’s also the underlying reason why sample size and repetition matter in studies: a big positive or negative influence reported by one study could be an extreme result, and more testing can show whether any regression (less positive or negative results) happens to get a better understanding of what the expected/average results should be.

How long should you wait and see? It depends on the change. A policy that affects quarterly enrollment, for example, could start to take shape after about a year, whereas a comprehensive redesign could take several years as it takes time to readjust, work out the kinks, and review.