In a hard real-time system, it is useful to know the average values of the timing stats but it is even more important to know the worst-case (or min/max) values of the timing stats.

When the number of samples is very large, the average of those samples can be close to the ideal value. Undesirable events that occur periodically may grossly alter the system performance for a tiny moment but they may never skew the average value the slightest bit.

As a result, the “average” is a very misleading indicator of performance when the number of samples is large. Even the “variance” or “standard deviation” will not be helpful because after all, they are second order averages. This behavior is summarized as the Law of Large Numbers.

Soft real-time systems are designed so that glitches are expected and tolerated. In mission critical, hard real-time systems glitches are considered catastrophic. Guarding against such failures is dependent on the problem space. However a good starting point is instrumenting your system well so that histograms, windowed averages, maximum, minimum, and other critical data points can be obtained easily.