Some things cannot be forecast based on historical experience. Our knowledge of the past is limited. Even if we think we have a solid retrospective set of data, there are events — infrequent, but dramatic — that overwhelm our most carefully-built models. Some examples include:
- stunning performances by prodigies in music, athletics, chess, mathematics, etc.;
- violent market fluctuations, both up and down, in stocks, bonds, and commodity prices;
- unprecedented floods, earthquakes, hurricanes, blizzards, and other natural disasters;
- incredible "accidents", on land, sea, air, underground, and in space, that occur by the conjunction of unanticipated factors.
Why do good models fail? Sometimes the data we have are incomplete, and catastrophes actually have happened in the past — but we don't know about them because they came too long ago or in out-of-the-way corners of the world. Sometimes our models have subtle "bugs" in them, logical errors that testing has failed to reveal.
But more often, our forecasts go awry because things change. New influences emerge; patterns of behavior evolve; systems are no longer what they once were. Things change in nature, as atmospheric gas levels wax and wane, as ocean currents shift, as volcanoes erupt, as continents drift. And more suddenly, things change as humans choose and act. Profits are momentarily high in an economic sector, so investors plunge in. A dramatic incident is widely reported, so people shy away from a mode of transportation, or a medical treatment, or a type of food. Not rational in the long run, perhaps ... but it happens.
Scientists have studied noise and fluctuations in physical systems of all sorts, and have found patterns that help explain and predict events. One particularly powerful technique analyzes changes by frequency. Some prices, for instance, go up and down on an annual basis, perhaps because they're tied to climate or holiday spending or vacation schedules. Other activities have prominent daily, weekly, or monthly cycles. Sunspots have an ~11 year period; cicadas emerge every 17 years; eclipses recur in an ~18.03 year saros pattern.
Plotting the size of fluctuations against their frequency, "f", gives a power spectrum, analogous to a spectrum of light. Peaks and valleys tell something about the likelihood of events on various timescales. A power spectrum that's flat, equal at all frequencies, is called "white noise". A random walk, on the other hand, has a lot of long-term drift, but its high-frequency components cancel out. Its power spectrum falls like one over frequency squared.
In between the total chaos of white noise and the predictable, lethargic meanderings of random walks, there's the extraordinarily important zone called "1/f". The power spectrum of 1/f noise describes a host of interesting systems, from the flooding of the Nile to the ticking of an atomic clock, from the insulin dose of a diabetic to the noise in an electronic circuit.
A 1/f spectrum is divergent at low frequencies. In other words, rare but huge fluctuations are gonna occur — and the longer we wait, the bigger the spikes will be. There's no way to know when they're coming, but come they must. The bottom line: just because stocks (or anything else) have moved in such-and-such a pattern for years (or decades or centuries) doesn't mean that this pattern will persist.
Rather, the only certainty is change — of a form and magnitude beyond all expectation. We'd best keep our powder dry!
(see LongTails)
Tuesday, September 14, 1999 at 11:58:52 (EDT) = 1999-09-14
(correlates: LongTails, ChatTuringTest, TemporalUtilitarianism, ...)