Last week, I finished Nate Silver’s book The Signal and the Noise. In the final chapter, he used research and anecdotes about the US counter-terrorism field to address the point that when forecasting future events, we have to be mindful of the “unknown unknowns”. The signals missed or not assigned enough importance in historical attacks like WWII’s Pearl Harbor and 9/11 are unknown unknowns because they were beyond the scope of imagination, and therefore no probability was even assigned to them. If the national security intelligence had considered the probability of these events, then lots of useful signals wouldn’t have passed by unattended, but instead would have been used to adjust the probability of these events, therefore leading to different outcomes. Adopting probabilistic thinking and realizing there are unknown unknowns will be helpful in all fields of forecasting.
Silver’s take also reminds me of the portrayal of counter-terrorism in the hit drama Homeland (spoilers ahead). The show’s protagonist, CIA analyst Carrie Matheson, is often portrayed as a loose cannon who pursues leads that counter the conventional wisdom or are regarded as impossible by her peers. For example, she is the only person in the agency who even suspects that returning war hero Sgt. Brody could be turned by terrorists, based simply from a piece of intel she gathered from the field. Carrie refuses to discard her far-fetched theory and eventually builds enough evidence to demonstrate a realistic probability Brody really has been turned. For drama’s sake, Homeland often shows Carrie as overly emotional due to her bipolar personality, and characterizes her findings as “gut feelings” by quickly panning over the huge amount of data she has collected and the maps on her wall linking together random events. As an experienced 14-year analyst with rich field experience and so much data in her hand, her knowledge about the subject exceeds that of her peers, which allows her to reduce the unknown unknowns in her knowledge scope, and helps her perceive unlikely events as possible.
In statistics, there are measurements of central tendency, but what are also important are the variance in each data set, the outliers, the whales, which shouldn’t be ignored. Instead these factors should be properly considered and weighed in their context, and help reduce the number of unknown unknowns for us.