Jason Healey and Neil Jenkins, Columbia University and Cyber Threat Alliance

It is now the policy of the United States to use offensive cyber operations to force adversaries to stop or reduce their attacks. Critics believe such counteroffensive activity may only inflame adversaries, seeing the new U.S. actions as aggressive themselves, something to counter rather than back down from. There is currently no methodology to measure which is the case. This talk introduces a standard and transparent framework for the cyber threat intelligence community to better assess whether the new U.S. deterrence strategy and actions are suppressing or encouraging attacks.

Proving that deterrence is working is obviously hard. If incidents decrease, was this due to U.S. actions or other factors, as the community experienced after the Obama-Xi agreement of 2015. It can be easier, fortunately, to measure failure. If Chinese espionage operations skyrocketed after that agreement, it is a clearer signal. For cyber deterrence, John Bolton himself described a quick measure of effectiveness when announcing the new policy, saying the OPM intrusion was just “the kind of threat to privacy from hostile foreign actors that we're determined to deter.” So at the most basic level, if there is an increase in OPM-style incidents, it is at least suggestive the new strategy is not successful.

Of course, we can do better than just this gross metric. Other metrics could include several categories:

  • Volumetric metrics: Measures such as the number of particular kinds of incidents increasing or decreasing over time? Example: Number of cyber intrusions into companies apparently for commercial purposes.

  • Target set metrics: Measures of adversaries gaining access to particular target sets which the US has declared out of bounds. Example: Are adversaries intruding into election infrastructure or the electrical grid?

  • Metrics of Recklessness: Measures of adversaries conducting attacks well beyond norms such as having global or systematic effect, especially if unintentionally. Example: NotPetya and WannaCry

  • Metrics of Brazen-ness. Measures of adversaries crossing specific thresholds, especially causing death and physical destruction, especially if done intentionally and outside a zone of active conflict. Example: Perhaps Shamoon, but worse.

Such metrics have obvious flaws: they cannot prove causality and are not based on known U.S. deterrence actions, only the overall policy and pace of adversary attacks. Still, the community needs some framework to help decide this important policy question as we saw with the arguments over whether Obama-Xi agreement was working or not. A transparent and repeatable framework is of critical importance.