Background: Discussion with John Hanley (JH), Director of Strategy for Workplace of the Director of Nationwide Intelligence (ODNI). Prior to joining ODNI, JH served as an officer in the US Navy and held senior positions in the DOD. I requested John broad questions that primarily targeted on three topical areas – the distinction between analysis and analytic communities within the DOD and Intelligence Group (IC) based mostly on his experiences, his views on the present state of the IC given his position and place within the ODNI, and common comparisons between intelligence analysis and educational scholarship. Word: since the time of this interview on February 9, 2012, Dr. Hanley has retired from the ODNI.
Discussion: I initially requested JH about the differences and similarities between evaluation as carried out within the DOD/army and the IC given his experiences in each environments. He noted that his background as a submariner was instructive within the improvement of DOD’s analytic capabilities and the historical past of Operations Research (OR), which constituted an effort to use scientific methods (and working towards scientists) to the issues of Anti-Submarine Warfare (ASW). He noted that the ASW group went from having little analytic capability to turning into one of the premiere users of OR methods after a devoted, and sustained organizational dedication to the issues posed in WWII and the Chilly Warfare.
In comparison with the IC, notably the CIA, there was little comparable story with respect to OR. JH attributed this to variations in analytic cultures and noted that the DOD (notably the analytic and managerial organizations that have been legacies of McNamara) was largely a numeric culture while the IC was predominately as literature one. Thus, investments particularly analytic tools and methods that aligned with quantitative and formal modeling, which frequently discovered fertile floor in the DOD, weren’t necessarily long-lived inside the IC. JH was fast to notice that this was essentially because IC analysts lacked the capacity to work with such approaches, and certainly, he famous that analysts typically moved between communities, but that the organizations themselves emphasized totally different questions and served totally different clients. However, JH believed that the IC may benefit from utilizing extra formal analytic methods when issues warranted it. Nevertheless, JH also noted that DOD had typically turn into overly reliant on formal strategies in a number of instances, notably giant fight simulations, and might typically be overconfident in the fashions and susceptible to being misled by modeling results if not sufficiently familiar with their inside workings and embedded assumptions.
I requested whether or not the variations between the DOD and IC affected the roles and obligations of managers themselves. JH felt that from an organizational perspective that managers had very comparable jobs regardless of the group, however that differences typically existed based mostly on their experiences. Thus, differences might be seen, but they have been the outcome of private experiences fairly than organizational calls for.
Because it was clear that an ideal imbalance existed relating to the use of formal models between the DOD and the IC, I needed to know if this utilized to the larger class of simulations, which also included wargames. JH felt that as exploratory instruments, video games served an analogous position in each communities, however that the DOD had several distinct benefits when dealing with army points. Specifically, as a result of army questions typically concerned the improvement and use of weapons methods, many of the questions raised by a recreation might subsequently be explored by way of workouts and concerted knowledge gathering efforts to validate modeling results towards real-world outcomes. Likewise, questions on parameters concerning operational or technical variables may be decided empirically. Thus, a tighter loop and suggestions mechanisms might help wargames on the DOD in comparison with comparable games in the IC. So, regardless that wargames performed comparable roles with respect to evaluation within the DOD and IC, the character of the video games, notably with respect to their fidelity or capability to be linked with operations or supported by formal fashions, was larger on the DOD aspect consequently of combination of culture and assets.
JH’s characterization of the linkages between analytic or exploratory wargames and fashions with operators prompted me ask about the general relationship between analysis and improvement, evaluation, and operations. JH again drew upon his navy expertise and mentioned how necessary shut relationships between the three really have been and that many of the main problems affecting the DOD at this time have been a end result of this relationship breaking down and turning into overly inflexible and tied to necessities processes that inhibit groups from working collectively. Specifically, he mentioned the improvement of the AEGIS missile defense system, where R&D and operators labored intently collectively in an evolutionary course of that explored how mixtures of know-how and operational use might push radar performance previous previously anticipated performance limitations, leading to a system that may have never been envisioned by the necessities process. JH famous that this course of was heavy vested in tight feedback loops, the collection of empirical knowledge on operational efficiency, exploration of the potential (versus the emphasis on the feasible that results from the requirements course of), and a minimized position for contractors with the majority of the design, engineering, and testing carried out in-house.
JH’s discussion of the AEGIS system denoted the significance of evolutionary approaches to drawback solving where prototypes are developed quickly, and then experimented with in operational contexts. He famous that such an strategy was increasingly scarce given the mixture of DOD paperwork and Congressional oversight nevertheless. In such instances, the IC and the categorized Particular Access Packages (SAPs) that operate with higher secrecy, and subsequently much less transparency, present larger alternatives for experimentation and danger taking in accordance with the evolutionary mannequin.
After discussing some of the features of technical improvement, I targeted on questions that have been more specific to the IC and the ODNI itself. My first question was about how the IC evaluates the quality of its analysis and analytic products. JH noted that it was a really troublesome drawback and something that was still in its infancy with respect to creating evaluative criteria past widespread production metrics. He emphasized the current improvement of analytic integrity standards, and a higher emphasis on evaluating prior analytic forecasts with actual outcomes systematically. JH felt that higher consideration needed to be given to strategies and analytic approaches, particularly focusing on performing cost-benefit analyses to be able to determine whether totally different methods are better than others with respect to generating useful insights for shoppers. I requested JH about the risks of such assessments, notably analytic integrity requirements, and he famous that there is all the time a danger that well-intentioned efforts to information producers might develop into check-lists which are employed with out consideration of context. JH additionally noted that the analytic integrity standards could possibly be seen as trying to cope with the challenges of discerning arguments based mostly on proof vs. these based mostly on intuition, and that the standards favored or promoted analysis grounded in proof over these based mostly on logical argumentation. From the perspective of modeling and simulation in the analytic process, resembling distinction could be problematic, provided that models are products of instinct/principle/simplifications, but generate knowledge upon which inferences are based mostly, and thus grounded in empirical observations of artificial methods.
Another query I requested JH associated to new fashions of intelligence help to policymakers, comparable to more and more contemplating blue capabilities and actions in assessments of overseas actors, the use of wargames or simulations with policymakers as gamers, and so forth. JH noted that there a shift has occurred in the policy world and that expectations are increasingly formed by social media applied sciences that place the consumer/shopper as an interactive participant in the delivery of info. He noted that providers of info are more and more confronted with very rich and fast suggestions from users that anticipate to have influence over the production course of—shaping its format and content material. In consequence, the prospects for utilizing simulations as an analytic software that permits for intelligence and policymakers to collectively explore complicated problems could also be rising in significance and effectiveness as a way of producing and delivering intelligence analysis (or perhaps intelligence “experiences”?).
The use of simulations as suppliers of experience that train policymakers and their staffs was also mentioned by Leon Fuerth in one other dialogue. Leon was involved with the time required to develop models, simulations, and games that have been suitable to the needs policymakers. When confronted with the similar query, JH acknowledged the concern, however was extra optimistic, noting that recreation and software program developers have been more and more using specialized instruments for creating merchandise that significantly reduce down improvement and testing time (certainly, many video games have now offered gamers with content material creation instruments for creating their own modifications and content material, and these tools require no programming expertise). Thus, JH believes that the infrastructure to help modeling and simulation is turning into more and more streamlined and much less complicated, and these improvements will scale back the quantity of time required to develop models and simulations for use in evaluation or with policymakers.
Nevertheless, JH also noticed that the abstract nature of fashions will all the time impose some limitations on the extent they can be anticipated to duplicate actuality. Consequently, video games and simulations might be less useful in disaster conditions when policymakers’ considerations are targeted on very specific programs of actions and options. As an alternative, as questions are more strategic and targeted on long-term considerations, fashions and simulations can play a extra influential position by serving to to outline the characteristics or properties of sound strategies, identifying vulnerabilities, or circumstances for opportunistic action. On this situation, JH used the phrase “striking while the iron’s cold” to spotlight the importance of interacting with policymakers when issues usually are not pressing or pressing and rational and important evaluation might be influential. Thus, on many strategic points, analysis have to be carried out prematurely and might even need to take a seat dormant until the interest in the question/drawback is raised.
JH concluded that many of the developments in tradecraft have been being made however that improvements have been largely confined to small groups and not inspired systematically. He repeated his perception that higher consideration to assessing tradecraft and evaluating methods was needed with a purpose to give managers and analysts a greater understanding of what works, what doesn’t, and how one can employ their assets in the manufacturing process.
My remaining questions have been about the similarities and variations between intelligence analysis and scholarship. JH famous that teachers are typically extra analytic than artificial – specializing in the isolation of specific variables or causes as a way to look at and explain occasions slightly than aggregating them to acquire a complete understanding. The result’s that much of scholarship, even from policy-oriented teachers, has been unhelpful to real-world coverage and analysts as a result of of their emphasis on learning the “lens” or concept and its applicability to many problems or instances, fairly than the particulars of specific issues. This once more, was reminiscent of John Lewis Gaddis’s argument about “particular generalization” during which argues that classes ought to be drawn from the research of particular person instances and examples, quite than envisioning a principle and then attempted to suit multiple instances to it.