As a couple of news outlets have reported recently, a US strike in Yemen hit what turned out to be a quarry, causing civilian casualties. As concerning as these events are on their own, what is different this time is the (hopefully) coincidental fact that the coordinates struck were exactly the same as those tweeted beforehand by an online investigator by the name of VleckieHond. Vleckie subsequently posted in a now deleted message that she had made a mistake in tweeting the coordinates but that she was sure that the Pentagon weren’t basing strikes on coordinates shared on social media alone. This is somewhat beside the point.
The reality is that whether or not Vleckie’s tweet was read by the Pentagon, her clear intention in identifying and sharing the coordinates was to have an intelligence effect. This means that the fact that she was wrong was an intelligence failure, one which some basic analytical rigour would have avoided.
This isn’t about one case though and Vleckie’s mistake isn’t unique and certainly doesn’t justify the abuse she has received since. Amy Zegart, a long-term Intelligence Studies scholar, discusses the mixed impact that early amateur nuclear sleuths have had in misidentifying testing and enrichment sites, and those of us who have watched the mutation of the Twitter-based online investigative community since buying blue ticks became a thing, have surely seen our share of false positives.
The fact is that while the online investigative community by and large means well and many have professional and impactful outputs, there are unfortunately plenty out there who have learned their skillset from fora that prioritise hard skills like geolocation and chronolocation above softer skills that are key to responsible intelligence practice. Taking Vleckie’s case as an example, we’ll go through a couple of those practices and explain how they could have helped. We’ll start with source/information validation, move on to hypothesis testing and finish with some thoughts on articulating analysis.
Source and Information Validation
Vleckie’s mea culpa post says that she had the site marked as a ‘possible’ but she was reportedly swayed to post it when another user, Galal Alsalahi claimed to have identified a missile launch pad at the site. When considering a source, professional intelligencers are trained to think about track record, access, and bias.
A scan of Alsalahi’s account doesn’t give much to go on for track record because most of his posts are not investigative outputs. His access is questionable because he reports himself as living in Houston and his bias, to the extent it’s possible to assess, is an anti-Houthi one. Now, it maybe that Vleckie was in contact with Alsalahi and knows more about these factors than we do, but at face value it would not be possible to base any level of confidence in his report based on him as a source.
That leaves us with the information itself. The tweet is just a screen recording from a phone scrolling in and out on the site with pins showing little more than flat open spaces. The text just reads ‘missile launch pad’ and the coordinates. If Alsalahi has done more work to identify the pinned features as launch pads, he doesn’t share it. Again, it is possible that the two tweeters shared information privately, but based on what you can see here, it would not be possible to derive any confidence from the raw information.
So Vleckie shouldn’t have taken Alsalahi’s information as corroborating her suspicions about the site because it really isn’t possible (absent private information) to say either that he knows what he’s talking about or that his ‘evidence’ is any good. That she did may have been a manifestation of ‘confirmation bias’ (the brain’s tendency to seek information that confirms what it already believes). The way that intelligencers guard against that bias is structured analysis, commonly using a technique called Analysis of Competing Hypotheses (ACH).
Analysis of Competing Hypotheses
The purpose of ACH is to avoid the trap that Richards J. Heuer calls ‘satisficing’ – choosing a favoured explanation that is good enough and then running with it without paying due attention to other explanations. It does this by forcing the analyst to come up with a few explanations which are then pitted against each other using the information available as a metric. The winner isn’t the one that the analyst likes best and it isn’t the one that is most consistent with the available information, it’s the one that is least inconsistent. That information is gathered into a matrix called a Heuer table. There are a couple of examples below.
To clarify, let’s think about murder. The hypotheses being considered might be that a number of different suspects did the deed and the information being used might be the standard: motive, opportunity and means. Suspect A appears to have a much stronger motive and much stronger means than the other two suspects. This means that the hypothesis that he did it is the most consistent with the available information. The eagle eyed will however notice that an alibi means that he had no opportunity and couldn’t have done it. This means that while he is more consistent with the information, the other two are less inconsistent and would therefore be more likely.
You’ll note however that the matrix includes a column that speaks to the credibility of the information. This is particularly important because, as our example demonstrates, often the question of which hypothesis wins hinges on one or two pieces of information. In this case, the alibi may well be bogus.
So how does this apply to Vleckie’s case? For ease, let’s consider just two hypotheses. We don’t have Vleckie’s workings, so we’ll consider this as just a simplified hypothetical. Clearly, when laid out in a Heuer table we can see that what we know about the site is broadly consistent with both a concealed subterranean base and a mine site. This is probably why Vleckie had it down as only ‘possible’ in her internal notes. The thing that tips it over the edge in our example, is the Alsalahi tweet. As we have discussed though, that tweet gave us nothing on which to base our confidence and so should not tip the scale. Moreover, if we did not have that tweet, we may well have entered ‘lack of identifiable military equipment or activity’ into the relevant information, which only would have increased the inconsistency of the hidden base hypothesis.
It is reasonable to say therefore, that had Vleckie laid out her information and hypotheses in this way, she would have seen just how much rested on this one piece of hard to credit information and might have thought twice about tweeting it.
To give her the benefit of the doubt though, maybe she did know it wasn’t a strong case, maybe her audience simply misread her meaning. This is a problem that intelligencers solve with clear and consistent analytical language.
Articulating Analysis
Both Alsalahi and Vleckie made the mistake of not expressing either the probability judgement underlining their assessment or their confidence in that judgement. Vleckie has stated that she had the site down as ‘possible’ but if the history of intelligence failures tells us one thing, it is that readers’ understanding of what terms like ‘possible’ means vary wildly depending on a variety of factors, not least of which is what they want to believe. While the Pentagon denies having based their strike on the tweet, it is entirely possible that an analyst there could have seen Vleckie’s work and fallen victim to confirmation bias, just as Vleckie reportedly did with Alsalahi’s.
We can’t control how other people will use information that we put out publicly, but we can control the number of ways the information we put out can be interpreted. We do this by using clear and consistent probability language. The UK Intelligence Community uses a system call the ‘Probability Yardstick’ to achieve this aim. The yardstick sets percentage ranges for set probability terms. Those percentages aren’t statistical, they’re just ballpark ranges that ensure the reader (and the authors) know roughly what each term means. An accompanying confidence statement (low, medium, high, etc) expresses how confident the analyst is (based on confidence in the underlying information) in that assessment.
Online sleuths in the investigative community need not adopt this exact standard, but if an investigator is going to routinely tweet assessments, they could do far worse than defining the language they intend to use and pinning the tweet.
As I said, VleckieHond is not the first amateur intelligencer to publicly post an erroneous assertion and she almost certainly won’t be the last. It also certainly isn’t the case that professional intelligencers don’t make these mistakes… for this strike to have happened, several professional intelligencers obviously did. The point of the above is not therefore to pick apart everything that Vleckie did wrong, it is simply to use this tragedy to make a case for the popularisation of standards of practice in source/information validation, structured analysis, and the articulation of analysis, that have been generated by the intelligence profession over a couple of centuries of trial and error.[1]
The online investigative community can have and has had incredible impacts on the world. If we want to keep our impacts on the ‘good’ side of the ledger, we have to start holding ourselves to these standards.
References
[1] For more information see: How Spies Think by David Omand and Psychology of Intelligence Analysis by Richards J Heuer.
