Edit: Although not limited to the US, some of the most glaring applications of the prosecutor’s fallacy have occurred there. It’s a problem that continues, with the US Supreme Court having just had to decide another case (McDaniel v Brown [2010] USSC No 08-559) where an expert on behalf of the State used the traditional flawed reasoning [at 9]:
And when the prosecutor asked Romero, in a classic example of erroneously equat-ing source probability with random match probability, whether “it [would] be fair to say . . . that the chances that the DNA found in the panties—the semen in the panties—and the blood sample, the likelihood that it is not TroyBrown would be .000033,” id., at 460, Romero ultimately agreed that it was “not inaccurate” to state it that way, id., at 461–462.
The Supreme Court held that other, more persuasive evidence supported the juries’ finding, despite the misleading evidence that had been given.
Today Chief Commissioner Simon Overland announced that his police won’t be using DNA evidence in any cases run between now and the end of the year.
Overland denied that it was a result of concerns of over the reliability of the evidence, but there’s no doubt that the case of Farah Jama gives cause for concern about the validity of forensic processes. Mr Jama served fifteen months of a six year sentence after he was convicted by a County Court jury substantially relying on contaminated DNA evidence, before his release was ordered by the Court of Appeal this week.
Contamination of forensic evidence is not new. During the investigation into the disappearance of Jaidyn Leskie, a DNA sample from an unrelated investigation was transferred to the boy’s clothing. At that time, various promises were made about taking measures to restore integrity to the system.
Leaving aside issues of provenance, even where the science has been carried out flawlessly evidence can be difficult to follow. It almost always relies on estimates of probability which are outside the experience of the general population, requiring jurors to depend on the accuracy of information provided to them by experts.
Cross on Evidence [at 9090-9095] has a useful discussion on the difference between probability when it’s used in a legal context as opposed to a statistical one. The author points out that while statistical probabilities are endorsed as a way of predicting outcomes in things like economics and insurance, the courts often find discussion of mathematical probabilities to be of little assistance (in fact, a distraction) to deciding the case before them. Forensic sciences such as blood analysis and DNA, where expert evidence is commonly received and accepted, are an exception to this general rule.
The science involved in locating and testing DNA is the (relatively) easy part to understand. The hard part is putting the information to any practical use. Say, for example, a sample is located at a crime scene and matched to a test sample provided by a suspect. The result comes back a perfect match. The boffins say that the likelihood that a member of the general population would match the sample is 1 in 2,000,000. But what does that mean?
It might be tempting to say that there is only a 1 in 2 million (or 0.00005%) chance that the suspect is innocent. This is an example of what is known as the prosecutor’s fallacy. It was a term developed (though not coined) by Jonathon Koehler in his article One in millions, billions and trillions. The fallacy incorrectly assumes that because the odds of an innocent person being matched to the sample are tiny, the possibility of their innocence is therefore equally remote.
A variation on the theme is where the improbability of a given event is used to prove an unrelated fact. Say you are involved in a plane crash and you of the 500 passengers are the only person to survive. Despite being innocent, you are accused of causing the plane to crash using unknown means. The only piece of evidence against you is the tremendous odds against one person living where all others perished. The odds against your good fortune are not identical to the odds against your innocence.
In R v Doheny and Adams, Phillips LJ of the English Court of Appeal said [at
372–373],
The Prosecutor’s Fallacy
It is easy, if one eschews rigorous analysis, to draw the following
conclusion:1. Only one person in a million will have a DNA profile which matches that of the crime stain.
2. The defendant has a DNA profile which matches the crime stain.
3. Ergo there is a million to one probability that the defendant left the crime stain and is guilty of the crime.
Such reasoning has been commended to juries in a number of cases by prosecuting counsel, by judges and sometimes by expert witnesses. It is fallacious and it has earned the title of ‘The Prosecutor’s Fallacy’ …
Taking our example, the prosecutor’s fallacy can be simply demonstrated. If one person in a million has a DNA profile which matches that obtained from the crime stain, then the suspect will be 1 of perhaps 26 men in the United Kingdom who share that characteristic. If no fact is known about the Defendant, other than that he was in the United Kingdom at the time of the crime the DNA evidence tells us no more than that there is a statistical probability that he was the criminal of 1 in 26.
The significance of the DNA evidence will depend critically upon what else is known about the suspect. If he has a convincing alibi at the other end of England at the time of the crime, it will appear highly improbable that he can have been responsible for the crime, despite his matching DNA profile. If, however, he was near the scene of the crime when it was committed, or has been identified as a suspect because of other evidence which suggests that he may have been responsible for the crime, the DNA evidence becomes very significant. The possibility that two of the only 26 men in the United Kingdom with the matching DNA should have been in
the vicinity of the crime will seem almost incredible and a comparatively slight nexus between the defendant and the crime, independent of the DNA, is likely to suffice to present an overall picture to the jury that satisfies them of the defendant’s guilt.The reality is that, provided there is no reason to doubt either the matching data or the statistical conclusion based upon it, the random occurrence ratio reduced from the DNA evidence, when combined with sufficient additional evidence to give it significance, is highly probative. As the art of analysis progresses, it is likely to become more so, and the stage may be reached when a match will be so comprehensive that it will be possible to construct a DNA profile that is unique and which proves the guilt of the defendant without any other evidence. So far as we are aware that stage has not yet been reached.
It’s stating the obvious to say that we’re still not there yet.