What Risk Assessment Validation Tells Us about Pretrial Failures: They’re Lower than We Think

Suppose I told you that we could categorize defendants into six categories for risk of failure to appear (FTA) in court as required, with 1 being the lowest risk category and 6 being the highest. What is your guess as to the percentage of defendants who appear in court as required at risk level 1? At risk level 6? When I ask this question of North Carolina stakeholders, most guess that the percentage of defendants who appear in court at risk level 1 is about 50% and that the percentage who appear at risk level 6 is about 20%. They are wrong. Risk assessment validation done in North Carolina shows that 87.4% of risk level 1 defendants appear in court as required and that 61.2% of risk level 6 defendants do so. In fact, that validation shows that at all risk levels, a majority of defendants appear in court as required.

Suppose I also told you that we could categorize defendants into six risk categories for risk of new criminal activity during the pretrial period, with 1 being the lowest risk category and 6 being the highest. What is your guess as to the percentage of defendants who engage in new criminal activity risk level 1? At the other risk levels? Stakeholders guesses on these questions vary, but they always are surprised to learn the facts. Risk assessment validation done in North Carolina shows that 90.9% of defendants categorized at risk level 1 have no new criminal activity during the pretrial period. In fact, even at risk levels 2 through 5, the majority of defendants have no new criminal activity during the pretrial period. Only at the very highest risk level—risk level 6—do we see a minority of defendants (46.4%) being successful as to this pretrial metric.

There is an ongoing national debate about whether or not it is appropriate to use empirical risk assessment tools in pretrial decision-making. As discussed in this primer, pretrial empirical risk assessment tools use factors (things like the defendant’s criminal history and the nature of the current charge) to estimate the likelihood that a defendant will appear in court as required and pick up no new arrests during the pretrial period. One empirical risk assessment tool used in dozens of jurisdictions nationwide is the Public Safety Assessment (PSA). That tool uses nine factors from a defendant’s history to produce two risk scores: one representing the likelihood of a new crime being committed and another representing the likelihood of a FTA. The PSA also indicates if the defendant has an elevated risk of a new violent criminal activity. Mecklenburg County, NC is one jurisdiction that uses the PSA. Mecklenburg uses PSA risk scores in connection with a county-developed decision-making framework (DMF) that provides a pretrial release recommendation. Under the DMF, release conditions become more restrictive as PSA risk levels go up. Importantly, Mecklenburg has validated the PSA for local populations. When a jurisdiction adopts an empirical risk assessment tool it does a validation to ensure that the tool is sufficiently predictive. Among other things, the validation looks at pretrial failures for defendants in various risk categories. For a tool that’s functioning well, you’d expect to see higher failures at higher risk levels. One thing frequently lost in the national debate about empirical risk assessment tools is what validation reports of the tools teach us about pretrial failures. Specifically, that stakeholders typically overestimate the rate of pretrial failures. Consider the PSA validation done in Mecklenburg County. See Public Safety Assessment (PSA) Validation in Mecklenburg County, NC (on file with author) [hereinafter Mecklenburg PSA Validation]. As shown in Table 1, below, that validation shows that the majority of defendants appear in court as required at all risk levels.

Table 1. Mecklenburg County: Percentage of Defendants Who Appear in Court, By Risk Level

Source: Mecklenburg PSA Validation at p.2.

Table 2 shows the percentage of defendants who have no new criminal activity during the pretrial period, again by risk level.

Table 2. Mecklenburg County: Percentage of Defendants Who Have No New Criminal Activity, By Risk Level

Source: Mecklenburg PSA Validation at p.5.

The results from the Mecklenburg validation are not an anomaly. A validation in Kentucky—that whole state uses the PSA—shows even higher pretrial success rates, with the majority of all defendants at each risk level having no failures either with respect to court appearance or new criminal activity. See Matthew DeMichele et al., The Public Safety Assessment: A Re-Validation and Assessment of Predictive Utility and Differential Prediction by Race and Gender in Kentucky 24, 28 (April 30, 2018) (Working Paper), available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3168452.

I’m not suggesting that the percentages of defendants who succeed pretrial in Mecklenburg holds true throughout the state. In fact there may be good reasons why other counties could see higher or lower success rates. As far as I know, however, Mecklenburg is the only county that has done such a validation; thus, it’s the only North Carolina data we have on the issue. And I think it’s important. Why? First, regardless of how you feel about empirical risk assessment tools and their ability to calculate risk levels, the validation studies produce pretrial failure rates for all released defendants. In the Kentucky study (sample size 164,597), the base FTA rate for all released defendants was 14.8%, the base new criminal activity rate was 10.6% and the base new violent criminal activity rate was 1.1%. Id. at 21. In the Mecklenburg validation (sample size 12,082) those numbers are 21.5%, 34.8%, and 9.8% respectively. See Mecklenburg PSA Validation at 2, 5, 8. These data show that the majority of those released pretrial had no pretrial failures. Second, when people dramatically overestimate pretrial failures, they may view the notion of pretrial reform through a warped lens. For example, if you incorrectly believe that the majority of low risk people released pretrial FTA and commit new crimes (facts the Mecklenburg and Kentucky validations disprove), you may be unwilling to support a system change that encourages greater release of such individuals. On the other hand, if you understand that the vast majority of the lowest risk defendants commit no new crimes pretrial and appear in court as required (90.9% and 87.4% respectively per the Mecklenburg validation), you’ll likely be willing to work on reforms to get those defendants out of pretrial incarceration, especially given the high cost of those pretrial detentions in terms of public safety and taxpayer resources (for detail on that last point, see my blog post here). And finally, knowing pretrial success rates can help jurisdictions figure out how to most effectively deploy limited pretrial support and supervision services. For example, if decision makers know that 90.9% of risk level 1 defendants commit no new criminal activity while on release, they might decide not to expend pretrial supervision resources on that population but rather to deploy resources to individuals who present a higher pretrial risk (a decision that also would be supported by the research on pretrial supervision effectiveness).