412: Fault Lines in Forensic Proficiency Testing

Scurich N et al., PNAS - A concise breakdown of a PNAS perspective that analyzes forensic proficiency testing practices using the 2023 CTS firearms test as a case study. The authors identify test design and administration flaws—easy items, consensus scoring, handling of inconclusives, nonblind verification, shot‑to‑shot variability, and contextual bias—that undermine claims about examiner accuracy and the utility of reported error rates for courts and laboratories. Key terms: forensic proficiency testing, firearm identification, false positive rate, test design, cognitive bias.

Study Highlights:
The authors analyze CTS Test 23‑5262 and report a false‑positive rate of about 20% for bullet comparisons, highlighting broader concerns. They identify structural issues in proficiency testing including ceiling effects from easy items, inconsistent scoring of inconclusives, reliance on consensus rather than ground truth, and nonblind verification. The paper argues these features confound examiner performance with test properties, limiting the tests’ ability to estimate operational error rates and inform court decisions. The authors recommend better test design, blind verification, confidence ratings, and use of objective metrics to improve validation and lab practice.

Conclusion:
Current forensic proficiency testing practices can conceal fundamental weaknesses in examiner performance and test design; systemic, evidence‑based reform is needed to provide courts and laboratories with meaningful estimates of reliability and to reduce risks of wrongful outcomes.

Music:
Enjoy the music based on this article at the end of the episode.

Article title:
Assessing the foundations of forensic identification evidence: A critical examination of proficiency test design and results

First author:
Scurich N

Journal:
PNAS

DOI:
10.1073/pnas.2528192123

Reference:
Scurich N, Albright TD. Assessing the foundations of forensic identification evidence: A critical examination of proficiency test design and results. PNAS. 2026; Vol.123:e2528192123. doi:10.1073/pnas.2528192123

License:
This episode is based on an open-access article published under the Creative Commons Attribution 4.0 International License (CC BY 4.0) – https://creativecommons.org/licenses/by/4.0/

Support:
Base by Base is independent and ad-free — no sponsors, no paywall. If an episode was worth your time, chip in and keep the papers audited and the original songs coming:
❤️ Support monthly: https://buy.stripe.com/cNifZhclVebvagk2JDgEg01
☕ One-time donation: https://donate.stripe.com/7sY4gz71B2sN3RWac5gEg00
More at basebybase.com

On PaperCast Base by Base you'll discover the latest in genomics, functional genomics, structural genomics, and proteomics.

Episode link: https://basebybase.com/episodes/fault-lines-forensic-proficiency-testing

QC:
This episode was checked against the original article PDF and publication metadata for the episode release published on 2026-07-12.

QC Scope:
- article metadata and core scientific claims from the narration
- excludes analogies, intro/outro, and music
- transcript coverage: Audited the transcript sections describing CTS proficiency testing (Test 23-5262), the reported false-positive rates, consensus scoring and inconclusives, out-of-class eliminations, shot-to-shot variability and reproducibility, contextual bias and NIBIN leads, nonblind verification, signal detection theory as reform, a
- transcript topics: CTS proficiency testing overview and purpose; CTS Test 23-5262 design and ground truth; False-positive rate findings (around 20%; 19.2% excluding item 4; 22% at least one false positive); Consensus scoring and handling of inconclusives; Out-of-class eliminations (Item 4) and associated errors; Class characteristics and reproducibility concerns across tes...