Supporting the Design of Intervention Opportunities for Suicide Prevention with Language Model Assistants

Jaspreet Ranjit1, Hyundong J. Cho1, Claire J. Smerdon1, Yoonsoo Nam1, Myles Phung1, Jonathan May1, John R. Blosnich2 Swabha Swayampdipta1


1        2 
Paper (Coming Soon!) Code
The National Violent Death Reporting System (NVDRS) documents information about suicides in the United States, including free text narratives (e.g., circumstances surrounding a suicide). In a demanding public health data pipeline, annotators manually extract structured information from death investigation records following extensive guidelines developed painstakingly by experts. In this work, we facilitate data-driven insights from the NVDRS data to support the development of novel suicide interventions by investigating the value of language models (LMs) as efficient assistants to these (a) data annotators and (b) experts. We find that LM predictions match existing data annotations about 85\% of the time across 50 NVDRS variables. In the cases where the LM disagrees with existing annotations, expert review reveals that LM assistants can surface annotation discrepancies 38\% of the time. Finally, we introduce a human-in-the-loop algorithm to assist experts in efficiently building and refining guidelines for annotating new variables by allowing them to focus only on providing feedback for incorrect LM predictions. We apply our algorithm to a real-world case study for a new variable that characterizes victim interactions with lawyers and demonstrate that it achieves comparable annotation quality with a laborious manual approach. Our findings provide evidence that LMs can serve as effective assistants to public health researchers who handle sensitive data in high-stakes scenarios.
pitch
LM assistants can reduce the emotional burden of annotating NVDRS narratives when a codebook is available (top) and help experts efficiently develop new codebooks for novel variables (bottom). When a codebook is available for a variable, we incorporate it in the LM instruction for Chain-of-Thought predictions. For a new variable of interest without an existing codebook, we propose an efficient, LM-assisted expert codebook development algorithm. Here experts focus on providing feedback for incorrect LM predictions, reducing manual codebook development time from weeks to hours (bottom right). For the annotation of a novel variable: victim interactions with legal professionals, we find that our algorithm results in similar annotation quality to a fully manual approach.

BibTeX


    @misc{ranjit2025placeholder,
      title={Designing and Validating Intervention Opportunities for Suicide Prevention with Language Model Assistants}, 
      author={Jaspreet Ranjit and Hyundong J. Cho and Claire J. Smerdon and Yoonsoo Nam and Myles Phung and Jonathan May and John R. Blosnich and Swabha Swayamdipta},
      year={2025},
      eprint={2406.14883},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={}, 
    }