OATH-Frames: Characterizing Online Attitudes towards Homelessness with LLM Assistants

Jaspreet Ranjit1, Brihi Joshi1, Rebecca Dorn1, Laura Petry2, Olga Koumoundouros2, Jayne Bottarini1, Peichen Liu2 Eric Rice2 Swabha Swayampdipta1


1        2 
Paper Code Hugging Face

Snapshot of OATH-Frames

We introduce a novel framework to understand, synthesize and characterize large-scale public attitudes towards societal issues through a collaboration between social work experts and LLMs. Specifically, we introduce a framing typology: OATH-Frames, (Online Attitudes Towards Homelessness): nine hierarchical frames capturing public attitudes towards homelessness as expressed on Twitter. We provide three kinds of annotations for posts from Twitter: expert-only, LLM-assisted expert and predicted annotations from a multilabel classification model. Our LLM-assisted expert annotation for socially sensitive tasks follows a new scalable framework which incorporates expert insights and chain-of-thought (CoT) explanations for designing better LLM prompts. Our 2.4M OATH-Frames-annotated posts enable a large-scale analysis across states and time periods, revealing changing trends in attitudes with key sociopolitical events. OATH-Frames surface harmful language towards people experiencing homelessness (PEH), which we show are often mislabeled by popular sentiment and toxicity classifiers, highlighting the value of our typology.
pitch
Domain experts applied grounded theory to surface nine Issue-Specific frames, their corresponding definitions and annotation guidelines. We annotated posts with OATH-Frames via Experts, via Experts + LLM (GPT-4), and via a Multlilabel Classifier (Flan-T5-Large). Our Experts + LLM annotations pipeline consists of prompt editing based on insights from domain experts and chains-of-thought, and validation of predicted frames.

BibTeX


    @misc{ranjit2024oathframescharacterizingonlineattitudes,
      title={OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants}, 
      author={Jaspreet Ranjit and Brihi Joshi and Rebecca Dorn and Laura Petry and Olga Koumoundouros and Jayne Bottarini and Peichen Liu and Eric Rice and Swabha Swayamdipta},
      year={2024},
      eprint={2406.14883},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.14883}, 
    }