OATH-Frames: Characterizing Online Attitudes towards Homelessness with LLM Assistants

1        2 

Snapshot of OATH-Frames

We introduce a novel framework to understand, synthesize and characterize large-scale public attitudes towards societal issues through a collaboration between social work experts and LLMs. Specifically, we introduce a framing typology: OATH-Frames, (Online Attitudes Towards Homelessness): nine hierarchical frames capturing public attitudes towards homelessness as expressed on Twitter. We provide three kinds of annotations for posts from Twitter: expert-only, LLM-assisted expert and predicted annotations from a multilabel classification model. Our LLM-assisted expert annotation for socially sensitive tasks follows a new scalable framework which incorporates expert insights and chain-of-thought (CoT) explanations for designing better LLM prompts. Our 2.4M OATH-Frames-annotated posts enable a large-scale analysis across states and time periods, revealing changing trends in attitudes with key sociopolitical events. OATH-Frames surface harmful language towards people experiencing homelessness (PEH), which we show are often mislabeled by popular sentiment and toxicity classifiers, highlighting the value of our typology.
Domain experts applied grounded theory to surface nine Issue-Specific frames, their corresponding definitions and annotation guidelines. We annotated posts with OATH-Frames via Experts, via Experts + LLM (GPT-4), and via a Multlilabel Classifier (Flan-T5-Large). Our Experts + LLM annotations pipeline consists of prompt editing based on insights from domain experts and chains-of-thought, and validation of predicted frames.