Side-by-side Comparison Amplifies Dialect Bias in Language Models

Kritee Kondapally1, Claire J. Smerdon1, Pooja C. Patel1, Ogheneyoma Akoni1, Jevon Torres1, Jaspreet Ranjit1, Matthew Finnlayson1, Swabha Swayampdipta1


1       
Paper Code Dataset
Warning: This paper includes examples of offensive stereotypes based on dialect. In this work, we quantify covert dialect bias in online discourse by evaluating how LMs associate stereotypical traits (derived from social psychology research on racial bias) with intent-equivalent tweets in Standard American English (SAE) and African-American Vernacular English (AAVE). While prior work shows that LMs associate more negative stereotypes with AAVE when evaluating tweets in isolation, we are surprised to find that this bias is significantly exacerbated when SAE / AAVE tweet pairs are compared side-by-side, a setting that more closely reflects high-impact decision making contexts in which models are used to rank candidates. The bias only worsens when dialect labels are explicitly specified. Encouragingly, we show that counterfactual fairness finetuning can mitigate covert dialect bias for some stereotypical traits, reducing average disparities when evaluating tweets in isolation, however, these improvements do not consistently hold across traits when evaluating SAE / AAVE tweets side-by-side. Our findings show that existing evaluation settings for covert dialect bias may underestimate its severity, and our contrastive evaluation exposes amplified bias in certain settings.
pitch
: Evaluation (top) and mitigation (bottom) of covert dialect bias in language models. Top: We evaluate covert dialect bias by prompting language models to rate intent-equivalent SAE and AAVE tweet pairs on 12 traits (Likert 1–5). Using matched-guise probing, models are evaluated under two conditions: absolute prompting, where each tweet is rated independently, and contrastive prompting, where SAE and AAVE tweets are rated side-by-side. We find that bias is significantly exacerbated in the contrastive setting, and in some cases, worsens when explicit dialect labels are present. Bottom: We apply counterfactual fairness fine-tuning, training the model to assign identical trait scores to SAE/AAVE tweet pairs. We find this is effective in reducing effect sizes (e.g., bias towards AAVE) for a few traits, specifically: Unsophistication, Stupidity, Incoherence, Determination, and Sophistication. in the covert setting, but remain pronounced in the overt setting.

BibTeX


    @inproceedings{kondapally2026dialectbias,
      title={Side-by-side Comparison Amplifies Dialect Bias in Language}, 
      author={Kritee Kondapally and Claire J. Smerdon and Pooja C. Patel and Ogheneyoma Akoni and Jevon Torres and Jaspreet Ranjit and Matthew Finnlayson and Swabha Swayamdipta},
      booktitle={Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency},
      series={FAccT '26},
      year={2026},
      publisher={Association for Computing Machinery},
      address={New York, NY, USA},
      doi={10.1145/3805689.3812217}
    }