DILL Lab

The Data, Interpretability, Language and Learning, (DILL) lab, led by Swabha Swayamdipta, explores questions at the intersection of language models, NLP and machine learning.

Check out our latest publications and open positions.

Here are some questions we have worked on recently:

• What do we understand about the geometries of language models?

  Has language model generation reached a performance saturation or do language models still make systematic errors owing to their design? We studied the softmax bottleneck as one concrete limitation and how that affects language generation and justifies truncation sampling. Can we build better language generators? The softmax bottleneck also leads logits leaking information in closed LLM APIs.

• What are the limits of the generative capabilities of LLMs?

  How do language models handle specific distributions of language, such as ambiguous language, comparative language or the language of explanations? Can language models generate structured data?

• How reliable is comparative generative evaluation?

  What cannot be measured, cannot be improved. Can we reliably compare the generative performance of two different models, in either close-ended generation tasks such as summarization or in open-ended generation? What makes model A better than model B, or are our test sets somewhat misleading us?

• What does our data tell us about our models?

  What makes a data collection valuable for instruction tuning or finetuning large language models? Is all human feedback equally valuable under PPO or DPO? Our Dataset Cartography offers point estimates, and V-Information offers both point and aggregate estimates of data quality. How can we build similar estimates for generative models? Are all modalities and all data necessary in multimodal settings?

• How can our models help us understand our society?

  How far can language models go in helping us understand complex social phenomena such as homelessness? Is it possible to create collaborative setups between humans and generative models to this end? What role does conversational and social context play in this understanding? Can socio-technical solutions work well for all?