ThoughtTrace
Understanding User Thoughts in Real-World LLM Interactions
1Johns Hopkins University · 2MIT · 3Google Research
Overview
Conversational AI reaches billions of users, yet existing datasets capture only what people say, not what they think. ThoughtTrace is the first large-scale dataset that pairs real-world, multi-turn human–AI conversations with users' self-reported thoughts.
Thoughts come in two forms: reasons for sending a prompt and reactions to the assistant's response. They are distinct from messages, hard for frontier LLMs to infer, and useful for predicting user behavior and aligning models.
Findings
Thoughts are hard for LLMs to infer
GPT, Gemini, and Claude struggle to recover users' thoughts from context alone. On a 1–5 similarity scale, predicted thoughts score only 2.93 for reasons and 2.54 for reactions. Thoughts carry information that never appears in the messages.
Thoughts predict what users say next
Giving models the annotated thoughts raises next-message prediction similarity from 21.6 to 30.6 on average, a 41.7% relative gain.
Thoughts improve model alignment
Training on thought-guided rewrites beats message-guided rewrites on Arena-Hard by 4.5 points, and the base model by 25.6 points (style-controlled win rate).
My role
- Contributed to building the dataset.
- Developed the evaluation pipelines and ran experiments comparing model-predicted thoughts with human annotations.
Citation
@article{jin2026thoughttrace,
title = {ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions},
author = {Jin, Chuanyang and Li, Binze and Xie, Haopeng and Fang, Cathy Mengying and
Li, Tianjian and Longpre, Shayne and Gu, Hongxiang and Chen, Maximillian and
Shu, Tianmin},
journal = {arXiv preprint arXiv:2605.20087},
year = {2026}
}