After the redesign shipped, I partnered with the research team to test whether it actually closed the trust and comprehension gaps we'd designed for — running a mixed-method study across 28 participants against real analytics questions.

Research approach

Tested two scenarios — specific data queries and broader multi-dataset synthesis — against four quality dimensions: input interpretation, language clarity, explainability, and output readiness.

Key findings

• 58% of users said they'd act on the AI's response — though most still cross-checked the dashboard first, a sign trust was forming but not yet earned

• Explainability, not accuracy, drove usage. Users said "getting intent right" mattered most, but behaviorally, seeing why an answer was correct — its data source, date range, filters — is what got them to act on it

• Intent interpretation was the master quality — when the AI misread what a user meant, explainability and output quality broke down with it

• Gaps surfaced: text-and-chart mismatches were the biggest trust-breaker even when data was accurate; the AI often answered beyond the asked scope; users wanted clarifying questions instead of silent assumptions; reports needed a consistent summary → breakdown → visual structure.

Next steps

These findings became the criteria for a new set of LLM-judged production evals: explainability, intent interpretation, language clarity, and output readiness: with an 80%+ accuracy threshold set as the bar for production-readiness