LLM Evaluation Research Study
I interviewed project managers and software engineers to understand how teams currently evaluate their large language models (LLMs) and which features they need in an evaluation tool. The findings helped an LLM evaluation startup decide which features to build into its product.
Role:
UX Researcher
Executive Summary Writer
Competitive Teardown Writer
Team
Team of 3
Findings
01
Timeline
6 Weeks
Methods
Competitive Teardown
Research Areas
In-store and online shopping habits, including how users engage with their phones while browsing in-store
Development Process: How do TPMs and SWEs determine the minimum requirements for an AI agent to launch?
LLM Evaluation + Iteration: How do software teams evaluate AI agents? What are unmet needs and untapped opportunities in this evaluation / iteration process?
Evaluation of conversational AI, benchmarked against Target's implementation
02
Target users are TPMs, SWEs, and PMs: TPMs and SWEs drive AI feature implementation and testing, and PMs deploying AI agents in as consultants.
AI evaluation is largely manual, leaving edge cases unaddressed and making quantitative metrics difficult for non-TPMs to define.
03
No standardized AI launch criteria: Most features are manually tested and iterated postdeployment. No standardized AI launch criteria: Most features are manually tested and iterated postdeployment.
04
AI Evaluation tools need to be time saving, reduce token usage and account for edge cases.