In the rapidly evolving field of legal technology, the measure of effectiveness often boils down to sheer scale: larger AI models, extensive datasets, and a broad array of capabilities. During demonstrations, the focus is typically on the volume of tasks an AI system can handle—how many questions it can answer, the variety of issues it can identify, and the speed of its responses.

However, this approach misses a crucial point. The main issue with many legal AI tools isn't a lack of power; it's a lack of depth in realistic scenarios. Simply adding more AI capabilities does not remedy the fundamental problem of inadequate context.

This insight emerged from empirical classroom pilots conducted by the Product Law Hub, using an AI-based legal coach named Frankie. These pilots, designed to assess user engagement with AI in learning judgment-based legal skills, provided both quantitative and qualitative data. The findings were telling: scenarios that were fewer in number but richer in context led to deeper engagement, enhanced reasoning, and greater trust among users.

Demonstrations may deceive with their emphasis on volume, creating an illusion of competence inferred from the AI's quick and broad responses. Yet, in educational settings, this illusion quickly dissipates. Students faced with numerous but superficial prompts quickly disengage, with sessions becoming shorter and follow-up questions less frequent. In contrast, scenarios that offer richer, more ambiguous contexts keep students engaged longer, encouraging them to challenge assumptions and dive deeper into the material.

Ambiguity in scenarios, which might include stakeholder disagreements, incomplete information, or conflicting incentives, turns out to be particularly effective. These situations require users to exercise judgment rather than simply identify applicable rules, leading to higher completion rates and more sustained engagement.

Interestingly, the pilots also revealed that users react more negatively to repetition than to difficulty. Scenarios that recycled structures or language eroded trust quickly, as they seemed to merely pattern-match rather than truly reason. However, scenarios that presented complexity and uncertainty were met with greater tolerance and engagement, provided they felt authentic and took the problems seriously.

These findings suggest a shift in how legal tech buyers should evaluate AI tools. Instead of asking how many use cases a tool supports, a more pertinent question would be how well the tool handles complex, realistic cases. Depth and fidelity in scenario design appear to be more crucial for long-term utility than a wide array of superficial features.

Moreover, the design of scenarios not only affects learning outcomes but also influences the overall trust in and credibility of the AI system. When scenarios feel generic or overly simplified, users disengage; when they are realistic and grounded, users attribute more intelligence to the system.

The implications of these findings extend beyond educational settings into professional environments, where the quality of scenarios can determine whether lawyers view AI as a serious tool or merely a novelty. The data from these classroom pilots underscore a critical message for the future of legal AI: more isn't always better. What counts is the ability to effectively simulate real-world complexity and ambiguity, challenging users to develop genuine judgment and expertise.