In an era where legal professionals increasingly rely on Generative AI (GenAI) systems, understanding these tools' limitations is crucial. A recent paper by scientists Dylan Restrepo, Nicholas Restrepo, Frank Huo, and Neil Johnson sheds light on the phenomenon of "AI hallucinations," where AI systems generate false or misleading information. This issue is particularly concerning in the legal field, where accuracy is paramount.

The paper, titled "When AI Output Trips to Bad but Nobody Notices: Legal Implications of AI’s Mistakes," explores the engineering risks associated with AI. The authors argue that AI hallucinations are not random but foreseeable and occur when the AI is pushed beyond the boundaries of its training data. This typically happens during complex legal inquiries with little precedent, exactly when lawyers need the most reliable information.

AI systems like ChatGPT function by predicting the most likely next word or phrase in a sequence, without a genuine understanding of legal veracity. They perform well with routine questions about established legal principles or undisputed facts but falter with novel or complex questions where data is lacking. The AI, attempting to respond helpfully, may fabricate information.

This issue is compounded by the way legal professionals use AI. Initially, reliable answers on well-trodden topics build trust in the AI. However, as the user progresses to more complex queries, the risk of encountering false information increases. If a lawyer relies on these outputs without thorough verification, they risk basing legal arguments on incorrect data, potentially leading to serious professional consequences.

The findings suggest a sliding scale of verification for AI outputs: less scrutiny for well-understood topics and heightened vigilance for novel issues. Recognizing when and why AI errors occur allows for safer, more effective use.

For instance, while an AI might accurately direct you to the correct subway stop for a well-documented route, asking for directions involving less common variables could lead you astray. Similarly, while AI can provide general information on a statute of limitations, delving into nuanced applications of that law could trigger misleading responses.

The paper highlights the importance of adapting the legal standard of technological competence. Lawyers must not only know how to use AI tools but also understand their limitations and potential for error. This understanding is crucial in preventing reliance on AI-generated "hallucinations" and ensuring legal arguments and decisions are based on accurate and reliable information.

In conclusion, as GenAI continues to integrate into legal practices, professionals must remain vigilant. Knowing when to trust AI and when to double-check its work is becoming a critical skill, akin to knowing which subway stop will actually get you to your desired destination—and which might lead you astray.