AI in High-Stakes Legal Cases: Defensibility and Evaluation Metrics

High Stakes in Legal Cases

• ⚖️ Legal cases, particularly in big law firms, can involve hundreds of thousands or millions of documents, with millions or even billions of dollars on the line for plaintiffs and defendants.
• 💡 The high stakes necessitate a careful balance between speed and automation and the legal field's stringent standards for defensibility and due diligence.

Evaluating AI in Law

• 🎯 Standard evaluation metrics like recall and precision are often negotiated in legal contexts, sometimes even with opposing counsel or governmental bodies.
• 🔬 Rigorous evaluation processes are crucial, as data scientists must explain the implications of margin of error and other metrics to attorneys and judges.
• 🧩 Defensibility is ultimately determined by what a particular attorney is comfortable defending, considering proportionality and undue burden.

Challenges and Metrics

• ⚠️ When dealing with low prevalence relevance tags (searching for rare documents), sampling enough data for evaluation can become overly burdensome.
• 📊 The metric of "illusion" is used in e-discovery, where sampling predicted non-relevant documents and comparing them to human ground truth for relevant documents allows for an estimation of recall.
• 🚀 The proponent of "illusion" argues that if human review (linear review) is considered the gold standard, even with its potential inaccuracies, AI workflows should not be held to a higher standard.
• 💬 The defensibility of AI workflows is debated and depends on the specific case, the requesting party, and the burden on the producing party.