RAG LLMs in Regulated Industries: Risks and Safeguards with Sebastian Gehrmann

Limitations of Generic LLMs in Finance

• 🎯 Foundation models are not typically trained on finance-specific corporate knowledge, leading to limitations in both helpfulness and harmlessness.
• ⚠️ Out-of-the-box safeguards like Llama Guard or Shield Llama are designed for general populations and productivity tasks, not the specific risks and obligations of financial services.
• 💡 The same limitations apply to other highly domain-specific, regulation-heavy sectors such as healthcare and law.

Evaluating RAG Systems in Context

• 🔑 A core recommendation is to evaluate any AI system within the specific context it will be deployed.
• 🔬 For healthcare, evaluate with healthcare subject matter experts; for financial services, use financial services experts.
• 📊 Don't rely solely on benchmark scores from LLM providers; invest heavily in domain-specific evaluation.

Mitigating Risks in Regulated Domains

• 🛠️ Utilize existing frameworks like the NIST AI Risk Management Framework as starting points and adapt them to your specific domain.
• 🤝 Industry collaborations like ML Commons offer general-purpose taxonomies that can be customized.
• 🔍 Organize red teaming events where users actively try to break the system to identify and quantify risks.
• ✅ Measuring how often the system fails or provides incorrect information is crucial for understanding the risk surface.

Building Trustworthy AI Systems

• 🚀 By evaluating in context and actively measuring risks, organizations can build systems that are more trustworthy, reliable, and robust.
• 📈 This leads to better user adoption, as users are less likely to abandon the system after encountering bad answers.