AI Monitoring: Beyond Evals and LM Judges with Raindrop and Oleve

The Limitations of Offline Evals and LM Judges

• 💡 Defining "good" and "bad" outputs is inherently difficult, making traditional evals and LM judges prone to misuse and false confidence.
• ⚠️ LM judges can be expensive and hard to set up accurately, often failing to capture novel or nuanced issues.
• 🚀 While useful for local iteration and regression testing, offline evals don't reflect real-world production challenges.

Production Monitoring: Signals and Intents

• 🎯 Implicit signals like user frustration, task failures, or unexpected behavior in the AI's responses indicate potential issues.
• ✅ Explicit signals, such as thumbs down, regeneration requests, or copying/sharing of content, provide direct user feedback on AI performance.
• 🔍 Understanding user intents by combining these signals (e.g., user frustration during math homework) is crucial for diagnosing problems.

Exploring and Refining AI Issues

• 📊 Metadata tagging (e.g., by mode, model, or user plan) and keyword analysis help categorize and explore issues in production data.
• 💬 Direct user feedback channels (like Discord) are vital for capturing specific, nuanced problems that automated systems might miss.
• 🛠️ Tools like Raindrop's deep search enable users to define precise issues, track their occurrence over time, and refine monitoring criteria.

The Trellis Framework for Reliable AI

• 🧩 Trellis is a framework for creating reliable AI experiences by organizing AI outputs into controllable buckets.
• 🚀 It involves discretization of outputs, prioritization based on impact and strategic relevance, and recursive refinement to continuously improve AI performance.
• 📈 By converting user intents into semi-deterministic workflows, teams can focus engineering efforts on what truly matters and ensure changes are self-contained and attributable.

Practical Application and Iteration

• 📈 Prioritization involves considering volume, negative sentiment, and estimated achievable delta to focus on the most impactful improvements.
• 💡 Case studies like Oleve's "Unstuck" app demonstrate how Raindrop and the Trellis framework helped identify and fix issues, leading to user satisfaction and product growth.
• 🔄 Building reliable AI requires a continuous loop of monitoring, analysis, and refinement, ensuring AI magic is engineered, repeatable, and testable, not accidental.