Stax: Data-Driven AI Evaluation for Product Builders

Replacing Gut Feelings with Data-Driven AI Evaluation

• 💡 Building with Generative AI can feel like an art, often relying on manual testing of prompts and gut feelings.
• 🎯 Stax is introduced as an AI evaluation platform designed to replace subjective, manual processes with repeatable, data-driven evaluations.

Building an AI Travel Agent with Stax

• 🗺️ The demo focuses on creating an AI travel agent that recommends unique hidden gems.
• ✍️ Users can start by manually typing prompts in the playground, defining a system prompt to guide the AI's persona and recommendations.
• 📊 Stax automatically saves these examples, building an evaluation benchmark from user inputs and AI responses.
• ☁️ Alternatively, production data can be uploaded as a CSV to create the dataset.

Generating and Evaluating AI Outputs

• 🚀 Stax allows generating AI outputs at scale by choosing from major model providers or connecting custom models.
• ✅ Human ratings can be provided for individual outputs, but the core value lies in automatic grading using evaluators.
• 🧩 Stax includes preloaded evaluators for common criteria like instruction following, but also allows defining custom evaluators specific to product needs (e.g., scoring 'hidden gem' quality).

Analyzing Results and Making Decisions

• 📈 Stax displays evaluator scores for each data point, enabling analysis of specific failures and iteration on prompts or AI models.
• 📊 Aggregated metrics allow for goal setting and head-to-head comparison of different models or prompt iterations.
• ⚡ For example, Gemini 2.5 Flash scored 86 on a hidden gem evaluator with 11.4s latency, while GPT4 Turbo had a score of 62 with 6.3s latency, providing data for model selection based on priorities like speed vs. quality.
• 🔄 The platform provides a rerunnable dataset and evaluator for testing new models or prompts, ensuring data-driven decisions for specific use cases.