Building Agentic AI: Design Patterns, Evaluation, and Optimization with Sinan Ozdemir

Agentic AI vs. Workflows

• 💡 An agent is defined as an LLM with access to tools, capable of deciding which tools to use and in what order.
• ⚙️ A workflow, conversely, is a deterministic data and code path where the LLM's actions are predetermined and it does not choose its next step.
• ❓ To distinguish between them, analyze the existing process: if there are many conditional branching points, it suggests an agentic approach.

LLM Parameter Counts and Context Windows

• 📏 Small models (under 10 billion parameters) can run on a CPU and are suitable for simple retrieval tasks.
• 🚀 Medium-sized models (10-100 billion parameters) enable multi-turn agentic tasks and can be enhanced with fine-tuning.
• 🏢 Large models (100 billion+ parameters) are necessary for enterprise-wide, multilingual deployments and complex tasks.
• 🪞 Larger context windows are crucial for agents performing long-horizon tasks, but the LLM must also be capable of reasoning over the entire context.

Evaluating AI Performance

• 🎯 Accuracy alone is insufficient; evaluation must consider task-specific metrics.
• ⚖️ Precision is vital when false positives are expensive, measuring how often a 'yes' prediction is correct.
• 📉 Recall is critical when false negatives are expensive, measuring how many of the correct 'yes' instances were identified.
• 🧪 Reproducible experiments are essential, with evaluation language integrated into case studies.

Hybrid Systems and Optimization

• 🧩 Hybrid systems, combining predefined workflows with agentic behavior, are often the most powerful AI applications.
• ⚠️ Without a predefined pathway, a sophisticated auditing system is needed to ensure tasks stay on track.
• 🛠️ Optimization techniques like quantization, distillation, and LoRA aim to reduce cost and increase speed, but practitioners should expect a performance hit and potential differences in output compared to larger, unoptimized models.

Surprising Findings in AI Research

• 🤯 A surprising finding is the lack of consistent correlation between reasoning capabilities and LLM performance on certain benchmarks.
• 📈 Even when reasoning improves performance, the gains are often marginal (1-2%) and may not outweigh the increased cost.
• 🧭 Speculative decoding can offer speed and memory benefits, but its effectiveness is task-dependent, allowing for prediction of which questions will benefit most.