Michelle Yi on Trustworthy AI, Jailbreaking LLMs, and Startup Funding

Defining Trustworthy AI

• 💡 Trustworthy AI encompasses security against adversarial attacks and ensuring data integrity to prevent hallucinations and negative behaviors.
• 🎯 Defense strategies involve securing both the model and the data it interacts with.

Red Teaming and Evaluation

• 🛡️ Red teaming involves dedicated teams systematically testing AI models to uncover out-of-distribution use cases and edge cases, ensuring robust performance.
• 📊 Evaluation is crucial for understanding model performance beyond basic metrics, often overlooked in production deployments.
• 🚨 Attackers can exploit models through adversarial attacks, aiming to poison data or induce jailbreaks and hallucinations for nefarious purposes.

Advanced AI Security Threats

• 💥 Jailbreaking LLMs involves manipulating them to bypass safety protocols and generate unintended outputs, akin to jailbreaking a phone for unauthorized access.
• 🔒 Prompt stealing protects intellectual property by preventing competitors from extracting proprietary prompts used to fine-tune AI models.
• 📦 Slop squatting is a cybersecurity vulnerability where malicious packages are created with names similar to those hallucinated by AI, tricking developers into installing malware.
• 🕵️ PII extraction can be achieved indirectly by repeatedly prompting models with specific words, causing them to output sensitive personal information as if it were an end-of-sentence token.

Building Trustworthy AI Systems

• 🧠 Constitutional AI aims to align AI models with a set of overarching principles, acting as a guiding constitution for their behavior.
• 🌍 World models provide AI with a deeper understanding of the physical world, enabling them to simulate consequences and prevent dangerous or nonsensical outputs.
• 🛠️ Techniques like set theory attacks can manipulate models, even black-box commercial ones, by adding subtle, imperceptible perturbations to data, altering predictions.
• 📈 Evaluation (Eval) is critical for maintaining gold-standard benchmarks to track model performance over time and detect subtle degradations or new vulnerabilities.

Generational Equity and AI

• 🚀 Generationship is an organization focused on addressing the underfunding of early-stage female founders, aiming to increase their access to capital and resources.
• 📊 Statistics show that only about 2% of venture capital funding goes to female founders, highlighting a significant disparity.
• 🤝 Collaborations with initiatives like The Tech Bros help create a supportive ecosystem for women in tech and entrepreneurship.

Causal Graphs and AI

• 📈 Causal graphs help distinguish correlation from causation, enabling a deeper understanding of relationships within data.
• 🤖 LLMs can assist in structuring data and automating the labor-intensive process of building these graphs, facilitating analysis of interventions and confounding variables.