Speaker - DevFest Mediterranean 2025

Speaker at DevFest Mediterranean discussing the unpredictability of LLMs, the Air Canada case, and deterministic architectures using Instructor.

The Illusion of Control: Managing LLM Unpredictability

At DevFest Mediterranean 2025 (GDG Nebrodi), I presented a deep dive into the architectural challenges of integrating Large Language Models into production systems.

The talk kicked off with the infamous Air Canada chatbot incident, highlighting what happens when we blindly trust a probabilistic model to act as a customer service agent.

Key Concepts Explored:

• The LLM “People Pleaser” Problem: Why models often prioritize politeness and social grounding over logic and truth, leading to convincing hallucinations and sycophancy.
• Chain of Thought & Its Limits: Exploring how reasoning in LLMs is purely linguistic and emergent, and how intermediate reasoning steps can sometimes be unfaithful or logically flawed.
• The Model as an Interpreter: A paradigm shift—instead of letting the LLM take actions directly, we use it merely as an interpreter to translate natural language into structured data (JSON/XML).
• Deterministic Guardrails with Instructor: I showcased how to use libraries like Instructor and Pydantic to enforce strict schema validation, implement automatic retries for constraint violations (e.g. Firstname[space]Lastname), and delegate the final decision-making to rigid, deterministic code.

“We manage unpredictability by constraining the output into a rigid structure. We secure execution by delegating the final decision to deterministic algorithms.”

This presentation aimed to give developers the practical patterns needed to build resilient and safe AI applications.