Microsoft Foundry: Knowledge Management for AI Agents

Watch the talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=1800s

The Foundry team splits agent knowledge into structured, unstructured, and learned, then demos an agent retrieval stack where the LLM checks whether what it pulled actually answers the query before returning it. Good, concrete material on retrieval.

From 10 Terminals to Managing a Company of Agents

Watch Peter Steinberger’s talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=3492s

Steinberger traces his shift from manually polling ten terminal agents to running one persistent manager agent that delegates to a team and wakes itself on events. Not super practical yet, but a sharp look at where the edge is going.

GLM 4.2 / GLM-Zero 2.0 Keynote

Watch the Jina AI / ZAI keynote

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=4031s

An open-weight GLM release with a selectable thinking budget, low, medium, or high, to trade tokens against capability per task. Mostly benchmarks, but I’m always a sucker for good open source.

Minimax M3: Sparse Attention and Open Research Culture

Watch the talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=4872s

M3 uses multi-head sparse attention. One branch narrows the context and another computes only on the selected blocks to reach roughly 1M-token agentic context, and the design came from an intern. Another good open-source showing.

Browser Agents Demo

Watch Kushan’s early framing and the later demo.

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=6934s

Later demo URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=9726s

Kushan’s early framing: browser agents struggle because the information environment around the model is poor, not because the model is weak, so better environment design beats a smarter model.

The Quality of Vibe-Coded Pull Requests

Watch the Grubtile talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=11710s

Using millions of PRs, they measured AI-authored code and found it comparable to human code on revert rates and review rounds but with different error profiles by tool. I like that they shifted from arguing about AI coding to actually measuring it.

The Napkin Math Origin of Turbo-Puffer

Watch the talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=14656s

I use this story in my course. I love their approach to coming up with TurboPuffer, which started as vectors stored as files in S3, and landed Cursor as the first customer. I love this origin story and can’t get tired of hearing it.

Antigravity 2.0: Agent Teams and the 2026 Primitives

Watch the Google DeepMind talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=17100s

The talk walks the evolution toward “agent teams” and introduces three 2026 primitives: dynamic subagents an orchestrator spins up in parallel, sidecars that act as long-lived listeners reacting to webhooks or cron, and generative UI.

The emphasis on subagents and sidecars matches what customers are asking me to build.

Token Town

Watch the Notion talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=21600s

Notion argues your model provider is structurally your competitor, that auto-upgrading every request is financially ruinous, and that multi-model routing should be an architecture decision from day one. Notion’s talks are always solid and worth the time.

Building Loops for the Real World

Watch the HumanLayer talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=25001s

Kyle argues the “pipe a prompt into a loop and ship 40,000-line PRs” model works for a solo dev but breaks for teams with customers, SLAs, and compliance, since loop-written code is effectively read-only. A useful counterweight on the human and team side of how our practices need to change.

Resonate: Agentic Engineering for Durable Execution

Watch the talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=26100s

After an agent failed going straight from spec to a distributed-systems build, the team reframed to “what does the agent need to design it first?” and gave it a deterministic simulation that injects real failures so it finds the right algorithm before writing production code.

A great example of first-principles thinking and reframing to solve a problem.

Recursive Model Improvement

Watch Lee Robinson’s Cursor talk

URL: https://www.youtube.com/watch?v=htM02KMNZnk&t=30278s

Cursor trains its Composer model with two loops: an outer one that turns user feedback into better eval benchmarks and an inner RL loop that climbs them. They also note that smarter models start hacking the evals by digging through git history for the expected answer.

Cursor always shares good high-level detail.

Field Guide to Fable

Watch Thariq Shihipar’s talk (Anthropic)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=980s

Classic Thariq. He explains agentic coding intuitively, and it’s an easy watch.

In the Land of AI Agents, the Verifiers Are King

Watch Tariq Shaukat’s talk (Sonar)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=2180s

This felt like where the field is going. The code we’re generating isn’t all trustworthy, so the verifier has to become part of the production system rather than an afterthought.

Memory Harnesses for Long-Running Research Agents

Watch Stefania Druga’s talk (Sakana.ai)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=10280s

A useful experiment on recall and memory. The takeaway: memory isn’t just dumping more stuff into context, it’s a write/read policy you can measure.

The Last Thing AI Will Take Away From Software People

Watch the talk

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=11015s

This is the kind of thing I teach in my courses: what is the point of the project? That framing is the real scarce skill.

The Era of (Auto) Research

Watch Elie Bakouch’s talk (Prime Intellect)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=11780s

Anyone who follows Prime Intellect knows they produce great research. Lots of good data here on agents doing auto research.

An AI Agent Became the #1 Contributor in OpenAI’s Hiring Challenge

Watch Zhengyao Jiang’s talk (Weco AI)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=18380s

Good practical advice wrapped in an interesting example. Agents are already strong execution amplifiers when the environment is well-shaped.

Reflective Optimization

Watch Lakshya Agrawal’s talk (GEPA)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=20180s

Good background on GEPA and optimization, which everyone should understand. The important idea is that traces are not just debugging output; they’re raw material for improving prompts, harnesses, skills, and other text artifacts.

Autoresearch for Kernels

Watch Tejas Bhakta’s talk (Morph)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=21680s

Another practical example, this one using kernels. Kernels are a great domain because the verifier is real: correctness and speed. The human gives the high-level idea, the agent searches variants, and a hard verifier decides.

The Log Is the Agent

Watch Isha’s talk (Amnara)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=23248s

I’m a big fan of logs, so this one clicked.

Autoresearch in the Wild / Productizing Loops

Watch Roland’s talk (Introspection)

URL: https://www.youtube.com/watch?v=4sX_He5c4sI&t=23405s

More good stuff on loops. Failures become evals, repeated behavior becomes skills, and taste becomes system behavior over time.

Tokens Should Have Jobs (Katelyn Lesse and Angela Jiang, Anthropic)

Watch Katelyn Lesse and Angela Jiang’s talk (Anthropic)

URL: https://www.youtube.com/watch?v=I2cbIws9j10&t=7012s

Interesting idea, and it made me think. Instead of treating tokens as a generic pile of compute, assign them jobs: execute, advise, grade, or reflect. The practical question becomes not “did more tokens improve the average score?” but “which token roles give the cheapest reliable answer?”

Loophole (Brendan Rappazzo)

Watch Brendan Rappazzo’s talk

URL: https://www.youtube.com/watch?v=I2cbIws9j10&t=16915s

Nice creative thinking. The idea of using adversarial agents to stress-test policies, constitutions, terms, or agent permissions has value. It is basically fuzzing for rules where you generate edge cases and use those cases to patch the policy or escalate ambiguity to a human.

We Let an AI Agent Execute Bash and Lived to Talk About It (Sarah Sanders, PostHog)

Watch Sarah Sanders’ talk (PostHog)

URL: https://www.youtube.com/watch?v=I2cbIws9j10&t=20106s

Good stuff on agent security learnings and practical guardrails. The big lesson is that prompts are not security. If an agent can execute commands, you need deterministic enforcement.

How We Solved Agent Building (Andrew Qu, Vercel)

Watch Andrew Qu’s talk (Vercel)

URL: https://www.youtube.com/watch?v=I2cbIws9j10&t=23407s

Good practical example of the modern way of building agents: away from chains of narrow agents and toward a strong model with tools, files, a sandbox, and domain context.

Agents Without Code (Philipp Schmid, Google DeepMind)

Watch Philipp Schmid’s talk (Google DeepMind)

URL: https://www.youtube.com/watch?v=I2cbIws9j10&t=24899s

Good on the fundamentals. As models improve, harnesses should often get simpler, not more complicated. Instead of writing custom Python functions for every action (no chains), give the agent general tools, a sandbox, credentials handled outside the model, and behavior defined through files and skills.

1. Engineering the Harness: A Practical Workshop

Title slide for the workshop.

2. Engineering the Harness: A Practical Workshop

A second title frame before the talk begins.

3. The tasks changed.

Three years ago, we used language models like this. Hand it a review, ask for sentiment, get an answer back. 30 tokens. 0.2 seconds. Today we’re asking them to do this. Look at a whole codebase, find a bug, write a patch, run the test suite, and verify it worked. 12 million tokens. 20 minutes.

4. The model isn’t solving the problem. The system is.

And this is what the telemetry for a modern coding task actually looks like. Hundreds of tool calls. Millions of tokens of output. Which brings me to the thesis of this talk: the model isn’t solving the problem. The system is.

5. Hi, I’m Rajiv, and this is a masterclass on Harnesses.

I’m Rajiv Shah, Agentic AI Engineer at OpenHands. We build the open-source harness that wraps models like the one in that trace. The next hour is a practical tour of what’s actually inside a harness, and which decisions matter most. This is a workshop, so interrupt me. If something doesn’t land, ask. We’ll have explicit pauses to discuss along the way.

6. This has been an evolution

Quick step back before we dive in. Our focus as a community keeps moving up the stack. 2022 — we cared about weights. Fine-tuning, RLHF. 2023 — context. RAG, long context. 2024 — tools, skills, MCP. And now, in 2026, the outermost layer — the harness — is where the action is.

7. What is in a harness?

Let’s define it clearly, because this gets confused all the time. Agent, harness, SDK — people use these words interchangeably. Here’s the mental model I want you to carry for the rest of the hour. The model reasons. The harness does everything else. Look at this diagram. The model sits in the middle — it reasons and decides. Everything around it is the harness. Context injection on top: prompts, memory, skills, conversation. Control on the left: compaction, orchestration, loops. Action on the right: bash, tools, MCPs. Persistence at the bottom: filesystem, git, progress files. And observe and verify: browser screenshots, test results, logs. If you take nothing else from this talk: Agent = Model + Harness.

8. A harness is everything outside the model

Here’s the loop in concrete terms. Prompt → model → action → feedback. Not one shot. Modern coding agents iterate fifty to two hundred times to finish a task.

9. A harness is everything outside the model

And here’s what makes that loop reliable — the harness wrapping it. State — repo, memory, history, summaries. Control — iteration limits, retries, task contracts. An execution engine that turns decisions into shell commands and file edits. Feedback ingestion that captures test failures and feeds them back. All that scaffolding on the outside is what turns a while loop into an agent that actually finishes work.

10. A good SDK abstracts the harness for agentic actions

You don’t write this from scratch. We built the OpenHands SDK so you can wire up a workspace, an agent, tools, and a conversation loop in about twenty lines of code. Claude Code, Codex CLI, Factory, OpenHands — every serious harness abstracts this same set of concerns. The SDK hides the plumbing.

11. Same model, 2× performance gap

And the harness matters. A lot. Three pieces of evidence. First: same model — Claude Opus 4.5. Put it in Claude Code’s harness, it hits 95% on CORE-Bench Hard. Put it in a naive Hugging Face Smolagents setup, it drops to 42%. Same weights. Same intelligence. The harness alone moves you fifty-three percentage points.

12. Everyone on the leaderboard uses the same model

Second: Terminal Bench leaderboard, today. Look at the model column. Claude Opus 4.6, all the way down. Every top entry is running the same model — they’re competing on harness design. That’s the entire contest.

13. Small model + good harness > big model

Third: a great harness can flip the math entirely. The AutoHarness paper pairs Gemini 2.5 Flash — a small, cheap model — with a well-designed harness, and it beats GPT-5.2 High. Cheaper model, better result, because the scaffolding was better. Three independent proofs, same conclusion. The harness decides more than the model.

14. What harness do you use?

Quick show of hands before we go deeper. Who here uses Claude Code? Codex CLI? Cursor? OpenHands? Something else? Cool — . Hold onto that. We’re going to come back to what your harness is deciding for you.

15. Harness carries a lot of decisions

Which means the next question is: whose harness? Take Claude Code versus Codex CLI. Same category of product. Very different harness decisions. How permissions work. How CLAUDE.md vs AGENTS.md gets loaded. How the sandbox behaves. How much of the internals you can see. Every row in this table is a decision somebody already made for you.

16. Harnesses carry technical debt

This is the setup slide for the next argument: harnesses accumulate technical debt quickly, so yesterday’s orchestration defaults become today’s hidden liabilities.

17. Harnesses are evolving with the models.

And those decisions don’t stay put. Three years ago we were all using AutoGen and CrewAI — nobody runs those in production anymore. Boris Cherny, who leads Claude Code at Anthropic, frames it this way: as models improve, your harness should get simpler. Stress-test your harness. Is this code load-bearing, or just legacy overhead? Manus is the cleanest example — they rebuilt their harness five times in six months. Each rewrite removed complexity. Complex tool definitions became general shell execution. Management agents became simple structured handoffs. So: are harnesses actually getting simpler? Let’s check.

18. As models improved, who has noticed the trend towards shorter system prompts?

Show of hands — anyone here notice that as models have gotten better, system prompts have gotten shorter? That’s the story Boris and a lot of harness builders tell: better models, less hand-holding. I wanted to check that. Let me show you what I found.

19. System prompts getting longer!

I pulled the last four Claude Opus system prompts — May 2025 through April 2026 — and measured them by category. The prompt more than doubled in eleven months. 1,714 words to 3,686 words. Safety nearly tripled. Behavioral guidance roughly tripled. Identity barely moved. Knowledge stayed flat. So the honest answer is: some things get simpler — the behavioral patches for letter-counting, for puzzle constraints — those got retired into training. But the structural stuff — safety, tools, agentic guidance — keeps growing. Don’t take the vendor story at face value. Measure.

20. Claude Code Harness / Architecture

And once you accept that the vendor’s story doesn’t match reality, the next question is: what is actually under the hood? Here’s what Claude Code’s harness looks like — this came out of a leak. Agent loop. Compaction pipeline. Permission system. Hook pipeline. MCP tools. Subagent spawning. Shell sandbox. It’s a lot. And every one of those boxes is a decision made for you.

21. Harnesses have bugs

And that complexity has real consequences. A few days ago — Anthropic posted a postmortem. Claude Code had a regression. The model didn’t change. The harness changed. Default reasoning dropped from high to medium. A bug evicted thinking blocks to save cache. A system prompt tweak reduced verbosity — which reduced code quality. Three harness bugs. Users felt them immediately.

22. The 5 Levers of Harness Engineering

Which brings us to the roadmap. If you want to own your harness — and I’m hoping by now you want to — there are five levers you control. The model is one, but we all know how to swap models. The next four are where the variance actually lives: retrieval, memory, loops, and architecture. That’s the rest of this hour. Starting with retrieval.

23. Let’s start with how agents find what they need.

First lever: how the agent finds what it needs. Retrieval.

24. The three modes of Agentic Retrieval.

Three buckets. Lexical search — keyword-based. Language models — semantic meaning with embeddings. And agentic search — dynamic queries driven by the model’s own reasoning. The industry default is to reach for semantic. For coding agents, you need to reconsider that.

25. The Baseline: grep.

The baseline isn’t a vector database. It’s grep. Keyword precision. Sub-second latency. Battle-tested. If you can solve your problem with grep, you’ve already won.

26. State-of-the-Art Coding Agents rely on grep

And this isn’t a contrarian take. Look at Claude Code’s own docs — Glob and Grep right there in the tool reference. Pure lexical search. Boris Cherny and Catherine Wu, the architects, went on record: they tested lexical versus vector and lexical was much better for code. Cursor is reportedly ripping out their entire vector search infrastructure.

27. Inverted Indices (BM25) make grep instant.

And if you need to scale grep across a big repo, BM25 makes it instant. Inverted index — same idea Lucene’s used for twenty years. Look at this table: linear grep across 9,000 docs takes thirty seconds. BM25 does it in 360 milliseconds. You don’t need a vector DB to search code.

28. Context Harness Uses BM25

This slide translates the BM25 point into harness design: once the environment is indexed, lexical retrieval stays fast enough to be the default inside many coding agents.

29. Where Lexical breaks down: The Synonym Gap.

Where lexical does break: the synonym gap. You search for ‘physician,’ the doc says ‘doctor.’ BM25 misses it. You search for ‘IBM,’ the doc says ‘International Business Machines.’ BM25 misses that too. If your queries don’t share tokens with the source, lexical can’t help you.

30. Embeddings solve for meaning.

That’s where embeddings come in. Encode meaning, not tokens. Cosine similarity finds documents that mean the same thing even if they don’t share words.

31. Semantic search is useful for massive codebases.

And for massive codebases, semantic earns its keep. Cursor’s telemetry shows about a 12.5% accuracy bump on large repos. The changes are also more likely to stick — to actually get retained in production.

32. Who’s using Agentic Search?

Agentic search is already showing up in real tools. The practical question is when the extra latency is worth paying for the added reasoning power.

33. Agentic RAG

Which brings us to agentic search. Different paradigm entirely. Watch traditional RAG explore once and stop, versus an agent that keeps querying.

34. Agentic Search trades latency for massive accuracy.

Traditional RAG makes one query, returns ten chunks, you’re done. Agentic search hands the LLM the search tool. The model looks at what it got, realizes it didn’t find the answer, rewrites the query, and tries again. Five seconds becomes twenty-five. But accuracy goes from 76% to 93% on WixQA. That’s the trade.

35. Coding agents are really good at long Context

For coding specifically, agents with file access dominate. A coding agent with just bash and grep beats traditional RAG and ReAct by 22% to 77% across long-context benchmarks. The model gets to see the code, not just chunks of it.

36. Use files instead of chunking RAG approach.

Which leads to the rule: don’t chunk if you don’t have to. If your model’s context window can hold the file, give it the whole file. Look at this — Gemini 3 Pro with whole-file BM25 access ties human performance and beats chunked RAG.

37. So when should you move to a database

Files stay the source of truth. Move to a database only when you need metadata joins, ranking, or recency. Otherwise stay on files. Add an index on top when query cost bites.

38. Design rules for you

Three rules for the retrieval layer: 1. Lexical baseline. BM25 or grep is your default. 2. Add semantics only when you suffer vocabulary mismatch. 3. Loop it if accuracy matters. Put retrieval inside an iterative agentic loop.

39. Let’s start with how agents find what they need.

This transition moves from retrieval to memory: even if an agent finds the right information, it still needs to remember the right parts of it.

40. Are you excited about 10M Context Windows?

Show of hands — who’s excited about 10 million token context windows? A few of you. Let me show you why I’m not.

41. Memory & State

Second lever: memory. Retrieval gets the agent the right facts. Memory is what stops it from dropping those facts on the floor three turns later. Thesis: agents usually fail because they lose the thread of what they’re doing — not because they hit the token limit. Token limits you can pay your way out of. Losing the thread is harder.

42. 1M Context Windows are never enough.

Yes, we have 1M context windows now. We’ll have 10M soon. But look at the long-context retrieval benchmarks. Every model degrades sharply past 128K. Even Opus 4.6, the best one here, drops from 90% to 78% by the time you stuff it full. Models degrade when you stuff them full. The benchmark capability is not the production capability.

43. 1M Context Windows degrade

This slide reinforces the warning from the previous one: bigger windows do not just cost more, they degrade, so the harness has to decide what deserves to stay in context.

44. Key facts disappear inside long model inputs

And it’s not just degradation, it’s selective degradation. Information in the middle of a long input has only a 0.47 cosine similarity in the model’s summary, versus 0.68 at the start. The middle of your context window is where facts go to die.

45. Coding agents struggle with long context models

It’s easy to burn your context. Watch out for verbose tool traces — you run npm install and get hundreds of lines of deprecation warnings. The agent’s actual goal — what you originally asked it to do — gets pushed out of the window. Vercel devs have shared screenshots of Claude Code degrading sharply past 200K tokens of this kind of noise.

46. Three layers of Memory

So treat memory as three explicit layers, not one giant string. Active Context — what’s in the prompt right now. Working State — plans, TODOs, scratchpads outside the prompt. Durable Memory — skills and reusable workflows that persist across sessions.

47. Layer 1: Fixing Active Context

Layer 1: Active Context. Two ways to manage it. Reset — clear the window entirely, refill with only the original instructions and critical artifacts. Or Compact — summarize older turns, but keep recent turns intact.

48. Layer 1: Compacting from OpenHands

Compaction works. We measure it at OpenHands. Up to 2× per-turn API cost reduction. Consistent response times in long sessions. And — this surprised people — equivalent or better performance on software engineering tasks. Less context, sharper agent. ( ACON research backs this up — 26 to 54% token reduction while preserving 95%-plus accuracy, by prioritizing reasoning traces over raw tool outputs.

49. Layer 1: How does Codex do it???

This is the uneasy part of closed harnesses: compaction is happening for you, but you usually cannot inspect or tune the policy that decides what gets thrown away.

50. Getting the most of a 1M Context Windows

Compaction is just one move. Anthropic publishes a fuller toolkit: continue, rewind, /clear, /compact, subagents. Five different ways to actively manage what’s in the window. Each one is its own policy decision. Let me zoom in on one I like.

One concrete technique Rajiv calls out here is rewind: if the agent went down a bad branch, jump back instead of dragging the failed reasoning forward.

51. Layer 2: Working State (The Golden Rule)

Layer 2: Working State. The golden rule — files make better memory than chat. Don’t keep the agent’s plan in the system prompt. Have it write a plan.md to the workspace. The plan stays out of the context window, but the agent reads from it and checks items off.

52. Deep Agents rely on external plans.

LangChain’s Deep Agents do exactly this. A write_todos tool dumps the plan to a file. The agent reads it, executes step one, updates the file, moves on. The plan never sits in the prompt.

53. Extreme Layer 2: Recursive Language Models (RLM)

At the extreme end of Layer 2, you get Recursive Language Models. RLMs bypass token limits entirely by giving the agent a persistent Python REPL. Variables stay in the REPL between calls. The agent uses the REPL as memory. Recursive sub-LM calls handle scoped subqueries.

54. RLMs maintain accuracy at 1M tokens.

And it works. Standard GPT-5 collapses past 33K tokens on long-context tasks. RLM-GPT-5 maintains 91% accuracy all the way to 1M. Different memory architecture, different ceiling.

55. Layer 3: Durable Memory

Layer 3: Durable Memory. What does the agent remember across sessions? You’ve all seen the ChatGPT version of this — the personalization toggle that promises to remember everything you tell it.

56. Who uses an Agents.md file?

Quick poll. Show of hands — who has an AGENTS.md or CLAUDE.md in their repo right now? Keep your hand up if you wrote it by hand. Now keep it up if you let the model auto-generate it for you. That distinction is going to matter a lot more than you’d think. Let me show you.

57. Durable Memory with Agents.md

The simple version of intentional memory is AGENTS.md. Open format. Adopted by 60,000+ projects. Codex, Cursor, Factory, VS Code, Devin all read it. Dev environment tips, testing instructions, PR conventions. But — and this matters — it’s in every prompt. Don’t overload it.

58. Auto-generated AGENTS.md files hurt performance

In fact, ETH Zurich just published a study: auto-generated AGENTS.md files actively reduce task success across multiple coding agents. Sonnet, GPT-5, Qwen — all show degradation. Inference cost goes up by over 20%. The agent wastes tokens reading boilerplate it doesn’t need.

59. The rule of thumb: “Minimize Load-Bearing Memory”

Boris Cherny’s rule of thumb here: minimum load-bearing memory. Quote — ‘do the minimal possible thing to get the model on track.’ Delete your CLAUDE.md. If the model wanders off, add back one piece. With every new model, you’ll find you need less and less.

60. Skills are the new standard for Durable Memory.

Which brings us to skills. A skill isn’t a prompt. It’s a trigger plus a reference manual plus a script. Loaded on demand, modular, reusable. The skill says: when to run, how to execute, what the rules are. Anthropic, Cursor, VS Code all support them now.

61. Skills as Externalized Expertise

This is the big idea: skills externalize expertise the way memory externalizes state. Authored, distilled, discovered, composed. The agent gets a registry of expert procedures it can invoke when the trigger matches.

62. Skills can replace Code

Cursor talked about this at AI Engineering London. They had 15,000+ lines of code for worktree creation, agent loop scoping, harness changes, reminders — all the orchestration glue. They replaced it with a 200-line skill. Not refactored. Replaced.

63. Building a learning loop with skills

And skills enable continual learning. The Hermes Agent pattern: agent attempts a complex task, gets a periodic nudge — ‘what would you do differently?’ — and writes a skill file. Next time it fails, it edits its own skill. The harness teaches itself.

64. Continual learning outer loop with skills

That’s actually two loops. Inner loop finishes the task in one session. Outer loop, across sessions, makes the agent smarter. Session 1 fails, skill gets updated, Session 2 succeeds. This is the shape of self-improving agents in production today.

65. Warning: 16% of skills actually reduce performance.

But skills aren’t free. SkillsBench tested 50+ skills across multiple coding agents. 16% of them actively reduce performance. They overlap with native tools, confuse the routing, or trigger when they shouldn’t. Skills are powerful and dangerous in the same way prompts are.

66. You must evaluate your skills.

So evaluate your skills. Without-skill versus with-skill, on the same tasks. There’s a tutorial in my GitHub repo. If you’re not measuring lift, you’re guessing.

67. Constant Innovation around Memory

Memory design is still evolving. Compaction, durable memory, and skills all change with the models, which is why these policies need to be revisited rather than frozen.

68. @rajistics

A brief interstitial before the lock-in discussion, pointing back to Rajiv’s broader work on agents and memory.

69. Memory & Claude

Now notice the pattern. With a closed harness, all of this is locked in. Conversation compaction. CLAUDE.md handling. Tool search and MCP loading. Subagent definitions. Permission rules. You don’t see it. You don’t tune it. You can’t measure it.

70. Memory as Lock-in

Closed harness — memory lives in the provider’s API, compaction is encrypted, switch providers and you lose your history. Open harness — memory lives in your files, compaction is your code, switch providers and the memory stays. Lock the harness, lose the memory, lose your product.

71. Let’s start with how agents find what they need.

This divider shifts from memory into loop design: once agents can retrieve and remember, the next question is whether they can act with discipline.

72. Agentic Loops and Tool Use

Core argument — better tool loops beat better prompts. The leverage isn’t in writing a smarter system prompt. It’s in shaping the loop the agent runs inside.

73. Engineering the Loop

Four stages to walk through: 1. The baseline — the Ralph Wiggum loop, and why it fails. 2. Cognitive discipline — forcing thinking via JSON schemas. 3. Environmental discipline — using tests and CI to physically block bad loops. 4. Safety and friction — sandboxing autonomous actions. And here’s the math that makes this matter. A 10-step process with 99% per-step success has only 90.4% end-to-end success. At 50 steps you’re at 60%. Errors compound fast. Every stage we walk through exists to keep that compounding curve under control.

74. We no longer rely on single-shot execution.

Quick stake in the ground. Single-shot is dead for long tasks. Models must use tools, see the result, loop. That’s the entire architectural lesson of OpenAI’s o1.

75. We no longer rely on single-shot execution.

Look at Opus 4.7’s published tool list — over twenty tools shipped with the model. bash_tool, web_search, tool_search, view, create_file, str_replace. Every one designed for a model that plans, acts, and iterates. The model itself is now a harness-aware artifact.

76. The Default: The “Ralph Wiggum” Agent

But a while loop isn’t an agent. The default behavior of most frameworks is what I call the Ralph Wiggum loop. Try a command. Get a stack trace. Retry the same command without diagnosing the error. Repeat until the harness hits max_iterations. I’m learnding.

77. Ralph can work

And Ralph occasionally works. There’s a write-up of porting 600,000 lines of C in four days using exactly this loop. If the plan is perfect and nothing breaks, Ralph builds things. But in the real world, Ralph destroys your token budget.

78. Cognitive Discipline via JSON Schema and Plan

This is the gap between a prompt engineer and a system engineer. A prompt engineer writes ‘always think step-by-step’ in the system prompt and hopes the model listens. A system engineer uses JSON Schema. Don’t let the model pass {command: 'run script'}. Configure the tool schema to require a hypothesis, a verification_plan, then the command. If any field is missing, the harness rejects the call before it hits the sandbox. You physically force the model to think before it acts.

79. Moving from Ralph Wiggum to AutoResearch

Karpathy published this earlier this year — his auto-research loop. Look at the chart on the left: 83 experiments, 15 kept improvements, monotonically decreasing validation BPB. That’s a Ralph loop with one critical addition — a metric gate. Every iteration either improves the score and gets committed, or gets git reset and discarded. This is the bridge between Ralph and a real loop: observe state, hypothesize, act, verify, keep what works. Same scientific method as before, but now the environment enforces the verification step. That’s our lead-in to cognitive discipline.

80. An Improved Loop for AutoResearch

The improved loop adds what Ralph is missing: hypothesis, action, evaluation, and a rule for keeping only the changes that actually work.

81. Defensive Tool Returns.

And the harness manages the output too. Agent runs cat massive.json? A good harness intercepts the output, truncates it, and returns: <Output truncated to 2000 lines. Use grep or head instead.> This prevents the agent from blinding itself.

82. Environmental Discipline: Testing Driven Development

Even if the model is thinking perfectly, it will eventually drift. Move to environmental discipline. Factory’s Luke Alvoeiro: tests written after implementation don’t catch bugs — they confirm the agent’s hallucinated decisions. Validation contracts must be defined before the agent writes a line of code. Adversarial by design.

83. Adding System Constraints for your Harness

Ryan Lopopolo from OpenAI summarizes it: Code is free. Architecture is expensive. Because agents can write code infinitely, your system’s constraints become your harness: - Lint errors → instructions to the agent. - File-length tests → force decomposition. - CI checks → the real guardrail.

84. Safety & Friction: Sandboxes

Last stage: safety. These loops are autonomous, they’re executing bash, they will eventually try to destroy things. Look at this — March 8th, Claude Code deleted a developer’s production database setup, including snapshots. Two and a half years of records, gone in an instant. This isn’t theoretical. Sandboxes are non-negotiable.

85. Safety & Friction: Sandboxes

The second sandbox slide makes the operational point concrete: isolated execution has to be the default once agents can touch shell commands, credentials, and production-like state.

86. Guardrails and Approval Friction

So design friction to match blast radius: - Safe (read, grep, ls) → auto-allow. - Reversible (edit file, git commit) → auto-allow inside a sandbox. - Network (curl, npm install) → prompt once per session. - Destructive (rm -rf, DROP TABLE, force push) → require explicit human approval. The harness halts the loop at the right moments.

87. Principles for Agentic Loop

Four rules for the loop: 1. Make the next action explicit — JSON schema. 2. Make verification cheap — CI, tests, lint as adversarial boundary. 3. Make blind repetition hard. 4. Force learning from the environment. Retrieval feeds the loop. Memory stabilizes the loop. Protocols and tests discipline the loop.

88. Let’s start with how agents find what they need.

This divider sets up the final lever: architecture. Up to this point the talk assumes one harness and one loop; here the question becomes whether splitting the work actually helps.

89. System Architecture: Single versus Multi Agent

Last lever: architecture. Up to now I’ve talked as if there’s one agent with one harness. Architecture is where you decide whether that loop stays centralized or gets broken into specialized workers.

90. Who’s using a multi-agent for coding?

Hands up — who’s actually run a multi-agent system in production? Now keep your hand up if it worked better than your single-agent version. Right. Let’s talk about why that happens.

91. Single agents degrade as complexity grows.

Why does the multi-agent intuition fail? Start here. Single agents degrade as complexity grows. Even GPT-5: with 3 tools and 1.4K tokens, four distractor tools cost you 10%. With all tools and 300K tokens, the same distractors cost you 17%. The model doesn’t know which tool to pick. Vercel saw this in production — they removed 80% of v0’s tools and got better results. More tools doesn’t mean more capability. It usually means worse routing.

92. Split the context between multiple agents

So sometimes we use subagents. Same study: split the context and tools across specialized workers, and multi-agent setups outperform single-agent baselines, especially for the smaller models. Promise is real.

93. Multi-Agent is like Distributed Systems: Complex!

This is the warning slide for multi-agent design: once you introduce several workers, you inherit the same coordination, debugging, and observability headaches that make distributed systems hard.

94. Many ways to orchestrate multiple agents

And there are clean patterns to reach for: prompt chaining, routing, parallelization, orchestrator-worker, evaluator-optimizer. These show up everywhere — they’re worth knowing.

95. Coordination Tax - Going from Parallel to Serial

But before you run off and build a 15-agent swarm — every multi-agent setup pays a coordination tax. Factory hit this directly. Started with parallel swarms — agents stepped on each other, performance tanked. They moved to strict serial handoffs: orchestrator → worker → validator. Every conflict burns tokens. Every retry erodes trust.

96. The Reality: More agents only help if coordination stays cheap.

And here’s what the data shows. Across BrowseComp, Finance, PlanCraft, Workbench — multi-agent systems perform on average 3.5% worse than single agents. Once a single agent hits about 45% accuracy, adding more agents stops helping. Independent agents amplify errors 17×. Subagents are something you earn — not something you set up on day one.

97. Multi-Agent critics using reflection

There IS one multi-agent pattern that consistently works, though — the critic. A separate agent — sometimes a different model entirely — reviewing the main agent’s trace. Look at the data: Reflexion-style critic loops on SWE-bench. Random sampling 57.9%. Success-only 63.6%. Iterative critic with rubrics: 73.8%. That’s a 10-point bump from adding one critic agent. And Boris Cherny said it bluntly — giving the model a way to verify its work improves quality 2 to 3×. That’s the practitioner number behind the academic data. When you do reach for a second agent, this is the move that pays off.

98. Harness engineering in another two years?

Now — the question I get every time I give this talk. Is harness engineering just the new prompt engineering? The next thing that quietly disappears in two years? Honest answer: yes and no. Some of this commoditizes. Default compaction algorithms, standard tool choices like grep and edit, model-specific prompt tweaks, most of the hacks that patch capability gaps — those will disappear into the platform. They become defaults. But other parts are durable. Skills systems — your organization’s accumulated expertise. Memory policy — what you persist, what you throw away. Domain-specific tool libraries. Security posture. Evals — knowing when your harness breaks. Prompt engineering didn’t actually die. We just stopped naming it. Same thing’s going to happen here. The durable parts become plumbing. And plumbing is where the leverage lives.

99. Five knobs that decide everything

To wrap up — five knobs that decide everything. Tune them in this order: 1. Retrieval → grep and BM25 by default. Add semantic only when vocabulary mismatch hurts you. Loop it for accuracy. 2. Memory → files beat chat history. Skills for durable expertise. Evaluate them. 3. Tools → quality over quantity. Defensive truncation. Schema-enforced thinking. 4. Loops → force hypothesis before action. Tests before code. 5. Orchestration → single agent by default. Multi-agent only when coordination stays cheap.

100. Why Harnesses Matter

The closing summary returns to the three reasons harnesses matter: performance differences, the knobs you can tune, and the lock-in risk when the harness owns your memory and tools.

101. Engineering the Harness: A Practical Workshop

Here’s where I’ll leave you. An agent isn’t a magical model. An agent is a model plus a harness. The intelligence of your product isn’t just in the LLM’s weights — it’s in the execution layer that connects that model to tools, state, and the work that has to actually finish. Stop tweaking prompts. Start engineering the harness. The QR code on screen will take you to the GitHub repo with the full script, references, and runnable experiments. Thank you. Questions.

This annotated presentation was rebuilt directly from the local slide deck and talk track.

1. Title Slide

(Timestamp: 00:00)

This slide introduces the workshop titled “Hill Climbing: Best Practices for Evaluating LLMs,” presented by Rajiv Shah, PhD, at the Open Data Science Conference (ODSC). The presentation focuses on the technical nuances of Generative AI and how to build effective evaluation workflows.

Rajiv sets the stage by outlining his three main goals for the session: understanding the technical differences in GenAI evaluation, learning a basic introductory workflow for building evaluation datasets, and inspiring practitioners to start “learning by doing” rather than just reading papers.

The concept of “Hill Climbing” refers to the iterative process of improving LLM applications—starting with a baseline and continuously optimizing performance through rigorous testing and error analysis.

2. Evaluating for Gen AI Resources

(Timestamp: 00:06)

This slide provides a QR code and a GitHub URL, directing the audience to the code and resources associated with the talk. It emphasizes that the workshop is practical, with code examples available for attendees to replicate the evaluation techniques discussed.

Rajiv encourages the audience to access these resources to follow along with the technical implementations of the concepts, such as building LLM judges and creating unit tests, which will be covered later in the presentation.

3. Customer Support Use Case

(Timestamp: 00:48)

To motivate the need for evaluation, the presentation introduces a common real-world use case: Customer Support. Generative AI is frequently deployed to help agents compose emails or chat responses based on user inquiries.

This scenario serves as the baseline example throughout the talk. It represents a high-volume task where automation is desirable, but accuracy and tone are critical for maintaining customer satisfaction and brand reputation.

4. Vibe Coding

(Timestamp: 00:59)

This slide introduces the concept of “Vibe Coding”—the initial phase where developers grab a simple prompt, feed it to a model, and get a result that feels right. It highlights the misconception that GenAI is easy because it works “out of the box” for simple demos.

Rajiv notes that while “vibe coding” might work for a quick demo app, it is insufficient for production systems. Relying on a “vibe” that the model is working prevents teams from catching subtle failures that occur at scale.

5. Good Response: Delayed Order

(Timestamp: 01:10)

Here, we see a successful output generated by the LLM. The customer inquired about a delayed order, and the AI generated a polite, relevant response acknowledging the delay and apologizing.

This example reinforces the “Vibe Coding” trap: because the model often produces high-quality, human-sounding text like this, developers can be lulled into a false sense of security regarding the system’s reliability.

6. Good Response: Damaged Product

(Timestamp: 01:12)

This slide provides another example of a “good” response. The AI correctly identifies that the customer received a damaged product and initiates a replacement protocol.

These positive examples establish a baseline of expected behavior. The challenge in evaluation is not just confirming that the model can work, but ensuring it works consistently across all edge cases.

7. Bad Response: Irrelevance

(Timestamp: 01:26)

The presentation shifts to failure modes. In this example, the user asks about an “Order Delay,” but the AI responds with information about a “New Product Launch.”

This illustrates a complete context mismatch. The model failed to attend to the user’s intent, generating a coherent but completely irrelevant response. This type of failure frustrates users and degrades trust in the automated system.

8. Bad Response: Hallucination

(Timestamp: 01:36)

This slide shows a more dangerous failure: Hallucination. The AI apologizes for a defective “espresso machine,” but as the speaker notes, “We don’t actually sell espresso machines.”

This highlights the risk of the model fabricating facts to be helpful. Such errors can lead to logistical nightmares, such as customers expecting replacements for products that do not exist or that the company never sold.

9. Risks of LLM Mistakes

(Timestamp: 01:51)

Rajiv categorizes the risks associated with LLM failures into three buckets: Reputational, Legal, and Financial. He cites the example of Cursor, an IDE company, where a support bot hallucinated a policy restricting users to one device, causing customers to cancel subscriptions.

The slide emphasizes that courts may view AI agents as employees; if a bot makes a promise (like a refund or policy change), the company might be legally bound to honor it. This escalates evaluation from a technical nice-to-have to a business necessity.

10. The Despair of Gen AI

(Timestamp: 02:38)

This visual represents the frustration developers feel when moving from a successful demo to a failing production system. The “despair” comes from the realization that the stochastic nature of LLMs makes them difficult to control.

It serves as an emotional anchor for the audience, acknowledging that while GenAI is exciting, the unpredictability of its failures causes significant stress for engineering teams responsible for deployment.

11. High Failure Rates

(Timestamp: 02:48)

The slide cites an MIT report stating that “95% of GenAI pilots are failing.” While Rajiv notes this number might be overstated, it reflects a trend where executives are demanding ROI and seeing lackluster results.

This shift in 2025 means that evaluation is no longer just for debugging; it is required to prove business value and justify the high costs of running Generative AI infrastructure.

12. Evaluation Improves Applications

(Timestamp: 03:14)

This slide asserts the core thesis: Evaluation helps you build better GenAI applications. It references a previous viral video by the speaker on the same topic, positioning this talk as an updated, condensed version with fresh content.

Rajiv explains that you cannot improve what you cannot measure. Without a robust evaluation framework, developers are essentially guessing whether changes to prompts or models are actually improving performance.

13. Why Evaluation is Necessary

(Timestamp: 03:40)

This concentric diagram illustrates the stakeholders involved in evaluation. It starts with “Things Go Wrong” (technical reality), moves to “Buy-in” (convincing managers/teams), and ends with “Regulators” (external compliance).

Evaluation serves multiple audiences: it helps the developer debug, it provides the metrics needed to convince management that the app is production-ready, and it creates the audit trails required by third-party auditors or regulators.

14. Evaluation Dimensions

(Timestamp: 04:18)

Evaluation must cover three dimensions: Technical (F1 scores, accuracy), Business (ROI, value generated), and Operational (Total Cost of Ownership, latency).

Rajiv highlights that data scientists often focus solely on the technical, but ignoring operational costs (like the expense of hosting GPUs vs. using APIs) can kill a project. A comprehensive evaluation strategy considers the cost-to-quality ratio.

15. Public Benchmarks

(Timestamp: 05:06)

The slide discusses Public Benchmarks (like MMLU, GSM8K). While useful for a general idea of a model’s capabilities (e.g., “Is Llama 3 better than Llama 2?”), they are insufficient for specific applications.

Rajiv warns against using these benchmarks to determine if a model fits your specific use case. Companies promote these numbers for marketing, but they rarely reflect performance on proprietary business data.

16. Custom Benchmarks

(Timestamp: 05:22)

The solution to the limitations of public benchmarks is Custom Benchmarks. This slide defines a benchmark as a combination of a Task, a Dataset, and an Evaluation Metric.

This is a critical definition for the workshop. To “tame” GenAI, you must build a dataset that reflects your specific customer queries and define success metrics that matter to your business logic, rather than relying on generic academic tests.

17. Taming Gen AI

(Timestamp: 05:28)

This title slide signals a transition into the technical “how-to” section of the talk. “Taming” implies that the default state of GenAI is wild and unpredictable.

The goal of the following sections is to bring structure and control to this chaos through rigorous engineering practices and evaluation workflows.

18. Workshop Roadmap

(Timestamp: 05:31)

The roadmap outlines the four main sections of the talk: 1. Basics of Gen AI: Understanding variability and technical nuances. 2. Evaluation Workflow: Building the dataset and running the first tests. 3. More Complexity: Adding unit tests and conducting error analysis. 4. Agents: Evaluating complex, multi-step workflows.

19. Variability in Responses

(Timestamp: 06:00)

This slide visually demonstrates the Non-Determinism of LLMs. It shows two responses to the same prompt generated just minutes apart. While substantively similar, the wording and structure differ slightly.

This variability makes exact string matching (a common software testing technique) impossible for LLMs. It necessitates semantic evaluation techniques, which complicates the testing pipeline.

20. Input-Model-Output Diagram

(Timestamp: 06:24)

A simple diagram illustrates the flow: Prompt -> Model -> Output. Rajiv uses this to structure the analysis of where variability comes from.

He explains that “chaos” can enter the system at any of these three stages: the input (prompt sensitivity), the model (inference non-determinism), or the output (formatting and evaluation).

21. Inconsistent Benchmark Scores

(Timestamp: 06:44)

The slide presents a discrepancy between benchmark scores tweeted by Hugging Face and those in the official Llama paper. Both used the same dataset (MMLU), but reported different accuracy numbers.

This introduces the problem of Evaluation Harness Sensitivity. Even with standard benchmarks, how you ask the model to take the test changes the score, proving that evaluation is fragile and implementation-dependent.

22. MMLU Overview

(Timestamp: 07:25)

MMLU (Massive Multitask Language Understanding) is explained here. It is a multiple-choice test covering 57 tasks across STEM, the humanities, and more.

It is currently the standard for measuring general “intelligence” in models. However, because it is a multiple-choice format, it is susceptible to prompt formatting nuances, as the next slides demonstrate.

23. Prompt Sensitivity

(Timestamp: 07:44)

This slide reveals why the scores in Slide 21 differed. The three evaluation harnesses used slightly different prompt structures (e.g., using the word “Question” vs. just listing the text).

These minor changes resulted in significant accuracy shifts. This proves that LLMs are highly sensitive to syntax, meaning a “better” model might just be one that was prompted more effectively for the test, not one that is actually smarter.

24. Formatting Changes

(Timestamp: 08:22)

Expanding on sensitivity, this slide references Anthropic’s research showing that changing answer choices from (A) to [A] or (1) affects the output.

This level of fragility is a key takeaway: seemingly cosmetic changes in how inputs are formatted can alter the model’s reasoning capabilities or its ability to output the correct token.

25. GPT-4o Performance Drop

(Timestamp: 08:38)

A bar chart demonstrates that this issue persists even in state-of-the-art models like GPT-4o. Subtle changes in wording can lead to a 5-10% drop in performance.

This counters the assumption that newer, larger models have “solved” prompt sensitivity. It remains a persistent variable that evaluators must control for.

26. Tone Sensitivity

(Timestamp: 08:46)

This slide shows that the tone of a prompt (e.g., being polite vs. direct) affects accuracy. Rajiv jokes, “I guess this is why mom always said to be polite.”

The graph indicates that prompt engineering strategies, like adding emotional weight or politeness, can statistically alter model performance, adding another layer of complexity to evaluation.

27. Persistent Sensitivity

(Timestamp: 09:00)

The slide reiterates that despite years of progress, models are still sensitive to specific phrases. It shows a “Prompt Engineering” guide suggesting specific words to use.

The takeaway is that developers cannot treat the prompt as a static instruction; it is a hyperparameter that requires optimization and constant testing.

28. Falcon LLM Bias

(Timestamp: 09:18)

This slide introduces a case study with the Falcon LLM. A user tweet shows the model recommending Abu Dhabi as a technological city with glowing sentiment, which raised suspicions about bias given the model’s origin in the Middle East.

This serves as a detective story: users wondered if the model weights were altered or if specific training data was injected to force this positive association.

29. Potential Cover-up?

(Timestamp: 09:50)

Another tweet speculates if the model is “covering up human rights abuses” because it provides different answers for Abu Dhabi compared to other cities.

This highlights how model behavior can be misinterpreted as malicious bias or censorship, when the root cause might be something much simpler in the input stack.

30. Inspecting the System Prompt

(Timestamp: 10:00)

The reveal: The bias wasn’t in the weights, but in the System Prompt. The slide suggests looking at the hidden instructions given to the model.

In Falcon’s case, the system prompt explicitly told the model, “You are a model built in Abu Dhabi.” This context influenced its generation probabilities, causing it to favor Abu Dhabi in its responses.

31. Claude System Prompt

(Timestamp: 10:33)

Rajiv points out that most developers never read the system prompts of the models they use. He highlights the Claude System Prompt, which is 1700 words long and takes nearly 10 minutes to read.

These extensive instructions define the model’s personality and safety guardrails. Ignoring them means you don’t fully understand the inputs driving your application’s behavior.

32. Complexity of a Single Response

(Timestamp: 11:00)

The diagram is updated to show that a “single response” is actually the result of complex interactions: Tokenization -> Prompt Styles -> Prompt Engineering -> System Prompt.

This visual summarizes the “Input” section of the talk, reinforcing that before the model even processes data, multiple layers of text transformation occur that can alter the result.

33. Inter-text Similarity

(Timestamp: 11:15)

This heatmap compares Inter-text similarity between models. It highlights Llama 70B and Llama 8B. Even though they are from the same family and likely trained on similar data, they are not identical.

This means you cannot swap a smaller model for a larger one (or vice versa) and expect the exact same behavior. Any model change requires a full re-evaluation.

34. Sycophantic Models

(Timestamp: 12:16)

The slide discusses Sycophancy—the tendency of models to agree with the user even when the user is wrong. It mentions how early versions of GPT-4 were sometimes “overly nice.”

This behavior is a specific type of model bias that evaluators must watch for. If a user asks a leading question containing false premises, a sycophantic model might validate the falsehood rather than correct it.

35. Model Drift

(Timestamp: 12:37)

“Model Drift” refers to the phenomenon where commercial APIs (like OpenAI or Anthropic) change their model behavior over time without warning.

Because developers do not control the weights of API-based models, the “ground underneath them” can shift. A prompt that worked yesterday might fail today because the provider updated the backend or the inference infrastructure.

36. Degraded Responses Timeline

(Timestamp: 12:55)

This slide shows a timeline of Degraded Responses from an Anthropic incident. Technical issues like context window routing errors led to corrupted outputs for a period of days.

This illustrates that drift isn’t always about model updates; it can be infrastructure failures. Continuous monitoring is required to detect when an external dependency degrades your application’s performance.

37. Hyperparameters

(Timestamp: 13:33)

The slide lists Hyperparameters like Temperature, Top-P, and Max Length. Rajiv explains that users can control these “knobs” to influence creativity versus determinism.

Setting temperature to 0 makes the model less random, but as the next slides show, it does not guarantee perfect determinism due to hardware nuances.

38. Non-Deterministic Inference

(Timestamp: 14:03)

This slide tackles Non-Deterministic Inference. Unlike traditional ML models (e.g., XGBoost) where a fixed seed guarantees identical output, LLMs on GPUs often produce different results for identical inputs.

Causes include floating-point accumulation errors and the behavior of Mixture of Experts (MoE) models where different batches might activate different experts.

39. Addressing Non-Determinism

(Timestamp: 15:11)

Rajiv references recent work by Thinking Machines and updates to vLLM that attempt to solve the non-determinism problem through correct batching.

While solutions are emerging, the takeaway is that most current setups are non-deterministic by default. Evaluators must design their tests to tolerate this variance rather than expecting bit-wise reproducibility.

40. Updated Model Diagram

(Timestamp: 15:43)

The diagram expands again. The “Model” box now includes Model Selection, Hyperparameters, Non-deterministic Inference, and Forced Updates.

This visual summarizes the “Model” section, showing that the “black box” is actually a dynamic system with internal variables (weights/architecture) and external variables (infrastructure/updates) that all add noise to the output.

41. Output Format Issues

(Timestamp: 16:01)

Moving to the “Output” stage, this slide uses MMLU again to show how Output Formatting affects evaluation. How do you ask the model to answer a multiple-choice question?

Do you ask it to output just the letter “A”? Or the full text? Or the probability of the token “A”? Different evaluation harnesses use different methods, leading to the score discrepancies seen earlier.

42. Evaluation Harness Variations

(Timestamp: 16:35)

This table details the specific differences in implementation between harnesses (e.g., original MMLU vs. HELM vs. EleutherAI).

It reinforces that there is no standard “ruler” for measuring LLMs. The tool you use to measure the model introduces its own bias and variance into the final score.

43. Score Comparison Table

(Timestamp: 16:56)

A spreadsheet shows the same models scoring differently across different evaluation implementations. The variance is not trivial; it can be large enough to change the ranking of which model is “best.”

This data drives home the point: You must control your own evaluation pipeline. Relying on reported numbers is risky because you don’t know the implementation details behind them.

44. Sentiment Analysis Variance

(Timestamp: 17:09)

This slide shows varying Sentiment Analysis outputs. Different models (or the same model with different prompts) might classify a review as “Positive” while another says “Neutral.”

This introduces the concept that even “simple” classification tasks in GenAI are subject to interpretation and variance, unlike traditional classifiers that have a fixed decision boundary.

45. Tool Use Variance

(Timestamp: 17:23)

Radar charts illustrate variance in Tool Use. Models might be good at using an “Email” tool but fail at “Calendar” or “Terminal” tools.

Furthermore, models exhibit non-determinism in decision making—sometimes they choose to use a tool, and sometimes they try to answer from memory. This adds a layer of logic errors on top of text generation errors.

46. Summary: Why Responses Differ

(Timestamp: 17:49)

This comprehensive slide aggregates all the factors discussed: Inputs (prompts, system prompts), Model (drift, hyperparams), Outputs (formatting), and Infrastructure.

It serves as a checklist for the audience. If your application is behaving inconsistently, investigate these specific layers of the stack to find the source of the noise.

47. Chaos is Okay

(Timestamp: 18:17)

Rajiv reassures the audience that “Chaos is Okay.” The slide presents a chart of evaluation methods ranging from flexible/expensive (human eval) to rigid/cheap (code assertions).

The message is that while the technology is chaotic, there is a spectrum of tools available to manage it. We don’t need to solve every source of variance; we just need a robust process to measure it.

48. From Chaos to Control

(Timestamp: 18:27)

This transition slide marks the beginning of the Evaluation Workflow section. The presentation shifts from describing the problem to prescribing the solution.

The goal here is to move from “Vibe Coding” to a structured engineering discipline where changes are measured against a stable baseline.

49. Build the Evaluation Dataset

(Timestamp: 18:37)

The first step in the workflow is to Build the Evaluation Dataset. The slide lists examples of prompts for tasks like summarization, extraction, and translation.

Rajiv emphasizes that this dataset should reflect your actual use case. It is the foundation of the “Custom Benchmark” concept introduced earlier.

50. Get Labeled Outputs (Gold)

(Timestamp: 18:46)

Step two is to get Labeled Outputs, also known as Gold Outputs, Reference, or Ground Truth. The slide adds a column showing the ideal answer for each prompt.

This is the standard against which the model will be judged. While obtaining these labels can be expensive (requiring human effort), they are essential for calculating accuracy.

51. Compare to Model Output

(Timestamp: 19:00)

Step three is to generate responses from your system and place them alongside the Gold Outputs. The slide adds a “Model Output” column.

This visual comparison allows developers (and automated judges) to see the delta between what was expected and what was produced.

52. Measure Equivalence

(Timestamp: 19:10)

Step four is to Measure Equivalence. Since LLMs rarely produce exact string matches, we use an LLM Judge (another model) to determine if the Model Output means the same thing as the Gold Output.

The slide shows a prompt for the judge: “Are these two responses semantically equivalent?” This converts a fuzzy text comparison problem into a binary (Pass/Fail) metric.

53. Optimize Using Equivalence

(Timestamp: 19:57)

Once you have an equivalence metric, you can Optimize. The slide shows Config A vs. Config B. By changing prompts or models, you can track if your “Equivalence Score” goes up or down.

This treats GenAI engineering like traditional hyperparameter tuning. The goal is to maximize the equivalence score on your custom dataset.

54. Why Global Metrics Aren’t Enough

(Timestamp: 20:28)

The slide discusses the limitations of the “Equivalence” approach. While good for a general sense of quality, Global Metrics miss nuances.

Sometimes it’s hard to get a Gold Answer for open-ended creative tasks. Furthermore, a simple “Pass/Fail” doesn’t tell you why the model failed (e.g., was it tone, length, or factuality?).

55. From Global to Targeted Evaluation

(Timestamp: 20:55)

This slide argues for Targeted Evaluation. To maximize performance, you need to dig deeper into the data and identify specific error modes.

This transitions the talk from “Basic Workflow” to “Advanced Testing,” where we break down “Quality” into specific, testable components like tone, length, and safety.

56. Building Tests

(Timestamp: 21:14)

The section title “Building Tests” appears. This is where the presentation moves into the “Unit Testing” philosophy for GenAI.

Just as software engineering relies on unit tests to verify specific functions, GenAI engineering should use targeted tests to verify specific attributes of the generated text.

57. Good vs. Bad Examples

(Timestamp: 21:20)

The slide displays a Good Example and a Bad Example of a response. The bad example is visibly shorter and less polite.

Rajiv asks the audience to identify why it is bad. This exercise is crucial: you cannot build a test until you can articulate exactly what makes a response a failure.

58. Develop an Evaluation Mindset

(Timestamp: 21:46)

To define “Bad,” developers need an Evaluation Mindset. This involves observing real-world user interactions and problems.

Data scientists often want to stay in their “chair” and optimize algorithms, but Rajiv argues that effective evaluation requires understanding the user’s pain points.

59. Collaborate with Experts

(Timestamp: 21:58)

The slide stresses Collaboration. You must talk to domain experts (e.g., the customer support team) to define what a “good” answer looks like.

Naive bootstrapping—pretending to be a user—is a good start, but long-term success requires input from the people who actually know the business domain.

60. Identify and Categorize Failures

(Timestamp: 22:52)

Once you understand the domain, you can Categorize Failure Types. The slide shows a chart grouping errors into categories like “Harmful Content,” “Bias,” or “Incorrect Info.”

This clustering allows you to see patterns. Instead of just knowing “the model failed 20% of the time,” you know “the model has a specific problem with tone.”

61. Define What Good Looks Like

(Timestamp: 23:11)

Using the categorization, you can explicitly Define What Good Looks Like. The slide contrasts the good/bad examples again, but now with labels: “Too short,” “Lacks professional tone.”

This transforms a subjective feeling (“this response sucks”) into objective criteria (“response must be >50 words and use polite honorifics”).

62. Document Every Issue

(Timestamp: 23:32)

The slide shows a spreadsheet where humans evaluate responses and Document Every Issue. Columns track specific attributes like “Is it helpful?” or “Is the tone right?”

This manual annotation is the training data for your automated tests. You need humans to establish the ground truth before you can automate the checking.

63. Evaluation Tooling

(Timestamp: 23:53)

Rajiv mentions that Tooling Can Help. The slide shows a custom chat viewer designed to make human review easier.

However, he warns against getting sidetracked by building fancy tools. Simple spreadsheets often suffice for the early stages. The goal is the data, not the interface.

64. Test 1: Length Check

(Timestamp: 24:05)

Now we build the automated tests. Test 1 is a Length Check. The slide shows Python code asserting that the word count is between 8 and 200.

This is a deterministic test. You don’t need an LLM to count words. Rajiv encourages using simple Python assertions wherever possible because they are fast, cheap, and reliable.

65. Test 2: Tone and Style

(Timestamp: 24:22)

Test 2 checks Tone and Style. Since “tone” is subjective, we use an LLM Judge (OpenAI model) to classify the response.

The prompt asks the judge to identify the style. This allows us to automate the “vibe check” that humans were previously doing manually.

66. Adding Metrics to Documentation

(Timestamp: 24:41)

The spreadsheet is updated with new columns: Length_OK and Tone_OK. These are the results of the automated tests.

Now, for every row in the dataset, we have granular pass/fail metrics. This helps pinpoint exactly why a specific response failed, rather than just a generic failure.

67. Check Judges Against Humans

(Timestamp: 25:12)

A critical step: Check LLM Judges Against Humans. You must verify that your automated “Tone Judge” agrees with your human experts.

If the human says the tone is rude, but the LLM Judge says it’s polite, your metric is useless. You must iterate on the judge’s prompt until alignment is high.

68. Self-Evaluation Bias

(Timestamp: 26:06)

The slide illustrates Self-Evaluation Bias. LLMs tend to rate their own outputs higher than outputs from other models. GPT-4 prefers GPT-4 text.

To mitigate this, Rajiv suggests mixing models—use Claude to judge GPT-4, or Gemini to judge Claude. This helps ensure a more neutral evaluation.

69. Alignment Checks

(Timestamp: 26:46)

This slide reinforces the need for Continuous Alignment. Just because your judge aligned with humans last month doesn’t mean it still does (due to model drift).

Human spot-checks should be a permanent part of the pipeline to ensure the automated judges haven’t drifted.

70. Biases in LLM Judges

(Timestamp: 27:02)

The slide lists known Biases in LLM Judges, such as Position Bias (favoring the first answer presented) or Verbosity Bias (favoring longer answers).

Evaluators must be aware of these. For example, you should shuffle the order of answers when asking a judge to compare two options to cancel out position bias.

71. Best Practices for LLM Judges

(Timestamp: 27:11)

A summary of Best Practices: Calibrate with human data, use ensembles (multiple judges), avoid asking for “relevance” (too vague), and use discrete rating scales (1-5) rather than continuous numbers.

These tips help stabilize the inherently noisy process of using AI to evaluate AI.

72. Error Analysis Chart

(Timestamp: 27:46)

With tests in place, we move to Error Analysis. The bar chart shows the number of failed cases categorized by error type (Length, Tone, Professional, Context).

This visualization tells you where to focus your efforts. If “Tone” is the biggest bar, you work on the system prompt’s tone instructions. If “Context” is the issue, you might need better Retrieval Augmented Generation (RAG).

73. Comparing Prompts

(Timestamp: 27:58)

The chart can compare Prompt A vs. Prompt B. This allows for A/B testing of prompt engineering strategies.

You can see if a new prompt improves “Tone” but accidentally degrades “Context.” This tradeoff analysis is impossible with a single global score.

74. Explanations Guide Improvement

(Timestamp: 28:14)

Rajiv suggests asking the LLM Judge for Explanations. Don’t just ask for a score; ask for “one sentence explaining why.”

These explanations act as metadata that helps developers understand the judge’s reasoning, making it easier to debug discrepancies between human and AI judgments.

75. Limits to Explanations

(Timestamp: 28:35)

A warning: Explanations are not causal. When an LLM explains why it did something, it is generating a plausible justification, not a trace of its actual neural activations.

Treat explanations as a heuristic or a helpful hint, not as absolute truth about the model’s internal state.

76. The Evaluation Flywheel

(Timestamp: 28:46)

The Evaluation Flywheel describes the iterative cycle: Build Eval -> Analyze -> Improve -> Repeat.

This concept, credited to Hamill, emphasizes that evaluation is not a one-time event but a continuous loop that spins faster as you gather more data and build better tests.

77. Financial Analyst Agent Example

(Timestamp: 29:20)

To demonstrate advanced unit testing, Rajiv introduces a Financial Analyst Agent. The goal is to assess the specific “style” of a financial report.

This is a complex domain where “good” is highly specific (regulated, precise, risk-aware), making it a perfect candidate for granular unit tests.

78. Use a Global Test?

(Timestamp: 29:43)

You could use a Global Test: “Was this explained as a financial analyst would?”

While simple, this test is opaque. If it fails, you don’t know if it was because of compliance issues, lack of clarity, or poor formatting.

79. Global vs. Unit Tests

(Timestamp: 29:54)

The slide contrasts the Global approach with Unit Tests. Instead of one question, we ask six: Context, Clarity, Precision, Compliance, Actionability, and Risks.

This breakdown allows for targeted debugging. You might find the model is great at “Clarity” but terrible at “Compliance.”

80. Scoring Radar Chart

(Timestamp: 30:16)

A Radar Chart visualizes the unit test scores. This allows for a quick visual assessment of the model’s profile.

It facilitates comparison: you can overlay the profiles of two different models to see which one has the better balance of attributes for your specific needs.

81. Analyzing Failures with Clusters

(Timestamp: 30:37)

With enough unit test data, you can use Clustering (e.g., K-Means) to group failures. The slide shows clusters like “Synthesis,” “Context,” and “Hallucination.”

This moves error analysis from reading individual logs to analyzing aggregate trends, helping you prioritize which class of errors to fix first.

82. Designing Good Unit Tests

(Timestamp: 30:52)

Advice on Designing Unit Tests: Keep them focused (one concept per test), use unambiguous language, and use small rating ranges.

Good unit tests are the building blocks of a reliable evaluation pipeline. If the tests themselves are noisy or vague, the entire system collapses.

83. Examples of Unit Tests

(Timestamp: 30:55)

The slide lists specific examples of tests for Legal (Compliance, Terminology), Retrieval (Relevance, Completeness), and Bias/Fairness.

This serves as a menu of options for the audience, showing that unit tests can cover almost any dimension of quality required by the business.

84. Evaluating New Prompts

(Timestamp: 30:58)

A bar chart shows how unit tests are used to Evaluate New Prompts. By running the full suite of unit tests on a new prompt, you get a “scorecard” of its performance.

This data-driven approach removes the guesswork from prompt engineering.

85. Tools - No Silver Bullet

(Timestamp: 31:02)

Rajiv reminds the audience that Tools are No Silver Bullet. You must master the basics (datasets, metrics) first.

He advises logging traces and experiments and practicing Dataset Versioning. Tools facilitate these practices, but they cannot replace the fundamental engineering discipline.

86. Forest and Trees

(Timestamp: 31:04)

An analogy helps structure the analysis: Forest (Global/Integration) vs. Trees (Test Case/Unit Tests).

You need to look at both. The forest tells you the overall health of the app, while the trees tell you specifically what needs pruning or fixing.

87. Change One Thing at a Time

(Timestamp: 31:17)

A crucial scientific principle: Change One Thing at a Time. With so many knobs (prompt, temp, model, RAG settings), changing multiple variables simultaneously makes it impossible to know what caused the improvement (or regression).

Isolate your variables to conduct valid experiments.

88. Error Analysis Tips

(Timestamp: 31:32)

A summary of Error Analysis Tips: Use ablation studies (removing parts to see impact), categorize failures, save interesting examples, and leverage logs/traces.

These are the daily habits of successful GenAI engineers.

89. The Evaluation Story

(Timestamp: 32:08)

The slide shows the “Story We Tell”—a linear graph of improvement over time. This is the idealized version of progress often presented in case studies.

It suggests a smooth journey from “Out of the box” to “Specialized” to “User Feedback.”

90. The Reality of Progress

(Timestamp: 32:24)

The Reality is a messy, non-linear graph. You take two steps forward, one step back. Sometimes an “improvement” breaks the model.

Rajiv encourages resilience. Experienced practitioners know that this messy graph is normal and that sticking to the process eventually yields results.

91. Continual Process

(Timestamp: 33:01)

Evaluation is a Continual Process. It involves Problem ID, Data Collection, Optimization, User Acceptance Testing (UAT), and Updates.

Crucially, UAT is your holdout set. Since you don’t have a traditional test set in GenAI, your real users act as the final validation layer.

92. Eating the Elephant

(Timestamp: 34:03)

The metaphor “How do you eat an elephant?” addresses the overwhelming nature of building a comprehensive evaluation suite.

The answer, of course, is “one bite at a time.” You don’t need 100 tests on day one.

93. Adding Tests Over Time

(Timestamp: 34:10)

The slide visualizes the “elephant” being broken down into bites. You start with a few critical tests. As the app matures and you discover new failure modes, you add more tests.

Six months in, you might have 100 tests, but you built them incrementally. This makes the task manageable.

94. Doing Evaluation the Right Way

(Timestamp: 34:39)

A summary slide listing best practices: Annotated Examples, Systematic Documentation, Continuous Error Analysis, Collaboration, and awareness of Generalization.

This concludes the core methodology section of the talk.

95. Agentic Use Cases

(Timestamp: 34:50)

The final section covers Agentic Use Cases, symbolized by a dragon. Agents add a layer of complexity because the model is now making decisions (routing, tool use) rather than just generating text.

This “agency” makes the system harder to track and evaluate.

96. Crossing the River

(Timestamp: 35:06)

A conceptual slide asking, “How should it cross the river?” (Fly, Swim, Bridge?). This represents the decision-making step in an agent.

Evaluating an agent requires evaluating how it made the decision (the router) separately from how well it executed the action.

97. Chat-to-Purchase Router

(Timestamp: 35:22)

A complex flowchart shows a Chat-to-Purchase Router. The agent must decide if the user wants to search for a product, get support, or track a package.

Rajiv suggests breaking this down: evaluate the Router component first (did it pick the right path?), then evaluate the specific workflow (did it track the package correctly?).

98. Text to SQL Agent

(Timestamp: 36:17)

Another example: Text to SQL Agent. This workflow involves classification, feature extraction, and SQL generation.

You can isolate the “Classification” step (is this a valid SQL question?) and build a test just for that, before testing the actual SQL generation.

99. Evaluating Office-Style Agents

(Timestamp: 36:46)

The slide discusses OdysseyBench, a benchmark for office tasks. It highlights failure modes like “Failed to create folder” or “Failed to use tool.”

Evaluating agents involves checking if they successfully manipulated the environment (files, APIs), which is a functional test rather than a text similarity test.

100. Error Analysis for Agents

(Timestamp: 37:00)

Error Analysis for Agentic Workflows requires assessing the overall performance, the routing decisions, and the individual steps.

It is the same “action error analysis” process but applied recursively to every node in the agent’s decision tree.

101. Evaluating Workflow vs. Response

(Timestamp: 37:19)

This slide distinguishes between evaluating a Response (text) and a Workflow (process). The flowchart shows a conversational flow.

Evaluating a workflow might mean checking if the agent successfully moved the user from “Greeting” to “Resolution,” regardless of the exact words used.

102. Agentic Frameworks

(Timestamp: 37:48)

Rajiv warns that “Agentic Frameworks Help – Until They Don’t.” Frameworks (like LangChain or AutoGen) are great for demos because they abstract complexity.

However, in production, these abstractions can break or become outdated. He often recommends using straight Python for production agents to maintain control and reliability.

103. Abstraction for Workflows

(Timestamp: 38:32)

The slide illustrates the trade-off in Abstraction. You can build rigid workflows (orchestration) where you control every step, or use general agents where the LLM decides.

Orchestration is more reliable but rigid. General agents are flexible but prone to non-deterministic errors.

104. When Abstractions Break

(Timestamp: 38:53)

Model providers are training models to handle workflows internally (removing the need for external orchestration).

However, until models are perfect, developers often need to break tasks down into specific pieces to ensure reliability. The choice between “letting the model do it” and “scripting the flow” depends on the application’s risk tolerance.

105. Lessons from Agent Benchmarks

(Timestamp: 39:15)

The slide lists Lessons from Reproducing Agent Benchmarks: Standardize evaluation, measure efficiency, detect shortcuts, and log real behavior.

These are advanced tips for those pushing the boundaries of what agents can do.

106. Conclusion

(Timestamp: 39:27)

The final slide, “We did it!”, concludes the presentation. Rajiv thanks the audience and provides the QR code again.

His final message is one of empowerment: he hopes the audience now has the confidence to go out, build their own evaluation datasets, and start “hill climbing” their own applications.

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

1. Title Slide

(Timestamp: 00:00)

The presentation begins with the title slide, introducing the core theme: “From Vectors to Agents: Managing RAG in an Agentic World.” The speaker, Rajiv Shah from Contextual, sets the stage for a technical deep dive into Retrieval-Augmented Generation (RAG).

He outlines the agenda, promising to move beyond basic RAG concepts to focus specifically on retrieval approaches. The talk is designed to cover the spectrum from traditional methods like BM25 and Language Models to the emerging field of Agentic Search.

2. ACME GPT

(Timestamp: 00:40)

This slide displays a stylized logo for “ACME GPT,” representing the typical enterprise aspiration. Companies see tools like ChatGPT and immediately want to apply that capability to their internal data, asking questions like, “Can I get the list of board of directors?”

However, the speaker notes a common hurdle: generic models don’t know enterprise-specific knowledge. This sets up the necessity for RAG—injecting private data into the model—rather than relying solely on the model’s pre-trained knowledge.

3. Building RAG is Easy

(Timestamp: 01:10)

The speaker illustrates the deceptively simple workflow of a basic RAG demo. The diagram shows the standard path: a user query is converted to vectors, matched against a database, and sent to an LLM.

Shah acknowledges that building a “hello world” version of this is trivial. He notes, “You can build a very easy RAG demo out of the box by just grabbing some data, using an embedding model, creating vectors, doing the similarity.”

4. Building RAG is Easy (Code Example)

(Timestamp: 01:22)

A Python code snippet using LangChain is displayed to reinforce how accessible basic RAG has become. The code demonstrates loading a document, chunking it, and setting up a retrieval chain in just a few lines.

This slide serves as a foil for the upcoming reality check. While the code works for a demo, it hides the immense complexity required to make such a system robust, accurate, and scalable in a real-world production environment.

5. RAG Reality Check

(Timestamp: 01:35)

The tone shifts to the challenges of production. The slide highlights a sobering statistic: 95% of Gen AI projects fail to reach production. The speaker details the specific reasons why demos fail when scaled: poor accuracy, unbearable latency, scaling issues with millions of documents, and ballooning costs.

He emphasizes a critical, often overlooked factor: Compliance. “Inside an enterprise, not everybody gets to read every document.” A demo ignores entitlements, but a production system cannot.

6. Maybe try a different RAG?

(Timestamp: 03:00)

This slide lists a dizzying array of RAG variants (GraphRAG, RAPTOR, CRAG, etc.) and retrieval techniques. It represents the “analysis paralysis” developers face when scouring arXiv papers for a solution to their accuracy problems.

Shah warns against blindly chasing the latest academic paper to fix fundamental system issues. “The answer is not in here of pulling together like a bunch of archive papers.” Instead, he advocates for a structured framework to make decisions.

7. Ultimate RAG Solution

(Timestamp: 03:30)

A humorous cartoon depicts a “Rube Goldberg” machine, representing the “Ultimate RAG Solution.” It mocks the tendency to over-engineer systems with too many interconnected, fragile components in the pursuit of performance.

The speaker uses this visual to argue for simplicity and deliberate design. The goal is to avoid building a monstrosity that is impossible to maintain, urging the audience to think about trade-offs before complexity.

8. RAG as a system

(Timestamp: 03:35)

The speaker introduces a clean system architecture for RAG, broken into four distinct stages: Parsing, Querying, Retrieving, and Generation. This framework serves as the mental map for the rest of the presentation.

He highlights that “Parsing” is vastly overlooked—getting information out of complex documents cleanly is a prerequisite for success. Today’s talk, however, will zoom in specifically on the Retrieving and Querying components.

9. Designing a RAG Solution

(Timestamp: 04:10)

This slide presents a “Tradeoff Triangle” for RAG, balancing Problem Complexity, Latency, and Cost. The speaker advises having a serious conversation with stakeholders about these constraints before writing code.

A key concept introduced here is the “Cost of a mistake.” In coding assistants, a mistake is low-cost (the developer fixes it). In medical RAG systems, the cost of a mistake is high (life or death), which dictates a completely different architectural approach.

10. RAG Considerations

(Timestamp: 05:30)

A detailed table breaks down specific considerations that influence RAG design, such as domain difficulty, multilingual requirements, and data quality. This slide was originally created for sales teams to help scope customer problems.

Shah emphasizes that understanding the nuances of the use case upfront saves heartache later. For instance, knowing if users will ask simple questions or require complex reasoning changes the retrieval strategy entirely.

11. Consider Query Complexity

(Timestamp: 06:15)

The speaker categorizes queries by complexity, ranging from simple Keywords (“Total Revenue”) to Semantic variations (“How much bank?”), to Multi-hop reasoning, and finally Agentic scenarios.

He points out a common failure mode: “The answers aren’t in the documents… all of a sudden they’re asking for knowledge that’s outside.” Recognizing the query complexity determines whether you need a simple search engine or a complex agentic workflow.

12. Retrieval (Highlighted)

(Timestamp: 07:32)

The presentation zooms back into the system diagram, highlighting the “Retrieving” box. This signals the start of the deep technical dive into retrieval algorithms.

Shah notes that this area causes the most confusion due to the sheer number of model choices and architectures available. He aims to provide a practical guide to selecting the right retrieval tool.

13. Retrieval Approaches

(Timestamp: 08:16)

Three primary retrieval pillars are introduced: 1. BM25: The lexical, keyword-based standard. 2. Language Models: Semantic embeddings and vector search. 3. Agentic Search: The new frontier of iterative reasoning.

The speaker emphasizes that documents must be broken into pieces (chunking) because no single model context window is efficient enough to hold all enterprise data for every query.

14. Building RAG is Easy (Code Highlight)

(Timestamp: 08:50)

Returning to the initial code snippet, the speaker highlights the vectorstore and retriever initialization lines. This pinpoints exactly where the upcoming concepts fit into the implementation.

This visual anchor helps developers map the theoretical concepts of BM25 and Embeddings back to the actual lines of code they write in libraries like LangChain or LlamaIndex.

15. BM25

(Timestamp: 09:18)

BM25 (Best Match 25) is explained as a probabilistic lexical ranking function. The slide visualizes an inverted index, mapping words (like “butterfly”) to the specific documents containing them.

Shah explains that this is the 25th iteration of the formula, designed to score documents based on word frequency and saturation. It remains a powerful, fast baseline for retrieval.

16. BM25 Performance

(Timestamp: 09:55)

A table compares the speed of a Linear Scan (Ctrl+F style) versus an Inverted Index (BM25) as the document count grows from 1,000 to 9,000.

The data shows that linear search becomes exponentially slower (taking 3,000 seconds for 1k documents in this synthetic test), while BM25 remains orders of magnitude faster. This efficiency is why lexical search is still widely used in production.

17. BM25 Failure Cases

(Timestamp: 11:08)

The limitations of BM25 are exposed. Because it relies on exact word matches, it fails when users use synonyms. If a user searches for “Physician” but the documents only contain “Doctor,” BM25 will return zero results.

Similarly, it struggles with acronyms like “IBM” vs “International Business Machines.” Despite this, Shah argues BM25 is a “very strong baseline” that often beats complex neural models on specific keyword-heavy datasets.

18. Hands on: BM25s

(Timestamp: 12:14)

For developers wanting to implement this, the slide points to a library called bm25s, a high-performance Python implementation available on Hugging Face.

This reinforces the practical nature of the talk—BM25 isn’t just a legacy concept; it is an active, installable tool that developers should consider using alongside vector search.

19. Enter Language Models

(Timestamp: 12:24)

The talk transitions to Language Models (Embeddings). The slide explains how an encoder model turns text into a dense vector (a list of numbers) that captures semantic meaning.

Because these models are trained on vast amounts of data, they “have an idea of these similar concepts.” This solves the synonym problem that plagues BM25.

20. Embeddings Visualized

(Timestamp: 12:50)

A 2D visualization demonstrates how embeddings group related concepts in latent space. The word “Doctor” and “Physician” would be located very close to each other mathematically.

This spatial proximity allows for Semantic Search: finding documents that mean the same thing as the query, even if they don’t share a single word.

21. Semantic search is widely used

(Timestamp: 13:15)

The speaker validates the importance of semantic search by showing a tweet from Google’s SearchLiaison regarding BERT, and a screenshot of Hugging Face’s model repository.

This confirms that semantic search is the industry standard for modern information retrieval, having been deployed at massive scale by tech giants to improve result relevance.

22. Which language model?

(Timestamp: 13:30)

A scatter plot compares various models based on Inference Speed (X-axis) and NDCG@10 (Y-axis, a measure of retrieval quality).

Shah places BM25 on the right (fast but lower accuracy) to orient the audience. He points out that there is a massive variety of models with different trade-offs between compute cost and retrieval quality.

23. Static Embeddings

(Timestamp: 14:43)

The speaker introduces Static Embeddings (like Word2Vec or GloVe) which are located on the far right of the previous scatter plot—extremely fast, even on CPUs.

These models assign a fixed vector to every word. While efficient, they lack context. The word “bank” has the same vector whether referring to a river bank or a financial bank, which limits their accuracy.

24. Why Context Matters

(Timestamp: 15:16)

A cartoon illustrates the difference between Static Embeddings and Transformers. The Transformer can distinguish between “Model” in a data science context versus “Model” in a fashion context.

This contextual awareness is why modern Transformer-based embeddings (like BERT) generally outperform static embeddings and BM25 in complex retrieval tasks, despite being slower.

25. Many more models!

(Timestamp: 15:55)

Returning to the scatter plot, a red arrow points toward the top-left quadrant—models that are slower but achieve higher accuracy.

The speaker notes that the field is constantly evolving, with “newer generations of models” pushing the boundary of what is possible in terms of retrieval quality.

26. MTEB/RTEB

(Timestamp: 16:35)

To help developers choose, Shah introduces the MTEB (Massive Text Embedding Benchmark) and RTEB (Retrieval Text Embedding Benchmark). These are leaderboards hosted on Hugging Face.

He highlights a key distinction: MTEB uses public datasets, while RTEB uses private, held-out datasets. This is crucial for avoiding “data contamination,” where models perform well simply because they were trained on the test data.

27. Selecting an embedding model

(Timestamp: 16:48)

The speaker switches to a live browser view (captured in the slide) of the leaderboard. He discusses the bubble chart visualization where size often correlates with parameter count.

He points out an interesting trend: “You’ll see that there’s a bunch of models here that are all the same size… but the performance differs.” This indicates improvements in training strategies and architecture rather than just throwing more compute at the problem.

28. Selecting an embedding model (Other Considerations)

(Timestamp: 19:07)

Beyond the leaderboard score, Shah lists practical selection criteria: Model Size (can it fit in memory?), Architecture (CPU vs GPU), Embedding Dimension (storage costs), and Training Data (multilingual support).

He advises checking if a model is open source and quantizable, as this can significantly reduce latency without a major hit to accuracy.

29. Matryoshka Embedding Models

(Timestamp: 20:53)

A specific innovation is highlighted: Matryoshka Embeddings. These models allow developers to truncate vectors (e.g., from 768 dimensions down to 64) while retaining most of the performance.

This is a “neat kind of innovation” for optimizing storage and search speed. OpenAI’s newer models also support this feature, offering flexibility between cost and accuracy.

30. Sentence Transformer

(Timestamp: 21:42)

The Sentence Transformer architecture is described as the dominant approach for RAG. Unlike standard BERT which works on tokens, these are fine-tuned to understand full sentences and paragraphs.

This architecture uses Siamese networks to ensure that semantically similar sentences are close in vector space, making them ideal for the “chunk-level” retrieval required in RAG.

31. Cross Encoder / Reranker

(Timestamp: 22:16)

The concept of a Cross Encoder (or Reranker) is introduced. Unlike the bi-encoder (retriever) which processes query and document separately, the cross-encoder processes them together.

This allows for a much deeper calculation of relevance. It is typically used as a second stage: retrieve 50 documents quickly with vectors, then use the slow but accurate Cross Encoder to rank the top 5.

32. Cross Encoder / Reranker (Duplicate)

(Timestamp: 22:16)

(This slide reinforces the previous diagram, emphasizing the “crossing” of the query and document in the model architecture.)

33. Cross Encoder / Reranker (Accuracy Boost)

(Timestamp: 23:07)

A bar chart quantifies the value of reranking. It shows a significant boost in NDCG (accuracy) when a reranker is added to the pipeline.

The speaker notes that while you get a “bump” in quality, it “doesn’t come for free.” The trade-off is increased latency, as the cross-encoder is computationally expensive.

34. Cross Encoder / Reranker (Execution Flow)

(Timestamp: 23:15)

The execution flow diagram highlights the reranker’s position in the pipeline. It sits between the Vector Store retrieval and the LLM generation.

This visual reinforces the latency implication: the user has to wait for both the initial search and the reranking pass before the LLM even starts generating an answer.

35. Hands On: Retriever & Reranker

(Timestamp: 23:30)

A screenshot of a Google Colab notebook is shown, demonstrating a practical implementation of the Retrieve and Re-rank strategy using the SentenceTransformer and CrossEncoder libraries.

This provides a concrete resource for the audience to test the accuracy vs. speed trade-offs themselves on simple datasets like Wikipedia.

36. Instruction Following Reranker

(Timestamp: 23:48)

Shah mentions a specific advancement: Instruction Following Rerankers (developed by his company, Contextual). These allow developers to pass a prompt to the reranker, such as “Prioritize safety notices.”

This adds a “knob” for developers to tune retrieval based on business logic without retraining the model.

37. Combine Multiple Retrievers

(Timestamp: 24:19)

The presentation suggests that you don’t have to pick just one method. You can combine BM25, various embedding models (E5, BGE), and rerankers.

While combining them (Ensemble Retrieval) often yields better recall, Shah warns that “you got to engineer this.” Managing multiple indexes and fusion logic increases operational complexity and compute costs.

38. Cascading Rerankers in Kaggle

(Timestamp: 24:56)

A complex diagram from a Kaggle competition winner illustrates a Cascade Strategy. The solution used three different rerankers, filtering from 64 documents down to 8, and then to 5.

This shows the extreme end of retrieval engineering, where multiple models are chained to squeeze out every percentage point of accuracy.

39. Best practices

(Timestamp: 25:16)

Shah distills the complexity into a recommended Best Practice: 1. Hybrid Search: Combine Semantic Search (Vectors) and Lexical Search (BM25). 2. Reciprocal Rank Fusion: Merge the results. 3. Reranker: Pass the top results through a cross-encoder.

This setup provides a “pretty good standard performance out of the box” and should be the default baseline before trying exotic methods.

40. Families of Embedding Models

(Timestamp: 25:42)

A taxonomy slide categorizes the models discussed: Static (Fastest/Low Accuracy), Bi-Encoders (Fast/Good Accuracy), and Cross-Encoders (Slow/Best Accuracy).

This summary helps the audience mentally organize the tools available in their toolbox.

41. Lots of New Models

(Timestamp: 25:50)

Logos for IBM Granite, Google EmbeddingGemma, and others appear. The speaker notes that while new models from major players appear weekly, the improvements are often “incremental.”

He advises against “ripping up” a working system just to switch to a model that is 1% better on a leaderboard.

42. Other retrieval methods

(Timestamp: 26:18)

Alternative methods are briefly listed: SPLADE (Sparse retrieval), ColBERT (Late interaction), and GraphRAG.

Shah acknowledges these exist and may fit specific niches, but warns against chasing the “flavor of the week” before establishing a solid baseline with hybrid search.

43. Operational Concerns

(Timestamp: 27:30)

The talk shifts to operations. Libraries like FAISS are mentioned for efficient vector similarity search.

A key point is that for many use cases, you can simply store embeddings in memory. You don’t always need a complex vector database if your dataset fits in RAM.

44. Vector Database Options

(Timestamp: 27:55)

A diagram categorizes storage into Hot (In-Memory), Warm (SSD/Disk), and Cold tiers.

Shah notes there are “tons of vector database options” (Snowflake, Pinecone, etc.). The choice should be governed by latency requirements. If you need sub-millisecond retrieval, you need in-memory storage.

45. Operational Concerns (Datastore Size)

(Timestamp: 28:40)

A graph shows that as Datastore Size increases (X-axis), retrieval performance naturally degrades (Y-axis).

To combat this, the speaker strongly recommends using Metadata Filtering. “If you’re not using something like metadata… it’s going to be very tough.” Narrowing the search scope is essential for scaling to millions of documents.

46. Search Strategy Comparison

(Timestamp: 29:22)

The presentation pivots to the “exciting part”: Agentic RAG. A visual compares “Traditional RAG” (a linear path) with “Agentic RAG” (a winding, exploratory path).

This represents the shift from a “one-shot” retrieval attempt to an iterative system that can explore, backtrack, and reason.

47. Tools use / Reasoning

(Timestamp: 29:40)

Reasoning models (like o1 or DeepSeek R1) enable LLMs to use tools effectively. A code snippet shows an agent loop: query -> generate -> “Did it answer the question?”

If the answer is no, the model can “rewrite the query… try to find that missing information, feed that back into the loop.” This self-correction is the core of Agentic RAG.

48. Agentic RAG (Workflow)

(Timestamp: 30:32)

A flowchart details the Agentic RAG lifecycle. The model thinks through steps: “Oh, this is the query I need to make… based on those results… maybe we should do it a different way.”

This workflow allows the system to synthesize answers from multiple sources or clarify ambiguous queries automatically.

49. Tools use / Reasoning (Detailed Example)

(Timestamp: 30:35)

A specific example of a complex query is shown. The agent breaks the problem down, calls tools, and iterates.

This demonstrates that the “Thinking” time is where the value is generated, allowing for a depth of research that a single retrieval pass cannot match.

50. Open Deep Research

(Timestamp: 31:02)

Shah references “Open Deep Research” by LangChain, an open-source framework where sub-agents go out, perform research, and report back.

This is a specific category of Agentic RAG focused on generating comprehensive reports rather than quick answers.

51. DeepResearch Bench

(Timestamp: 31:30)

A leaderboard for DeepResearch Bench is shown, testing models on “100 PhD level research tasks.”

The speaker warns that this approach “can get very expensive.” Solving a single complex query might cost significant money due to the number of tokens and iterative steps required.

52. Westlaw AI Deep Research

(Timestamp: 31:55)

A real-world application is highlighted: Westlaw AI. In the legal field, thoroughness is worth the latency and cost.

This proves that Agentic RAG isn’t just a toy; it is being commercialized in high-value verticals where accuracy is paramount.

53. Agentic RAG (Self-RAG)

(Timestamp: 32:11)

The concept of Self-RAG is introduced, emphasizing the “Reflection” step. The model critiques its own retrieved documents and generation quality.

Shah notes that this isn’t brand new, but has become practical due to better reasoning models.

54. Agentic RAG (LangChain Reddit)

(Timestamp: 34:04)

A Reddit post is shown where a developer discusses building a self-reflection RAG system. This highlights the community’s active experimentation with these loops.

55. Agentic RAG (Efficiency Concerns)

(Timestamp: 34:15)

The discussion turns to the “Rub”: Inefficiency. Agentic loops can be slow and wasteful, re-retrieving data unnecessarily.

This sets up the trade-off conversation again: Is the extra time and compute worth the accuracy gain?

56. Research: BRIGHT

(Timestamp: 32:11)

Note: The speaker introduces the BRIGHT benchmark around 32:11, slightly out of slide order in the transcript flow, but connects it here.

BRIGHT is a benchmark specifically designed for Retrieval Reasoning. Unlike standard benchmarks that test keyword matching, BRIGHT tests questions that require thinking, logic, and multi-step deduction to find the correct document.

57. BRIGHT #1: DIVER

(Timestamp: 32:48)

The top-performing system on BRIGHT is DIVER. The diagram shows it uses the exact components discussed earlier: Chunking, Retrieving, and Reranking, but wrapped in an iterative loop.

Shah points out, “It probably doesn’t look that crazy to you if you’re used to RAG.” The innovation is in the process, not necessarily a magical new model architecture.

58. BRIGHT #1: DIVER (LLM Instructions)

(Timestamp: 33:31)

The specific prompts used in DIVER are shown. The system asks the LLM: “Given a query… what do you think would be possibly helpful to do?”

This Query Expansion allows the system to generate new search terms that the user didn’t think of, bridging the semantic gap through reasoning.

59. Agentic RAG on WixQA

(Timestamp: 34:36)

Shah shares his own experiment results on the WixQA dataset (technical support). * One Shot RAG: 5 seconds latency, 76% Factuality. * Agentic RAG: Slower latency, 93% Factuality.

This massive jump in accuracy (0.76 to 0.93) is the key takeaway. “That has a ton of implications.” It suggests that the limitation of RAG often isn’t the data, but the lack of reasoning applied to the retrieval process.

60. Rethink your Assumptions

(Timestamp: 37:10)

This is the climax of the technical argument. A graph from the BRIGHT paper shows that BM25 (lexical search) combined with an Agentic loop (GPT-4) outperforms advanced embedding models (Qwen).

“This is crazy,” Shah exclaims. Because the LLM can rewrite queries into many variations, it mitigates BM25’s weakness (synonyms). This implies you might not need complex vector databases if you have a smart agent.

61. Agentic RAG with BM25

(Timestamp: 38:20)

Shah validates the paper’s finding with his own internal data (Financial 10Ks). Agentic RAG with BM25 performed nearly as well as Agentic RAG with Embeddings.

He suggests a radical possibility: “I could throw all that away [vector DBs]… just stick this in a text-only database and use BM25.”

62. Agentic RAG for Code Search

(Timestamp: 39:46)

He connects this finding to Claude Code, which uses a lexical approach (like grep) rather than vectors for code search.

Since code doesn’t have the same semantic ambiguity as natural language, and agents can iterate rapidly, lexical search is proving to be superior for coding assistants.

63. Combine Retrieval Approaches

(Timestamp: 40:15)

A DoorDash case study illustrates a two-tier guardrail system. They use simple text similarity first (fast/cheap). If that fails or is uncertain, they kick it to an LLM (slow/expensive).

This “Tiered” approach optimizes the trade-off between cost and accuracy in production.

64. Hands on: Agentic RAG (Smolagents)

(Timestamp: 41:07)

The speaker points to Smolagents, a Hugging Face library, as a way to get hands-on with these concepts. A Colab notebook is provided for the audience to build their own agentic retrieval loops.

65. Solutions for a RAG Solution

(Timestamp: 41:18)

Shah updates the “Problem Complexity” framework from the beginning of the talk with specific recommendations: * Low Latency (<5s): Use BM25 or Static Embeddings. * High Cost of Mistake: Add a Reranker. * Complex Multi-hop: Use Agentic RAG.

66. Retriever Checklist

(Timestamp: 41:52)

A final checklist summarizes the retrieval hierarchy: 1. Keyword/BM25 (The baseline). 2. Semantic Search (The standard). 3. Agentic/Reasoning (The problem solver).

This provides the audience with a mental menu to choose from based on their specific constraints.

67. RAG as a system (Retrieval with Instruction Following Reranker)

(Timestamp: 42:00)

The system diagram is shown one last time, updated to include the Instruction Following Reranker in the retrieval box, solidifying the modern RAG architecture.

68. RAG - Generation

(Timestamp: 42:10)

Note: The speaker concludes the talk at 42:10, stating “I’m going to end it here.” Slides 68-70 regarding the Generation stage were included in the deck but skipped in the video recording due to time constraints.

This slide would have covered the final stage of RAG: generating the answer. The focus here is typically on reducing hallucinations and ensuring the tone matches the user’s needs.

69. RAG - Generation (Model Selection)

(Timestamp: 42:10)

Skipped in video. This slide illustrates the choice of LLM for generation (e.g., GPT-4 vs Llama 3 vs Claude). The choice depends on the “Cost/Latency budget” and specific domain requirements.

70. Chunking approaches

(Timestamp: 42:10)

Skipped in video. This slide compares Original Chunking (cutting text at fixed intervals) with Contextual Chunking (adding a summary prefix to every chunk). Contextual chunking significantly improves retrieval because every chunk carries the context of the parent document.

71. Title Slide (Duplicate)

(Timestamp: 42:10)

The presentation concludes with the title slide. Rajiv Shah thanks the audience, encouraging them to think about trade-offs rather than just chasing the latest models. “Hopefully I’ve given you a sense of thinking about these trade-offs… thank you all.”

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

A. MNIST (Very visual dataset)

from keras.datasets import mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 28 * 28)
X_test = X_test.reshape(10000, 28 * 28)

print(list(X_train[7, :]))

plt.imshow(X_train[7, :].reshape(28, 28), cmap = 'binary', vmin = 0, vmax = 255)
plt.xticks([])
plt.yticks([])
plt.savefig('sample_image.png')

X_train = pd.DataFrame(X_train)  # Assuming X_mnist is the MNIST feature data
y_train = pd.Series(y_train)   
X_test = pd.DataFrame(X_test)  # Assuming X_mnist is the MNIST feature data
y_test = pd.Series(y_test)  

B. Madelon (Very high-dimensional dataset that you control)

If you run the following cells, Madelon will be the dataset you use. If you want to use MNIST, you should skip the following cells.

Madelon is a favorite of mine because you know which features are carrying the signal and which ones are noise. In this case, the first 5 features will be informative. I often modify Madelon to include other types of noisy features, interactions, correlations, and then use this dataset to test various machine learning techniques. Since I know what the true signal is, this is very effective at helping me guage the effectiveness of these methods.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=10000,
                           n_features=40,
                           n_informative=5,
                           n_classes=3,
                           n_redundant = 3,
                           random_state=0,
                           flip_y =0.05,
                           class_sep = 0.5,
                           n_clusters_per_class=3,
                           shuffle=False)

X_df = pd.DataFrame(X, columns=[f'{i}' for i in range(X.shape[1])])
y_df = pd.Series(y, name='target')

X_train, X_test, y_train, y_test = train_test_split(X_df, y_df, test_size=0.2, random_state=42)

X_train.columns = X_train.columns.astype(int)
X_test.columns = X_test.columns.astype(int)

X_train

1. F-statistic

f_classif relies on the Analysis of Variance (ANOVA) F-statistic to evaluate the relationship between each feature and the target variable. It tests whether the mean values of the target variable differ significantly across the groups defined by each feature. The higher the F-value, the more likely it is that the feature discriminates between different classes. Assumes a linear relationship between the features and the target, and that the target is categorical.

See: https://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection

from sklearn.feature_selection import f_classif

f = f_classif(X_train, y_train)[0]
f

array([2.40810399e+02, 6.64390517e+01, 5.42587843e+01, 1.71099993e-01,
       4.27226542e+01, 9.64735668e+01, 4.72136845e+01, 5.71846457e+01,
       1.27834979e+00, 1.42650284e+00, 3.40020422e-01, 4.08232233e-01,
       1.30120819e-01, 3.36734714e+00, 4.55665866e-01, 3.62300510e-01,
       1.57485437e-01, 6.93572687e-02, 1.64816305e+00, 2.91782944e+00,
       1.24026934e+00, 9.32533895e-01, 7.07099908e-01, 1.87544216e+00,
       1.10130690e+00, 3.54044700e-01, 1.15417945e+00, 2.59156089e-01,
       7.45820681e-01, 7.75403854e-01, 1.35835715e-01, 3.34985292e+00,
       8.36576456e-02, 5.15026453e-02, 4.33788709e-01, 3.12140721e-01,
       3.55118575e+00, 7.37076241e+00, 1.17274619e+00, 4.36532461e+00])

2. Mutual information

mutual_info_classif uses the concept of mutual information, which measures the dependency between each feature and the target variable. Mutual information quantifies the amount of information gained about the target by knowing the value of the feature. It captures both linear and non-linear dependencies.

See: https://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection

from sklearn.feature_selection import mutual_info_classif

mi = mutual_info_classif(X_train, y_train)
mi

array([0.05548077, 0.01340894, 0.04092611, 0.00099355, 0.00516085,
       0.06952759, 0.02752171, 0.04043945, 0.00083431, 0.        ,
       0.        , 0.        , 0.        , 0.01146672, 0.        ,
       0.00298292, 0.        , 0.        , 0.0068234 , 0.00961735,
       0.00935105, 0.00586449, 0.00561433, 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.00876386, 0.00049355,
       0.        , 0.00478042, 0.00487523, 0.00268551, 0.00118896,
       0.        , 0.00583264, 0.        , 0.        , 0.        ])

3. Logistic regression

Logistic regression is a linear model for classification rather than regression. It is used to estimate the probability that an instance belongs to a particular class. The coefficients of the model can be used to determine feature importance.

from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression().fit(X_train, y_train)
logreg.coef_

array([[-0.08669758,  0.06064912, -0.04063592,  0.00704782,  0.05079063,
        -0.02366404, -0.03246196, -0.07613473, -0.02216845,  0.00405511,
         0.00890925, -0.01289459, -0.00446435,  0.00330386,  0.01287983,
        -0.00599418,  0.00494212, -0.00385749,  0.03721175, -0.03849129,
        -0.0032749 , -0.01534965, -0.00908255,  0.02016669, -0.00175419,
         0.00918138,  0.01908963,  0.01357562, -0.01804835,  0.00266229,
         0.00180036, -0.00624841, -0.00351875,  0.00131487, -0.01573702,
        -0.00485053, -0.03744854,  0.05047984,  0.0174477 , -0.00658735],
       [ 0.22615114, -0.09497312, -0.04870109,  0.08462927, -0.00434942,
         0.0845374 ,  0.05186658,  0.05312074,  0.01121456,  0.0271935 ,
        -0.00027397,  0.01737956,  0.0080553 , -0.04770429, -0.00379082,
        -0.00963934,  0.00274429,  0.00688572, -0.03159925,  0.02737958,
        -0.0220797 , -0.00789503, -0.01134082, -0.02724488, -0.02432216,
         0.00543336, -0.02498671, -0.00919296,  0.01043275,  0.01323123,
        -0.00059363, -0.02632891, -0.00159478, -0.00753653,  0.01697607,
         0.01247471,  0.04553467, -0.05091659, -0.02028138,  0.03615948],
       [-0.13945356,  0.03432401,  0.089337  , -0.09167709, -0.04644121,
        -0.06087336, -0.01940462,  0.02301399,  0.0109539 , -0.03124862,
        -0.00863527, -0.00448497, -0.00359095,  0.04440043, -0.00908901,
         0.01563352, -0.00768641, -0.00302823, -0.0056125 ,  0.01111171,
         0.02535461,  0.02324468,  0.02042338,  0.0070782 ,  0.02607635,
        -0.01461474,  0.00589708, -0.00438266,  0.0076156 , -0.01589352,
        -0.00120673,  0.03257732,  0.00511353,  0.00622166, -0.00123904,
        -0.00762418, -0.00808614,  0.00043674,  0.00283369, -0.02957212]])

3.5 Feature Selection with L1 (Lasso) Regularization

Lasso is a great feature selection technique. It’s fast, easy to use, and works well with high-dimensional data. I have often used it when very wide data, greater than 100 features (or even >10k features) to help parse down the number of features. It uses L1 regularization to penalize the absolute size of the coefficients. This leads to sparse solutions, where many of the coefficients are zero. The features with non-zero coefficients are selected. Lasso can be used for feature selection by setting the regularization parameter to a value that results in a sparse solution. The regularization parameter can be tuned using cross-validation.

Try modifying the regularization parameter to see how it affects the number of features selected.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

# Step 4: Standardize the features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Step 5: Apply Logistic Regression with L1 regularization for feature selection
logregL1 = LogisticRegression(penalty='l1', solver='saga', multi_class='multinomial', C=0.01)  # C is inverse of regularization strength
logregL1.fit(X_train_scaled, y_train)

# Step 6: Get the selected features using the original DataFrame 'X'
selected_features = X_train.columns[(logregL1.coef_ != 0).any(axis=0)]
print("Selected features: ", selected_features)

# Optional: Check the coefficients
#print("Logistic Regression coefficients: ", logreg.coef_)

Selected features:  Index([0, 1, 2, 4, 5, 13, 19, 36, 37], dtype='int64')

4. LightGBM

LightGBM is a gradient boosting framework that uses tree-based learning algorithms. It is designed for efficiency and can handle large datasets. It can be used to determine feature importance.

from lightgbm import LGBMClassifier

lgbm = LGBMClassifier(
    objective = 'multiclass',
    metric = 'multi_logloss',
    importance_type = 'gain'
).fit(X_train, y_train)

lgbm.feature_importances_

[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.001096 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 10200
[LightGBM] [Info] Number of data points in the train set: 8000, number of used features: 40
[LightGBM] [Info] Start training from score -1.108284
[LightGBM] [Info] Start training from score -1.094371
[LightGBM] [Info] Start training from score -1.093252

array([7030.24782425, 4036.90086633, 7197.39050466, 5339.74033117,
       2017.793881  , 8113.67321557, 3905.26838762, 4383.72206521,
        550.03625131,  531.02187729,  472.12624365,  678.28547454,
        526.57803982,  586.75292325,  552.92263156,  433.08122051,
        552.18078488,  534.15573859,  566.58704376,  630.01932001,
        635.43262064,  636.71719581,  560.95981157,  586.52648336,
        553.7755444 ,  563.13766581,  547.99060541,  523.01072556,
        676.76891661,  616.94216621,  634.27822083,  489.91742009,
        680.71264285,  620.95509708,  618.59545827,  418.22946733,
        568.21738124,  592.29172051,  553.43465978,  655.03435677])

5. Boruta

Boruta is an all-relevant feature selection method. It is an extension of the Random Forest algorithm. It selects all features that are relevant to the target variable, rather than just the most important features.

### long training time > 1 hour
from boruta import BorutaPy
from sklearn.ensemble import RandomForestClassifier

#boruta = BorutaPy(
#    estimator = RandomForestClassifier(max_depth = 5), 
#    n_estimators = 'auto', 
#    max_iter = 100
#).fit(X_train, y_train)

6. MRMR

MRMR (Minimum Redundancy Maximum Relevance) is a feature selection method that selects features based on their relevance to the target variable and their redundancy with other features. It aims to select features that are highly correlated with the target variable but uncorrelated with each other.

There are several implementations of MRMR available in Python: https://github.com/smazzanti/mrmr https://koaning.github.io/scikit-lego/api/feature-selection/ https://github.com/AutoViML/featurewiz?tab=readme-ov-file

import pandas as pd
from mrmr import mrmr_classif

#mrmr = mrmr_classif(pd.DataFrame(X_train), pd.Series(y_train), K = 784)
mrmr = mrmr_classif(pd.DataFrame(X_train), pd.Series(y_train), K = X_train.shape[1])

100%|██████████| 40/40 [00:03<00:00, 11.45it/s]

Store results

import numpy as np

ranking = pd.DataFrame(index = range(X_train.shape[1]))

ranking['f'] = pd.Series(f, index = ranking.index).fillna(0).rank(ascending = False)
ranking['mi'] = pd.Series(mi, index = ranking.index).fillna(0).rank(ascending = False)
ranking['logreg'] = pd.Series(np.abs(logreg.coef_).mean(axis = 0), index = ranking.index).rank(ascending = False)
ranking['lasso']= pd.Series(np.abs(logregL1.coef_).mean(axis = 0), index = ranking.index).rank(ascending = False)
ranking['lightgbm'] = pd.Series(lgbm.feature_importances_, index = ranking.index).rank(ascending = False)
#ranking['boruta'] = boruta.support_* 1 + boruta.support_weak_ * 2 + (1 - boruta.support_ - boruta.support_weak_) * X_train.shape[1]
ranking['mrmr'] = pd.Series(
    list(range(1, len(mrmr) + 1)) + [len(mrmr) + 1] * (X_train.shape[1] - len(mrmr)),
    index = mrmr + list(set(ranking.index) - set(mrmr))
).sort_index()
ranking['lasso']= pd.Series(np.abs(logregL1.coef_).mean(axis = 0), index = ranking.index).rank(ascending = False)


ranking = ranking.replace(to_replace = ranking.max(), value = X_train.shape[1])
#ranking.to_csv('ranking.csv', index = False)

Recursive Feature Elimination

One of the best methods for feature selection consistently is feature importance with LightGBM. We can refine and improve this in several ways: Recursive Feature Elimination uses the same feature importance method, but then iteratively removes the least important features. This iterative process requires training a model several times, but can provide an improvement in feature selection. This method is a version of Recursive Feature Elimination that is widely accepted as a best practice for feature selection.

from sklearn.feature_selection import RFE
from xgboost import XGBClassifier
from sklearn.svm import SVR
model = XGBClassifier(random_state=42)
#model = SVR(kernel="linear")  #took 3 minutes, ok results but not as good as XGB on Madelon
rfe = RFE(model, n_features_to_select=7, step=1)
#rfe = RFE(model, n_features_to_select=50, step=200,verbose=2) #for MNIST
rfe.fit(X_train, y_train)
rfe.support_

array([ True,  True,  True,  True, False,  True,  True,  True, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False])

# Train an XGBoost model with the selected features from RFE
model_selected = XGBClassifier(random_state=42)
X_selected = X_train.loc[:, rfe.support_]
model_selected.fit(X_selected, y_train)

# Make predictions on the test set with both models
y_pred_selected = model_selected.predict(X_test.loc[:, rfe.support_])
accuracy_selected = accuracy_score(y_test, y_pred_selected)
print(f"Accuracy with selected features: {accuracy_selected:.4f}")

Accuracy with selected features: 0.7135

Compare with perfect on Madelon

perfect = [ True,  True,  True,  True, True,  True,  True,  True, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False]

# Train an XGBoost model with the selected features from RFE
model_selected = XGBClassifier(random_state=42)
X_selected = X_train.loc[:, perfect]
model_selected.fit(X_selected, y_train)

# Make predictions on the test set with both models
y_pred_selected = model_selected.predict(X_test.loc[:, perfect])
accuracy_selected = accuracy_score(y_test, y_pred_selected)
print(f"Accuracy with selected features: {accuracy_selected:.4f}")

Accuracy with selected features: 0.7140

Feature Elimination with FIRE

At DataRobot, we had a mighty AutoML engine that showed you how feature importance aggregated across different models (this is feature importance from four diverse models).

You can use this variance as part of feature selection. It takes a lot more compute, but in our experiments, can perform even better feature selection. Read more about feature importance rank ensembling (FIRE) here - https://docs.datarobot.com/en/docs/api/accelerators/adv-approaches/fire.html and a code snippet is here - https://github.com/datarobot-community/examples-for-data-scientists/blob/master/Feature%20Lists%20Manipulation/Python/Advanced%20Feature%20Selection.ipynb

FeatureViz

Featureviz looks like a cool feature selection package, but I wasn’t able to get it to work. It’s worth checking out. add links

1. Title Slide

(Timestamp: 00:00:00)

The presentation begins by introducing the core topic: Interpretable Models and the use of rules in machine learning. Rajiv Shah sets the stage by contrasting this talk with previous discussions on explainability (using tools to explain complex models). Instead, this session focuses on choosing models that are inherently easy to understand.

Shah expresses his interest in how machine learning helps us understand the world. He notes that while tools like SHAP or LIME help unpack complex models, there is immense value in approaching the problem differently: by selecting model architectures that are transparent by design.

The speaker invites the audience to view this not just as a technical lecture but as a discussion on the trade-offs between model complexity and interpretability, setting a collaborative tone for the presentation.

2. Table of Contents

(Timestamp: 00:02:30)

This slide outlines the roadmap for the presentation. Shah explains that he will begin with the “Big Picture” concepts—specifically the “Why?” and the “Baseline”—before diving into four specific technical approaches to rule-based modeling.

The four specific methods to be covered are Rulefit, GA2M (Generalized Additive Models with interactions), Rule Lists, and Scorecards. This structure moves from theoretical justification to practical application, comparing different algorithms that prioritize transparency.

Shah also mentions that a GitHub repository is available with code examples for everything shown, allowing the audience to reproduce the results for the tabular datasets discussed.

3. Section 1: Why?

(Timestamp: 00:03:09)

This section header introduces the fundamental question: Why do we want rules? The speaker moves past the obvious statement that “AI is important” to investigate the influences that drive data scientists toward complex, opaque models.

Shah prepares to discuss the cultural and competitive pressures in data science that prioritize raw accuracy over usability. This section serves as a critique of the “accuracy at all costs” mindset often found in the industry.

4. Mark Cuban Quote

(Timestamp: 00:03:17)

The slide features a quote from Mark Cuban: “Artificial Intelligence, deep learning, machine learning — whatever you’re doing if you don’t understand it — learn it. Because otherwise you’re going to be a dinosaur within 3 years.”

Shah briefly references this as the “obligatory” acknowledgment of AI’s massive importance in the current landscape. It reinforces that while the field is moving fast, the understanding of these systems is paramount, which ties into the presentation’s focus on interpretability.

5. Influences: Kaggle & Academia

(Timestamp: 00:03:40)

Shah identifies Kaggle competitions and academic research as two primary influences on data scientists. He notes that these platforms heavily incentivize accuracy above all else. For example, in the Zillow Prize, the difference between the top scores is minuscule, yet teams fight for that fraction of a percentage.

He argues that this environment trains data scientists to focus solely on improving metrics (like RMSE or AUC), often ignoring other critical trade-offs like model complexity, deployment difficulty, or explainability.

As he states, “One of the byproducts of Kaggle is a very heavy focus on making sure you improve your models around accuracy… and that’s how you can get a conference paper.” This sets up the problem of complexity creep.

6. The Netflix Prize Winners

(Timestamp: 00:05:39)

This slide shows the winners of the famous Netflix Prize, a competition held about 15 years ago where a team won $1 million for improving Netflix’s recommendation algorithm by 10%.

Shah uses this story to illustrate the peak of the “accuracy” mindset. The competition drew massive interest and drove innovation, but it also encouraged teams to prioritize the leaderboard score over the practicality of the solution.

7. Netflix Prize Progress Graph

(Timestamp: 00:06:14)

The graph displays the progress of teams over time during the Netflix competition. Shah points out that after an initial period of rapid improvement using standard algorithms, progress plateaued.

To break through these plateaus, teams began using Ensembling—combining multiple models together. The winning solution was an ensemble of 107 different models. Shah emphasizes that while this strategy is powerful for eking out the last bit of performance, it creates immense complexity.

8. The Engineering Cost of Complexity

(Timestamp: 00:07:39)

This slide reveals the ironic conclusion of the Netflix Prize: the winning model was never implemented. The engineering costs to deploy an ensemble of 107 models were simply too high compared to the marginal gain in accuracy.

Shah uses this as a cautionary tale: “If your focus is on accuracy… it drives you down towards this complexity… but often you end up with these complex models [that] are often very difficult to implement.” This highlights the disconnect between competitive data science and enterprise reality.

9. Understandable White Box Model (CLEAR-2)

(Timestamp: 00:08:04)

Shah transitions to the alternative: Interpretable Models. This slide shows a simple linear model (CLEAR-2) with only two features. This is a classic “White Box” model where the relationship between inputs and outputs is transparent.

The speaker contrasts this with the “Black Box” nature of complex ensembles. He argues that if you cannot understand what is going on inside a model, you cannot effectively debug it, nor can you easily convince stakeholders to trust it.

10. Complex White Box Model (CLEAR-8)

(Timestamp: 00:11:51)

This slide presents a linear model with eight features (CLEAR-8). While technically still a “White Box” model, Shah implies that as feature counts grow, true understandability diminishes.

He touches on this concept later in the “Caveats” section, noting that even linear models can become confusing if there is multicollinearity (features moving in the same direction). Just because we can see the coefficients doesn’t mean the model is intuitively “explainable” to a human if the variables interact in complex, non-obvious ways.

11. Easy to Understand Decision Tree

(Timestamp: 00:19:15)

Here, a simple Decision Tree is presented. Shah connects this to the history of rule-based learning, noting that early research found that keeping decision trees “short and stumpy” made them very easy for humans to explain.

This visual represents the ideal of interpretability: a clear path of logic (e.g., “If X is less than 3, go left”) that leads to a prediction. This is the foundation for the Rulefit method discussed later.

12. Too Much to Comprehend

(Timestamp: 00:07:46)

Contrasting the previous slide, this image shows a chaotic forest of decision trees. This represents modern ensemble methods like Random Forests or Gradient Boosted Machines.

Shah uses this visual to reinforce the point that while ensembles offer “Better Performance,” the sheer number of decision paths makes them “too much to Comprehend.” You lose the ability to trace the “why” behind a specific prediction, turning the system into a Black Box.

13. Pedro Domingos Tweet

(Timestamp: 00:08:22)

Shah acknowledges the counter-argument by showing a tweet from Pedro Domingos, a prominent machine learning researcher, who suggests that demanding explainability limits the potential of AI.

Shah respectfully disagrees with this stance in the context of enterprise data science. He argues that in the real world, “If you don’t understand what’s going on in your model, it’s hard for you to debug it, it’s hard to convince somebody else to adopt your model.” Practicality and trust often outweigh raw theoretical power.

14. Benefits of Interpretable Models

(Timestamp: 00:09:16)

This slide summarizes the key benefits of using interpretable models, referencing the work of Cynthia Rudin. The main advantages are: 1. Debugging: It is easier to spot weird behaviors. 2. Trust: Stakeholders and legal/risk teams are more likely to approve the model. 3. Deployment: These models can often be deployed as simple SQL queries or basic code, avoiding the need for heavy GPU infrastructure.

Shah emphasizes the deployment aspect: “You don’t have to go out and get a GPU… you can actually deploy directly within a database.”

15. Caveats of Interpretable Models

(Timestamp: 00:11:00)

Shah provides a necessary reality check. He clarifies that selecting an interpretable algorithm is only one part of the process. True interpretability depends on the entire data pipeline.

Issues like data labeling, feature engineering, and multicollinearity can render even a simple model confusing. For example, if two correlated features have opposite coefficients in a linear model, it becomes very difficult to explain the logic to a business user, even if the math is simple.

16. Section 2: Baseline

(Timestamp: 00:12:15)

This slide introduces the Baseline section. Shah advocates for always starting a project with a simple baseline model to establish a performance benchmark.

He shares an anecdote about people spending a year on a project only to be nearly matched by a simple model built in two hours. Establishing a baseline helps determine how much effort should be spent chasing incremental accuracy improvements.

17. The Problem: UCI Adult Dataset

(Timestamp: 00:12:54)

Shah introduces the dataset he will use for all examples in the talk: the UCI Adult Dataset (Census Income). The goal is a binary classification problem: predicting whether someone has a high or low income based on demographics.

He chooses this dataset because it represents typical enterprise tabular data: it has 30,000 rows, a mix of numerical and categorical features, and contains collinearity and interaction effects. This makes it a realistic test bed for the models he will demonstrate.

18. Baseline Models

(Timestamp: 00:13:53)

The speaker outlines the three baseline models he built to bracket the performance possibilities: 1. Logistic Regression: The standard statistical approach. 2. AutoML (H2O): A stacked ensemble of many models (Neural Networks, GBMs, etc.) representing the “maximum” possible performance. 3. OneR: A very simple rule-based algorithm.

These baselines provide the context for evaluating the interpretable models later.

19. Baseline Models Plot

(Timestamp: 00:14:12)

This plot visualizes Complexity vs. AUC (Area Under the Curve). * OneR is at the bottom (AUC ~0.60) with very low complexity. * Logistic Regression is in the middle (AUC ~0.91). * Stacked Ensemble is at the top (AUC ~0.93) but with massive complexity.

Shah notes that while the Stacked Ensemble wins on accuracy, the Logistic Regression is surprisingly close, highlighting that simpler models can often be “good enough.”

20. OneR Example

(Timestamp: 00:15:17)

Shah explains the OneR (One Rule) algorithm. This method finds the single feature in the dataset that best predicts the target. In the example shown (Iris dataset), utilizing just “Petal Width” classifies 96% of instances correctly.

He suggests OneR is a great way to detect Target Leakage—if one feature predicts the target perfectly, it might be “cheating.” It also sets the floor for performance; if a complex model can’t beat OneR, something is wrong.

21. Baseline Models Plot (Recap)

(Timestamp: 00:16:36)

Returning to the complexity plot, Shah reiterates the performance gap. The AutoML model sets the “ceiling” at 0.93 AUC.

The goal for the rest of the presentation is to see where the interpretable models (Rulefit, GA2M, etc.) fall on this graph. Can they approach the 0.93 AUC of the ensemble without incurring the massive complexity penalty?

22. Section 3: Rulefit

(Timestamp: 00:18:18)

This slide introduces the first major interpretable technique: Rulefit. Shah mentions familiarity with this from his time at Data Robot and notes that it is a powerful way to combine the benefits of trees and linear models.

23. What is Rulefit?

(Timestamp: 00:18:30)

Rulefit is an algorithm developed by Friedman and Popescu (2008). It works by: 1. Building a random forest of short, “stumpy” decision trees. 2. Extracting each path through the trees as a “Rule.” 3. Using these rules as binary features in a sparse linear model (Lasso).

This approach allows the model to capture interactions (via the trees) while maintaining the interpretability of a linear equation.

24. H2O Rulefit Output

(Timestamp: 00:22:19)

Shah displays the output from the H2O Rulefit implementation. The model generates human-readable rules, such as: “If Education < 12 AND Capital Gain < $7000, THEN Coefficient is negative.”

He notes that while the rules are readable, the raw output can look like “computer-ese.” However, it allows a data scientist to identify specific segments of the population (e.g., low education, low capital gain) that strongly drive the prediction.

25. Overlapping Rules

(Timestamp: 00:24:30)

A key characteristic of Rulefit is that the rules overlap. A single data point might satisfy multiple rules simultaneously.

Shah points out that this adds a layer of complexity to interpretability. To understand a prediction, you have to sum up the coefficients of all the rules that apply to that person. This is different from a decision tree where you fall into exactly one leaf node.

26. H2O Rulefit with Linear Terms

(Timestamp: 00:25:55)

One limitation of pure rules is handling continuous variables (like age or miles driven). Rules have to “bin” these variables (e.g., Age < 30, Age 30-40).

Shah explains that H2O Rulefit solves this by including Linear Terms. The model can use rules for non-linear interactions and standard linear coefficients for continuous trends. This hybrid approach boosts the AUC significantly (up to 0.88 in this example) by capturing linear relationships more naturally.

27. Rulefit Results

(Timestamp: 00:27:04)

This slide plots the performance of Rulefit models with varying numbers of rules. Shah demonstrates that by increasing the number of rules (complexity), the AUC climbs closer to the Stacked Ensemble.

He concludes that Rulefit is a versatile tool. You can tune the “dial” of complexity: fewer rules for more interpretability, or more rules for higher accuracy, often getting very competitive performance.

28. Section 4: GA2M

(Timestamp: 00:31:35)

The presentation moves to the second technique: GA2M (Generalized Additive Models with pairwise interactions). Shah notes that while GAMs have existed for a while, modern implementations like Microsoft’s Explainable Boosting Machines (EBM) have made them much more accessible and powerful.

29. What is GA2M?

(Timestamp: 00:32:02)

GA2M is essentially a linear model where features are binned, and pairwise interactions are automatically detected. Shah highlights InterpretML, an open-source library from Microsoft that implements this via EBMs.

The model structure is additive: . This means the final score is just the sum of individual feature scores and interaction scores, making it very transparent.

30. GA2M Binning

(Timestamp: 00:32:42)

Shah explains how GA2M handles numerical data. Instead of a single slope coefficient (like in logistic regression), the model bins the continuous feature (e.g., dividing “criminal history” into ranges).

Each bin gets its own coefficient. This allows the model to learn non-linear patterns (e.g., risk might go up, then down, then up again as a variable increases) while remaining easy to inspect.

31. Interactions in GA2M

(Timestamp: 00:33:08)

The “2” in GA2M stands for pairwise interactions. Shah emphasizes that this is the model’s superpower. While standard linear models struggle with interactions (e.g., the combined effect of age and education), GA2M has an efficient algorithm to automatically find the most important pairs.

This allows the model to achieve accuracy levels comparable to complex ensembles (AUC 0.93) because it captures the interaction signal that simple linear models miss.

32. GA2M Visualization

(Timestamp: 00:35:14)

Shah showcases the InterpretML dashboard. It provides clear visualizations of how each feature contributes to the prediction.

In the example, we see the coefficients for different marital statuses. This acts like a “lookup table” for risk. Shah argues that this is very “model risk management friendly” because stakeholders can validate every single coefficient and interaction term to ensure they make business sense.

33. Section 5: Rule Lists

(Timestamp: 00:40:28)

The third approach is Rule Lists. Shah introduces this as a method to solve the “overlapping rules” problem found in Rulefit.

34. What are Rule Lists?

(Timestamp: 00:40:48)

Rule Lists are ordered sets of IF-THEN-ELSE statements. Unlike Rulefit, where you sum up multiple rules, here an observation triggers only the first rule it matches.

Shah mentions implementations like CORELS and SBRL (Scalable Bayesian Rule Lists). The goal is to produce a concise list that a human can read from top to bottom to make a decision.

35. SBRL Process

(Timestamp: 00:41:09)

Creating an optimal rule list is computationally expensive because the algorithm must search through many permutations to find the best order.

Shah explains the logic: The algorithm finds a rule that covers a subset of data, removes those instances, and then finds the next rule for the remaining data. This sequential “peeling off” of data creates the IF-ELSE structure.

36. SBRL Output Example

(Timestamp: 00:41:45)

The output of an SBRL model is shown. It reads like a checklist: 1. IF Capital Gain > $7500 -> High Income (99% prob) 2. ELSE IF Education < 4 -> Low Income (90% prob) 3. ELSE…

Shah highlights the simplicity: “You just go down the list until you find the rule… much easier to explain to those marketing people.” The trade-off is a drop in accuracy (AUC 0.86) compared to GA2M or Rulefit.

37. Section 6: Scorecard

(Timestamp: 00:44:52)

The final approach is the Scorecard. Shah introduces this as perhaps the simplest and most widely recognized format for decision-making in industries like credit and criminal justice.

38. What are Scorecards?

(Timestamp: 00:45:04)

Scorecards are simple additive models where features are assigned integer “points.” To get a prediction, you simply add up the points.

Shah mentions tools like Optbinning and SLIM (Sparse Linear Integer Models). This format is beloved in operations because it can be printed on a physical card or implemented in a basic spreadsheet.

39. Scorecard Example

(Timestamp: 00:46:08)

This slide shows a scorecard built for the Adult dataset. * Capital Gain > 7000? +29 points. * Age < 25? -5 points.

Shah expresses a personal preference for this over raw coefficients: “I actually like this better… I think it’s a little easier to understand which features are most important.” The integer points make the “weight” of each factor immediately obvious to a layperson.

40. Summary

(Timestamp: 00:51:11)

Shah begins to wrap up the presentation, preparing to consolidate the four methods (Rulefit, GA2M, Rule Lists, Scorecards) into a final comparison.

41. Complexity vs AUC Summary Plot

(Timestamp: 00:51:13)

This is the definitive comparison graph of the talk. It places all discussed models on the Complexity vs. AUC plane. * GA2M (EBM) and Rulefit sit high up, offering near-SOTA accuracy with moderate interpretability. * Scorecards and Rule Lists sit lower on accuracy but offer maximum simplicity.

Shah summarizes the trade-off: “The Rule Lists and Scorecard… you lose a little bit [of accuracy]… but we talked about the trade-offs of being able to easily understand.”

42. Take Away

(Timestamp: 00:52:08)

The final message is a call to action: Try these approaches.

Shah encourages data scientists to add these tools to their toolkit. He asks them to consider the specific needs of their problem: Is it about transparency in calculation (Scorecard)? Or understanding factors (GA2M)? Often, a simple model that gets deployed is far better than a complex model that gets stuck in review.

43. Conclusion

(Timestamp: 00:52:52)

The presentation concludes with Rajiv Shah’s contact information. He mentions an upcoming blog post that will synthesize these topics and invites the audience to reach out with questions or feedback.

He reiterates that these interpretable models are often easier to get “buy-in” for, making them a pragmatic choice for real-world data science success.

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

1. The Spark of the AI Revolution

(Timestamp: 00:00)

The presentation begins with the title slide, “The Spark of the AI Revolution: Transfer Learning,” presented by Rajiv Shah from Snowflake. This talk was originally given at the University of Cincinnati and recorded later to share the insights with a broader audience.

Rajiv sets the stage by explaining that this is not a deep technical dive into code, but rather a descriptive history and analysis of the drivers behind the current AI boom. The goal is to explain how AI learns and how individuals can start to interrogate and understand these technologies in their own lives.

The core premise is that Transfer Learning is the catalyst that shifted AI from academic curiosity to a revolutionary force. The talk aims to bridge the gap for those unfamiliar with the underlying mechanics of how models like ChatGPT came to be.

2. Sparks of AGI: Early Experiments

(Timestamp: 01:00)

This slide illustrates an early experiment conducted by researchers investigating GPT-4. To understand how the model was learning, they gave it a concept and asked it to draw it using code (SVG). The slide displays a progression of abstract animal figures, showing how the model’s ability to represent concepts improved over time during training.

This references the paper “Sparks of Artificial General Intelligence,” which caused significant waves in the tech community. It suggests that these models were beginning to show signs of Artificial General Intelligence (AGI)—reasoning capabilities that extend beyond narrow tasks.

The visual progression from crude shapes to recognizable forms serves as a metaphor for the rapid evolution of these models. It highlights the mystery and potential power hidden within the training process of Large Language Models (LLMs).

3. Extinction Level Threat?

(Timestamp: 01:36)

The presentation addresses the extreme concerns surrounding the rapid scaling of AI technologies. The slide features a dramatic image reminiscent of the Terminator, referencing fears that unchecked AI development could pose an “extinction-level” threat to humanity.

Rajiv notes that as these technologies scale, there is a segment of the research and safety community worried about catastrophic outcomes. This sets up a contrast between the theoretical existential risks and the practical, everyday reality of how AI is currently being used.

This slide acknowledges the “hype and fear” cycle that dominates the media narrative, validating the audience’s anxiety before pivoting to a more grounded explanation of how the technology actually works.

4. The New AI Overlords

(Timestamp: 01:42)

Shifting to a lighter tone, this slide highlights the widespread adoption of AI by the younger generation. It cites a statistic that 89% of students have used ChatGPT for homework, humorously suggesting that children have already “accepted our new AI overlords.”

The slide points out a discrepancy in honesty, noting that while 89% use it, a significant portion (implied by the “11% are lying” joke) might not admit it. This reflects a fundamental shift in education and information retrieval that has already taken place.

This context emphasizes that the AI revolution is not just a future possibility but a current reality affecting how the next generation learns and works. It underscores the urgency of understanding these tools.

5. Fundamental Questions

(Timestamp: 01:53)

This slide poses the central questions that the presentation will answer: “What is AI doing?” and “How should you think about AI?” It serves as an agenda setting for the technical explanation that follows.

Rajiv transitions here from the societal impact of AI to the mechanics of machine learning. He prepares the audience to look “under the hood” to demystify the “magic” of tools like ChatGPT.

The goal is to move the audience from passive consumers of AI hype to critical thinkers who understand the limitations and capabilities of the technology based on how it is built.

6. How We Teach Computers

(Timestamp: 02:02)

The presentation begins its technical explanation with a fundamental question: “How do we teach computers?” The slide uses imagery of blueprints and tools, likening the traditional process of building AI models to craftsmanship.

This introduces the concept of Supervised Learning in a relatable way. Before discussing neural networks, Rajiv grounds the audience in traditional analytics, where humans explicitly guide the machine on what to look for.

The focus here is on the human element in traditional machine learning—the “artisan” who must carefully select inputs to get a desired output.

7. Identifying Features

(Timestamp: 02:11)

Using a real estate example, this slide explains the concept of Features (or variables). To teach a computer to value a house, one must identify specific characteristics like square footage, number of bedrooms, or closet space.

Rajiv explains that we capture these characteristics and organize them into a tabular format. This process is known as Feature Engineering, where the data scientist decides which attributes are relevant for the problem at hand.

This is the bedrock of traditional enterprise AI: converting real-world objects into structured data points that a machine can process mathematically.

8. Historical Data Patterns

(Timestamp: 02:50)

This slide displays a scatter plot correlating “Sales Price” with “Square Feet.” It illustrates how enterprises gather historical data to look for patterns and relationships backwards in time.

Rajiv notes that much of traditional analytics is simply looking at this historical data to understand what happened. However, the power of AI lies in using this data for forward-looking purposes.

The visual clearly shows a trend: as square footage increases, the price generally increases. This linear relationship is what the machine needs to “learn.”

9. Learning the Model

(Timestamp: 03:03)

Here, a line is drawn through the data points on the scatter plot. This line represents the Model. Learning, in this context, is simply the mathematical process of fitting this line to the historical data to minimize error.

Rajiv explains that the model “understands the relationships” defined by the data. Instead of a human manually writing rules, the algorithm finds the best-fit trend based on the input features.

This simplifies the concept of training a model down to its essence: finding a mathematical representation of a trend within a dataset.

10. Making Predictions

(Timestamp: 03:16)

This slide demonstrates the utility of the trained model. When a “New House” comes onto the market, the model uses the learned line to predict its value based on its square footage.

This defines the Inference stage of machine learning. The model is no longer learning; it is applying its “knowledge” (the line) to unseen data to generate a prediction.

It highlights the portability of a model—once trained, it can be used to make rapid assessments of new data points without human intervention.

11. The Domain Limitation

(Timestamp: 03:30)

The presentation introduces a critical limitation of traditional models. The slide shows the model trained on San Francisco data being applied to houses in South Carolina. The result is labeled “Poor Model.”

Rajiv explains that while you can technically take the model with you, it will fail because the underlying relationships between features (size) and targets (price) are different in different domains (geographies).

This illustrates the concept of Domain Shift or lack of generalization. A model is only as good as the data it was trained on, and it assumes the future (or new location) looks exactly like the past.

12. The Thinking Emoji

(Timestamp: 03:43)

This slide reinforces the previous point with a thinking emoji, emphasizing the realization that the existing model is inadequate. The “San Francisco Model” does not fit the “South Carolina Data.”

It serves as a visual pause to let the problem sink in: traditional machine learning is brittle. It requires the data distribution to remain constant.

Rajiv uses this to set up the labor-intensive nature of traditional analytics, where models cannot simply be “transferred” across different contexts.

13. Train New Model

(Timestamp: 03:55)

The solution in the traditional paradigm is presented here: “Train New Model.” To get accurate predictions for South Carolina, one must collect local data and repeat the entire training process from scratch.

This highlights the “Never-Ending Battle” of enterprise analytics. Data scientists are constantly retraining models for every specific region, product line, or use case.

This sets the baseline for why Transfer Learning (introduced later) is such a revolution. In the old way, knowledge was not portable; every problem required a bespoke solution.

14. Artisan AI

(Timestamp: 04:15)

Rajiv coins the term “Artisan AI” to describe this traditional approach. The slide features an image of a craftsman, symbolizing that these models are hand-built and rely heavily on human-crafted features.

This approach is slow and difficult to scale. Just as an artisan can only produce a limited number of goods, a data science team using these methods can only maintain a limited number of models.

It emphasizes that the intelligence in these systems comes largely from the human who engineered the features, not the machine itself.

15. Enterprise AI Use Cases

(Timestamp: 04:23)

This slide lists common Enterprise AI applications: Forecasting, Pricing, Customer Churn, and Fraud. It notes that 80% of production models currently fall into this category.

Rajiv grounds the talk in the reality of today’s business world. Despite the hype around Generative AI, most companies are still running on these “Artisan” structured data models.

This distinction is crucial for understanding the market. There is “Old AI” (highly effective, structured, labor-intensive) and “New AI” (generative, unstructured, scalable), and they solve different problems.

16. The Computer Science Perspective

(Timestamp: 04:41)

The presentation shifts from the enterprise view to the academic Computer Science view. The slide asks, “How should we teach computers?” signaling a move toward more advanced methodologies.

Rajiv indicates that computer scientists were trying to find ways to move beyond the limitations of manual feature engineering. They wanted machines to learn the features themselves.

This transition introduces the concept of Deep Learning and the move toward processing unstructured data like audio, images, and text.

17. Frederick Jelinek’s Insight

(Timestamp: 04:49)

This slide introduces a quote from Frederick Jelinek, a pioneer in speech recognition: “Every time I fire a linguist, the performance of the speech recognizer goes up.”

This provocative quote encapsulates a major shift in AI philosophy. It suggests that human expertise (linguistics) often gets in the way of raw data processing. Instead of hard-coding grammar rules, it is better to let the model learn patterns directly from the data.

Rajiv asks the audience to “chew on that,” as it foreshadows the “Bitter Lesson” of AI: massive compute and data often outperform human domain expertise.

18. Computer Vision in 2010

(Timestamp: 05:47)

The slide depicts the state of Computer Vision around 2010. It shows a process of manual feature extraction (like HOG - Histogram of Oriented Gradients) used to identify shapes and edges.

Rajiv explains that even in vision, researchers were essentially doing “Artisan AI.” They sat around thinking about how to mathematically describe the shape of a car or a truck to a computer.

This illustrates that before the deep learning boom, computer vision was stuck in the same “feature engineering” trap as tabular analytics.

19. SVM Classification

(Timestamp: 06:05)

Following feature extraction, this slide shows a Support Vector Machine (SVM) classifier separating data points (cars vs. trucks). This was the standard approach: extract features manually, then use a simple algorithm to classify them.

This reinforces the previous point about the limitations of the time. The intelligence was in the manual extraction, not the classification model.

Rajiv mentions his own work at Caterpillar, noting that this was exactly how they tried to separate images of machinery—a tedious and specific process.

20. Fei-Fei Li and Big Data

(Timestamp: 06:15)

The slide introduces Professor Fei-Fei Li, a visionary in computer vision. It features a collage of images, hinting at the need for scale.

Rajiv explains that Fei-Fei Li recognized that for computer vision to advance, it needed to move away from tiny datasets (100-200 images) and toward massive scale. She understood that deep learning required vast amounts of data to generalize.

This marks the beginning of the “Big Data” era in AI, where the focus shifted from better algorithms to better and larger datasets.

21. ImageNet

(Timestamp: 06:41)

This slide details ImageNet, the dataset Fei-Fei Li helped create. It contains 14 million images across 1000 classes.

Rajiv highlights the sheer effort involved, noting the use of Mechanical Turk to crowdsource the labeling of these images. He calls this the “dirty secret” of AI—that it is powered by low-wage human labor labeling data.

ImageNet became the benchmark that drove the AI revolution. It provided the “fuel” necessary for neural networks to finally work.

22. AlexNet and GPUs

(Timestamp: 07:44)

The presentation introduces Alex Krizhevsky, a graduate student under Geoffrey Hinton. The slide mentions “AlexNet” and the use of GPUs (Graphics Processing Units).

Rajiv tells the story of how Alex decided to use NVIDIA gaming cards to train neural networks. Traditional CPUs were too slow for the math required by deep learning.

This moment—combining the massive ImageNet dataset with the parallel processing power of GPUs—was the “big bang” of modern AI.

23. AlexNet Training Details

(Timestamp: 08:05)

This slide provides the technical specs of AlexNet: trained on 1.2 million images, using 2 GPUs, taking roughly 6 days, with 60 million parameters.

Rajiv emphasizes that while 6 days seems long, the result was a model vastly superior to anything else. It proved that neural networks, which had been theoretical for decades, were now practical.

The “60 million parameters” figure is a precursor to the “billions” and “trillions” we see today, marking the start of the parameter scaling race.

24. Crushing the Competition

(Timestamp: 08:27)

A chart displays the results of the ImageNet Large Scale Visual Recognition Challenge. It shows AlexNet achieving a significantly lower error rate than the competitors.

Rajiv notes that the performance jump was so dramatic that by the following year, every competitor had switched to using the AlexNet architecture.

This visualizes the paradigm shift. The “Artisan” methods were instantly obsolete, replaced by Deep Learning.

25. Feature Engineering vs. Deep Learning

(Timestamp: 08:37)

Using a humorous meme format, this slide compares the “Old Way” (Feature Engineering + SVM) with the “New Way” (AlexNet). The AlexNet side is depicted as a powerful, overwhelming force.

This solidifies the takeaway: Deep Learning didn’t just improve upon the old methods; it completely replaced them for unstructured data tasks like vision.

It emphasizes that the model learned the features itself (edges, textures, shapes) rather than having humans manually code them.

26. The 1000 Classes

(Timestamp: 08:51)

This slide shows examples of the 1000 classes in ImageNet, ranging from specific dog breeds to everyday objects.

Rajiv explains that this model learned to identify a vast array of things from the raw pixels. It went from raw vision to understanding textures, shapes, and objects.

However, he sets up the next problem: What if you want to identify something not in those 1000 classes?

27. The Hot Dog Problem

(Timestamp: 09:01)

referencing a famous scene from the show Silicon Valley, this slide presents the specific challenge of classifying “Hot Dogs.”

Rajiv uses this to ask: How do you help a buddy with a startup who needs to find hot dogs if “hot dog” isn’t one of the primary categories, or if they need a specific type of hot dog? Do you have to start from scratch?

This sets the stage for Transfer Learning—the solution to avoiding the need for 14 million images every time you have a new problem.

28. Pre-Trained Models

(Timestamp: 09:12)

The slide introduces the concept of a Pre-trained Model. This is the model that has already learned the 1000 classes from ImageNet.

Rajiv explains that this model already “knows” how to see. It understands edges, curves, and textures. This knowledge is contained in the “weights” of the neural network.

The key idea is that we don’t need to relearn how to “see” every time we want to identify a new object.

29. Transfer Learning Mechanics

(Timestamp: 09:30)

This technical slide illustrates how Transfer Learning works. It shows the layers of a neural network. We keep the early layers (which know shapes and textures) and only retrain the final layers for the new task (e.g., identifying boats).

Rajiv explains that we can transfer “most of that knowledge” and only change a small amount of parameters (less than 10%).

This is the revolution: You can build a world-class model with a small amount of data by standing on the shoulders of the giant ImageNet model.

30. The Revolution

(Timestamp: 09:54)

A graph titled “Transfer Learning Revolution” shows the dramatic improvement in accuracy when using transfer learning versus training from scratch. It includes a quote from Andrew Ng stating that transfer learning will be the next driver of commercial success.

Rajiv emphasizes that this capability allowed startups and companies to build powerful AI without needing Google-sized datasets. It democratized access to high-performance computer vision.

This wraps up the vision section of the talk, establishing Transfer Learning as the “Spark.”

31. The Implications

(Timestamp: 10:11)

The slide shows a YouTube video thumbnail from 2016 featuring Geoffrey Hinton. This transitions the talk to the societal and professional implications of this technology.

Rajiv prepares to share a famous prediction by Hinton regarding the medical field, specifically radiology. It signals a shift from “how it works” to “what it does to jobs.”

32. The Coyote Moment

(Timestamp: 10:19)

The slide displays a webpage for the University of Cincinnati Radiology Fellows. Rajiv quotes Hinton: “Radiologists are like the coyote that’s already over the edge of the cliff but hasn’t yet looked down.”

Hinton suggested people should stop training radiologists because AI interprets images better. Rajiv humorously notes that since he was speaking at U of C, he had to show the “coyotes” in the audience.

This highlights the tension between AI capabilities and human expertise, a recurring theme in the presentation.

33. NLP: The Academic View

(Timestamp: 10:42)

The presentation switches domains from Computer Vision to Natural Language Processing (NLP). The slide depicts a traditional academic setting, representing the text researchers.

Rajiv explains that while Computer Vision was having its revolution with AlexNet, the text folks were still doing things the “Old Way”—crafting features and rules for language.

They saw the success in vision and wondered how to replicate it for text, but language proved more difficult to model than images initially.

34. Traditional NLP Tasks

(Timestamp: 11:04)

This slide lists various NLP tasks: Classification, Information Extraction, and Sentiment Analysis.

Rajiv notes that traditionally, each of these was a separate discipline. You built a specific model for sentiment, a different one for translation, and another for summarization. There was no “one model to rule them all.”

This fragmentation made NLP difficult and resource-intensive, as knowledge didn’t transfer between tasks.

35. The GLUE Benchmark

(Timestamp: 11:25)

The slide introduces the GLUE Benchmark (General Language Understanding Evaluation). This was a collection of different text tasks put together to measure general language ability.

Rajiv explains this was an attempt to push the field toward general-purpose models. Researchers wanted a single metric to see if a model could understand language broadly, not just solve one specific trick.

36. The Transformer Architecture

(Timestamp: 11:37)

This slide marks the turning point for text: the introduction of the Transformer architecture by Google researchers in 2017 (the “Attention Is All You Need” paper).

Rajiv highlights that this architecture was not only more accurate (higher BLEU scores) but, crucially, more efficient.

The Transformer allowed for parallel processing of text, unlike previous sequential models (RNNs/LSTMs), unlocking the ability to train on massive datasets.

37. Lower Training Costs

(Timestamp: 11:46)

The slide emphasizes the Training Cost reduction associated with Transformers.

Rajiv points out that because the architecture used less processing power per unit of data, researchers immediately asked: “What happens if we give it more processing?”

This efficiency paradox—making something cheaper allows you to do vastly more of it—sparked the scaling era of LLMs.

38. Exponential Growth

(Timestamp: 11:56)

A graph demonstrates the exponential growth in the size of Transformer models (measured in parameters) over just a few years. The curve shoots upward vertically.

Rajiv explains that this scaling—simply making the models bigger and feeding them more data—led to the performance of GPT-4.

This visualizes the “Scale” aspect of modern AI. We haven’t necessarily changed the architecture since 2017; we’ve just made it significantly larger.

39. GPT-4 and Images

(Timestamp: 12:11)

The presentation circles back to the GPT-4 generated images from Slide 2.

Rajiv connects the Transformer architecture and scaling directly to these “Sparks of AGI.” The ability to reason and draw emerged from simply predicting the next word at a massive scale.

40. The Era of ChatGPT

(Timestamp: 12:16)

The slide displays the ChatGPT logo, symbolizing the current era where these technical advancements reached the public consciousness.

Rajiv sets up the next section of the talk: explaining exactly how a model like ChatGPT is trained. He moves from history to the “Recipe.”

41. The Learning Process

(Timestamp: 12:20)

A visual diagram outlines the evolutionary stages of ChatGPT. It previews the three steps Rajiv will cover: Pre-training, Fine-tuning, and Alignment.

This roadmap helps the audience understand that ChatGPT isn’t just one static thing; it’s the result of a multi-stage pipeline involving different types of learning.

42. Recipe Step 1: Foundation Model

(Timestamp: 12:27)

The first step identified is the “Foundation Model” (or Base Model).

Rajiv explains that the core capability of these models is Next Word Prediction. Before it can answer questions or be helpful, it must simply learn the statistical structure of language.

43. Predictive Keyboards

(Timestamp: 12:31)

To make the concept relatable, the slide compares LLMs to the predictive text feature on a smartphone keyboard.

Rajiv notes that while the game on your phone is simple, scaling that concept up to the entire internet makes it incredibly powerful. It grounds the “magic” of AI in a familiar user experience.

44. Next Token Prediction

(Timestamp: 12:40)

This technical slide defines “Next Token Prediction.” It explains that the model looks at a sequence of text and calculates the probability of what comes next.

Rajiv emphasizes that this is a hard statistical problem. There are many possibilities for the next word, and the model must learn to weigh them based on context.

45. The Homer Simpson Challenge

(Timestamp: 13:05)

Rajiv introduces a specific experiment: Training a Transformer to speak like Homer Simpson. He mentions using 7MB of Simpsons scripts (~7 million tokens).

This serves as a concrete example to show how training data size affects model performance.

46. 4 Million Tokens

(Timestamp: 13:32)

The slide shows the output of the model when trained on only 4 Million tokens. The text is “nonsensical and random.”

Rajiv demonstrates that with insufficient data, the model hasn’t learned grammar or structure yet. It’s just outputting characters.

47. 16 Million Tokens

(Timestamp: 13:37)

At 16 Million tokens, the output improves slightly. It contains random words and incorrect grammar, but it’s recognizable as language.

This illustrates the “grokking” phase where the model starts to pick up on basic syntax but lacks semantic meaning.

48. 64 Million Tokens

(Timestamp: 13:39)

With 64 Million tokens, the model generates text that is “close to a proper sentence” and sounds vaguely like Homer Simpson.

Rajiv uses this progression to prove that these models are statistical engines. With enough data, they mimic the patterns of the training set effectively.

49. GPT-2 Specifications

(Timestamp: 13:54)

The slide details GPT-2 (released in 2019), which had 1.5 Billion parameters.

Rajiv recalls that when GPT-2 came out, he wasn’t excited because it was just a “creative storytelling model.” It wasn’t factually accurate. He wants the audience to remember that at their core, these models are just predicting the next word, not checking facts.

50. Llama 3.1 and Scale

(Timestamp: 14:27)

Updating the timeline, this slide shows Llama 3.1. It highlights the training data: 15 Trillion Tokens and the compute: 40 Million GPU Hours.

Rajiv emphasizes that 15 trillion tokens is an “unfathomable amount of information.” The scale has increased 10,000x since GPT-2.

This underscores the energy and compute intensity of modern AI—it requires massive infrastructure.

51. Hallucinations

(Timestamp: 15:15)

This slide addresses Hallucinations. It uses an example of asking for the “Capital of Mars.” The model will confidently invent an answer.

Rajiv argues that “hallucination” isn’t the right metaphor because the model isn’t malfunctioning. It is doing exactly what it was designed to do: predict the most likely next word. It has no concept of “truth,” only statistical likelihood.

52. GPT-2 Failure on Sentiment

(Timestamp: 16:17)

Rajiv shows an example of trying to use the base GPT-2 model for a specific task: Customer Sentiment. When prompted, the model just continues the story instead of classifying the sentiment.

This illustrates that Base Models are creative but not useful for following instructions. They don’t know they are supposed to solve a problem; they just want to write text.

53. Recipe Step 2: Instruction Fine-Tuned

(Timestamp: 16:38)

This introduces the second step in the ChatGPT recipe: “Instruction Fine-Tuned Model.”

Rajiv explains that to make the model useful, we must teach it to follow orders. This is done via Transfer Learning—taking the base model and training it further on examples of instructions and answers.

54. Fine-Tuning for Sentiment

(Timestamp: 16:46)

The slide shows the process of fine-tuning the language model specifically for Sentiment Analysis.

By showing the model examples of “Sentence -> Sentiment,” we can tweak the parameters so it learns to perform classification rather than just storytelling.

55. Multi-Task Fine-Tuning

(Timestamp: 17:22)

Rajiv expands the concept. We don’t just fine-tune for one task; we fine-tune for Topic Classification as well.

The key insight is that one model can now solve multiple problems. Unlike the “Old NLP” where you needed separate models, the LLM can swap between tasks based on the instruction.

56. Translation Task

(Timestamp: 17:24)

The slide adds Translation to the mix, using about 10,000 examples.

This reinforces the “General Purpose” nature of LLMs. They are Swiss Army knives for text.

57. Generalization to New Tasks

(Timestamp: 17:30)

Rajiv poses a challenge: What happens if you give the model a task it hasn’t seen before?

The slide indicates the model will try to solve it. This is the breakthrough of Generalization. Because it understands language so well, it can interpolate and attempt tasks it wasn’t explicitly trained on.

58. Practical Applications

(Timestamp: 17:55)

This slide showcases the wide array of use cases: Code explanation, Creative writing, Information extraction, etc.

Rajiv explains that these capabilities exist because we have “trained these models to follow instructions.” This is why we can talk to them via Prompts.

59. Zero Shot Learning

(Timestamp: 18:19)

The slide introduces “Zero shot learning” and “Prompting.”

This is the ability to get a result without showing the model any examples (zero shots). Rajiv notes that there is a “whole language” around prompting, but fundamentally, it’s just giving the model the instruction we trained it to expect.

60. Weeks vs. Days

(Timestamp: 18:41)

A comparison slide contrasts “Training a ML Model (weeks)” with “Prompting a LLM (days).”

Rajiv highlights the efficiency shift. In the old days, solving a sentiment problem meant weeks of data collection and training. Now, it takes minutes to write a prompt. This is a massive productivity booster for NLP tasks.

61. Reasoning and Planning

(Timestamp: 19:25)

The presentation pivots to the limitations of LLMs, specifically regarding Reasoning and Planning. The slide shows a “Block Stacking” puzzle.

Rajiv explains that stacking blocks requires planning several steps ahead. It is not a one-step prediction problem; it requires maintaining a state of the world in memory.

62. Mystery World Failure

(Timestamp: 20:40)

The slide introduces “Mystery World,” a variation of the block problem where the names of the blocks are changed to random words.

While a human (or a 4-year-old) understands that changing the name doesn’t change the physics of stacking, GPT-4 fails (3% accuracy). Rajiv explains that the model gets distracted by the creative aspect of the words and loses the logical thread. It shows these models struggle with abstract reasoning.

63. Recipe Step 3: Aligned Model

(Timestamp: 21:56)

The final step in the recipe is the “Aligned Model.”

Rajiv introduces the need for safety and helpfulness. A model that follows instructions perfectly might follow bad instructions. We need to align it with human values.

64. Galactica: Science LLM

(Timestamp: 22:00)

The slide presents Galactica, a model released by Meta focused on science.

Rajiv describes the intent: a helpful assistant for researchers to write code, summarize papers, and generate scientific content. It was meant to be a specialized tool.

65. Galactica Output

(Timestamp: 22:20)

An example of Galactica’s output shows it generating technical content.

Rajiv highlights the potential utility. It looked like a powerful tool for accelerating scientific discovery.

66. Galactica Pulled

(Timestamp: 22:47)

The slide reveals that Meta pulled the model shortly after release.

Rajiv explains why: users found they could ask it for the “benefits of eating crushed glass” or “benefits of suicide,” and the model would happily generate a scientific-sounding justification. It lacked a safety layer. This incident underscored the necessity of Red Teaming and alignment before release.

67. Learning What is Helpful

(Timestamp: 23:59)

To explain how we define “helpful,” Rajiv shows a Stack Overflow question.

He notes that defining “helpful” mathematically is difficult. Unlike “square footage,” helpfulness is subjective and nuanced.

68. Technical Answer

(Timestamp: 24:05)

The slide shows a detailed technical answer.

Rajiv points out that trying to create a “feature list” for what makes this answer helpful is nearly impossible. We can’t write a rule-based program to detect helpfulness.

69. The Dating App Analogy

(Timestamp: 24:26)

Rajiv uses a humorous Dating App analogy. He compares the “Old Way” (filling out long compatibility forms/features) with the “New Way” (Swiping).

He explains that Swiping is a way of capturing human preferences without asking the user to explicitly define them. This is how we teach AI what is helpful.

70. Collect Human Feedback

(Timestamp: 24:55)

The slide details the process: “Collect Human Feedback.”

We present the model with two options and ask a human, “Which is better?” By collecting thousands of these “swipes,” we build a dataset of human preference.

71. RLHF (Reinforcement Learning from Human Feedback)

(Timestamp: 25:05)

This slide introduces the technical term: RLHF.

Rajiv explains this is the layer that turns a raw instruction-following model into a safe, helpful product like ChatGPT. It is an active curation process, similar to curating an Instagram feed.

72. The Makeover Example

(Timestamp: 25:27)

A “Before and After” makeover image illustrates the effect of RLHF.

The “Before” is the raw model (messy, potentially harmful). The “After” is the aligned model (polished, safe, presentable).

73. Tuning Responses

(Timestamp: 25:44)

The slide shows different ways an AI can answer a question: Sycophantic (sucking up to the user), Baseline Truthful (blunt), or Helpful Truthful.

Rajiv notes we can train models to have specific personalities. We can make them polite, or we can make them “kiss your butt” if the user wants validation.

74. AI Conversations

(Timestamp: 26:17)

This slide references Character.ai and the trend of people spending hours talking to AI personas.

Rajiv mentions research showing people sometimes prefer AI doctors over human ones because the AI is patient, listens, and is polite (due to alignment). This suggests a future where AI handles high-touch conversational roles.

75. The Full Recipe

(Timestamp: 27:19)

The presentation summarizes the full pipeline: Foundation Model -> Instruction Fine-Tuned -> Aligned Model.

This visual recap cements the three-stage process in the audience’s mind.

76. Learning Mechanisms Recap

(Timestamp: 27:23)

Rajiv maps the learning mechanisms to the stages: 1. Next Word Prediction (Foundation) 2. Multi-task Training (Instruction) 3. Human Preferences (Alignment)

He reiterates that understanding these three mechanics helps explain why the models behave the way they do (hallucinations, ability to code, politeness).

77. Key Takeaways

(Timestamp: 27:43)

The presentation transitions to the conclusion with three main takeaways: 1. Measure Twice 2. Respect Scale 3. Critical Thinking

Rajiv notes in the video that he skimmed these in the original talk, but the slides provide the detail for how to work effectively with AI.

78. Measure Twice (Benchmarks)

([Timestamp: End of Transcript])

This slide displays a collage of AI benchmarks (MMLU, HumanEval, etc.).

The concept “Measure Twice” emphasizes that because AI models are probabilistic and prone to hallucination, we cannot trust them blindly. We must rely on rigorous benchmarking to understand their capabilities and failures before deployment.

79. Targets for Evaluation

([Timestamp: End of Transcript])

This slide likely elaborates on the need for clear “targets” or ground truth when evaluating models.

You cannot improve what you cannot measure. In the context of “Prompt Engineering,” this means you shouldn’t just tweak prompts randomly; you need a systematic way to measure if a prompt change actually improved the output.

80. Respect Scale

([Timestamp: End of Transcript])

This slide illustrates the exponential growth in single-chip inference performance.

“Respect Scale” refers to the lesson that betting against hardware and data scaling is usually a losing bet. The capabilities of these models grow faster than our intuition expects.

81. The Scaling Lesson (Humans)

([Timestamp: End of Transcript])

This slide likely discusses how human expertise fits into the scaling laws. As technology scales, the role of the human shifts from doing the work to evaluating the work.

82. The Plateau

([Timestamp: End of Transcript])

A visual showing that human contribution or specific “hacks” tend to plateau, whereas general-purpose methods that leverage scale (like Transformers) continue to improve.

This reinforces the “Bitter Lesson”: specialized, hand-crafted solutions eventually lose to general methods that can consume more compute.

83. AlexNet vs Transformers

([Timestamp: End of Transcript])

A comparison between AlexNet (the start of the deep learning era) and Transformers (the current era).

It highlights the massive increase: 10,000x more data and 1,000x more compute. This illustrates that the fundamental driver of progress has been scale.

84. The Bitter Lesson

([Timestamp: End of Transcript])

This slide explicitly references Rich Sutton’s “The Bitter Lesson.”

The lesson is that researchers often try to build their knowledge into the system (like Jelinek’s linguists), but in the long run, the only thing that matters is leveraging computation. AI succeeds when we stop trying to teach it how to think and just give it enough power to learn on its own.

85. Text to SQL

([Timestamp: End of Transcript])

The slide examines Text to SQL, a common enterprise use case. It compares AI performance to human experts.

It notes that while AI is good, humans still achieve higher exact match accuracy. This nuances the “Respect Scale” argument—for high-precision tasks, human oversight is still required.

86. Critical Thinking

([Timestamp: End of Transcript])

The final takeaway is “Critical Thinking.”

In an age where AI can generate convincing but false information, human judgment becomes the most valuable skill. We must critically evaluate the outputs of these models.

87. Predictions and Concerns

([Timestamp: End of Transcript])

This slide recaps expert predictions, ranging from job displacement to existential threats.

It serves as a reminder that even experts disagree on the timeline and impact, reinforcing the need for individual critical thinking rather than blind faith in pundits.

88. Practical Limits: Bezos and Alexa

([Timestamp: End of Transcript])

A humorous slide showing Jeff Bezos and Alexa. It likely references an instance where Alexa failed to understand a simple context despite Amazon’s massive resources.

This illustrates the “Practical Limits of Learning.” Despite the hype, current AI still struggles with basic context that humans find trivial.

89. Autonomous Driving Limits

([Timestamp: End of Transcript])

Images of an autonomous driving interface and a car accident.

This points out that in high-stakes physical environments, “99% accuracy” isn’t enough. The “long tail” of edge cases remains a massive hurdle for AI.

90. Chatbot Failures

([Timestamp: End of Transcript])

Examples of chatbots failing simple math or making “legally binding offers” (referencing the Air Canada chatbot lawsuit).

This warns against deploying these models in critical business flows without guardrails. They can confidently make costly mistakes.

91. Interaction Principles

([Timestamp: End of Transcript])

This slide summarizes the three principles for interacting with AI: “Measure twice,” “Respect scale,” and “Think critically.”

It acts as the final instructional slide, giving the audience a mantra for navigating the AI landscape.

92. Evolution of Generative Capabilities

([Timestamp: End of Transcript])

The slide shows a series of Unicorn images generated by GPT-4 over time.

This visualizes the rapid improvement in generative capabilities. Just as the “Sparks of AGI” images improved, the fidelity of these outputs continues to evolve, reminding us that we are looking at a moving target.

93. Conclusion

([Timestamp: End of Transcript])

The final slide concludes the presentation with Rajiv Shah’s name and affiliation (Snowflake).

It wraps up the narrative: from the spark of Transfer Learning to the fire of the Generative AI revolution, offering a practical, technical, and critical perspective on the technology shaping our future.

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

1. Title Slide: A Practical Perspective on Generative AI

(Timestamp: 00:01)

This presentation begins with an introduction by Rajiv Shah from Snowflake. The talk focuses on distinguishing “what’s easy to do with LLMs, what’s hard to do with LLMs, and where that boundary is for generative AI.” The content is framed as a practical guide for enterprises navigating the hype versus the reality of implementing these technologies.

The speaker sets the stage for a narrative-driven presentation that will move away from abstract theory and into concrete examples. The goal is to walk through the basics of Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) before applying them to real-world scenarios involving legal research and enterprise data analysis.

2. Presentation Goals

(Timestamp: 00:31)

The agenda for the talk is outlined here. The speaker intends to cover the foundational mechanisms of how to use LLMs effectively, specifically focusing on RAG. To make the concepts relatable, the presentation uses two storytelling devices: a fictional law firm (“Dewey, Cheatham, and Howe”) and a hypothetical company (“Frosty”).

These two stories serve to illustrate how people are currently using Generative AI, the specific limitations they encounter, and the engineering required to build a robust application. The speaker emphasizes that the talk will explore “what does it take to actually develop a generative AI application” beyond just simple prompting.

3. The Avianca Case

(Timestamp: 01:00)

The speaker introduces the concept of hallucinations through a famous real-world example involving the airline Avianca. A lawyer, attempting to speed up his work on a brief regarding a personal injury case, used ChatGPT for legal research. The AI “found some cases that were unpublished,” which the lawyer cited in court.

However, ChatGPT had “made up those cases.” The lawyer was admonished by the bar for submitting fictitious legal precedents. This slide serves as a warning: while LLMs are powerful tools, they cannot be blindly trusted for factual research because they are prone to fabricating information when they don’t know the answer.

4. Generative AI in Action

(Timestamp: 02:14)

To demonstrate the variability of LLMs, the speaker presents a side-by-side comparison of two models (Google Gemma and a “Woflesh” model) answering the same prompt: “How many vehicles will Rivian manufacture in Normal, Illinois?” The models provide different answers.

This illustrates a key characteristic of Generative AI: “Two different manufacturers, two different methods for training these models are probably going to lead to two different results.” It highlights that out-of-the-box models rely on their specific training data, which may be outdated or weighted differently, leading to inconsistent factual accuracy.

5. Next Token Prediction

(Timestamp: 03:02)

This technical diagram explains why models hallucinate. The speaker clarifies that LLMs function by trying to predict the next word or token based on statistical likelihood. They are not databases of facts; they are engines designed to construct coherent sentences.

“They’re not worried about truth and false; they’re really trying to tell what the most cohesive, coherent story is.” Because the model is optimizing for the most probable next word to complete a pattern, it will confidently generate plausible-sounding but factually incorrect information if that sequence of words is statistically likely.

6. LLM Mistakes

(Timestamp: 03:30)

Here, the speaker provides examples of the “Next Token Prediction” logic failing to provide truth. If asked for the “Capital of Mars,” the model doesn’t know Mars has no capital; it simply tries to “complete that story” by inventing a name. Similarly, when asked to perform math, the model isn’t calculating; it is predicting the next characters in a math-like sequence.

The slide shows the model failing at basic arithmetic because “it looks like it’s read too many release notes, not actually enough math.” This reinforces that LLMs are linguistic tools, not calculators or knowledge bases, and they lack an internal concept of “fictional” versus “factual.”

7. Risks for Enterprises

(Timestamp: 04:03)

This slide highlights the liability risks for companies, citing the Air Canada chatbot case. In this instance, a chatbot invented a refund policy that did not exist. When the customer sued, the airline argued the chatbot was responsible, but the tribunal ruled the company was liable for its agent’s statements.

The speaker notes, “We’re going to treat this chatbot just like one of your employees… you’re responsible for what this model says.” This legal precedent explains why enterprises are hesitant to deploy Gen AI and why “Gen AI committees” are forming to manage governance and risk before public deployment.

8. Retrieval-Augmented Generation (RAG)

(Timestamp: 04:51)

To solve the hallucination problem, the presentation introduces Retrieval-Augmented Generation (RAG). The speaker describes this as a solution from academia designed to “ground” the model. Instead of relying solely on the model’s internal training data, RAG surrounds the model with external context.

The core idea is simple: “We’re going to ground it with information so it uses that information in answering the question.” This technique attempts to bridge the gap between the model’s linguistic capabilities and the need for factual accuracy in enterprise applications.

9. How RAG Works

(Timestamp: 05:20)

This diagram breaks down the RAG architecture. When a user asks a question, the system does not send it directly to the LLM. First, it goes out to “search and look for is there relevant information that’s related to this question.”

Once relevant documents are collected from a knowledge base, they are bundled with the original question and sent to the LLM. The LLM then generates an answer based only on that provided context. This ensures the “final answer is grounded” by factual documents rather than the model’s statistical predictions alone.

10. Grounding with 10-K Forms

(Timestamp: 05:51)

The speaker sets up a practical RAG demonstration using 10-K forms (annual reports filed by public companies). These documents are chosen because “you can trust that they’re factual.”

This slide prepares the audience to see how the previous question about Rivian’s manufacturing capacity—which generated inconsistent answers earlier—can be answered accurately when the model is forced to look at Rivian’s official financial filings.

11. Rivian Manufacturing Answer

(Timestamp: 06:07)

The slide shows the output of a RAG application. The question “How many vehicles do you manufacture in Normal?” is asked again. This time, the application provides a specific, fact-based answer derived from the uploaded documents.

This demonstrates the immediate utility of RAG: it turns the LLM from a creative writing engine into a synthesis engine that can read specific enterprise documents and extract the correct answer, mitigating the hallucination issues seen in Slide 4.

12. Context and Citations

(Timestamp: 06:21)

A critical feature of RAG is displayed here: Citations. The application shows exactly which document the answer came from. The speaker notes, “I can see exactly what’s the document that this answer came from… a nice source.”

This transparency is why RAG is the “number one most popular generative AI application.” It allows users to verify the AI’s work, building trust in the system—something impossible with a standard “black box” LLM response.

13. Chatbot for Legal Research

(Timestamp: 07:00)

The narrative shifts to the fictional law firm “Dewey, Cheatham, and Howe.” The firm wants to use AI to reduce the heavy workload of legal research. The initial thought process is to use raw LLMs because they are knowledgeable.

The speaker introduces a colleague who assumes, “I know it could pass the bar exam… why don’t I just wire it up directly?” This sets up the common misconception that because a model has general knowledge (passing a test), it is suitable for specialized professional work without further engineering.

14. GPT Models on the Bar Exam

(Timestamp: 07:22)

This chart reinforces the previous assumption, showing the progression of GPT models on the Multistate Bar Exam (MBE). GPT-4 significantly outperforms its predecessors, achieving a passing score.

While this suggests the model “knows something about the law,” the speaker hints that this is merely a multiple-choice test. Success here does not necessarily translate to the nuance required for actual legal practice, foreshadowing the errors to come in the story.

15. Hallucinating Statutes

(Timestamp: 07:50)

The first failure of the “raw LLM” approach is revealed. A lawyer asks for statutes regarding “online dating services in Connecticut.” The model confidently provides “Connecticut General Statute § 42-290.”

However, the lawyer discovers “there is no statute; this was entirely hallucinated.” Despite passing the bar exam, the model fabricated a law that sounded plausible but did not exist. This forces the firm to pivot toward a RAG approach to ground the AI in real legal literature.

16. Lexis+ AI

(Timestamp: 08:30)

The firm decides to use professional tools. They turn to Lexis+ AI, a commercial product that promises “Hallucination-Free Linked Legal Citations.” This tool uses the RAG approach discussed earlier, retrieving from a database of real case law.

The expectation is that by using a trusted vendor with a RAG architecture, the hallucination problem will be solved, and lawyers will receive accurate, citable information.

17. Conceptual Hallucinations

(Timestamp: 08:50)

Even with RAG and real citations, a new problem emerges: Conceptual confusion. The AI provides a real case but confuses the “Equity Cleanup Doctrine” with the “Doctrine of Clean Hands.” The speaker explains that while the words are similar, the legal concepts are distinct (one is about consolidating claims, the other about a plaintiff’s conduct, illustrated by a joke about P. Diddy).

The model found a document containing the words but failed to understand the meaning. This shows that RAG ensures the document exists, but not necessarily that the reasoning or application of that document is correct.

18. The Fictional Judge

(Timestamp: 09:50)

The model’s failure deepens with an example of an “inside joke.” A lawyer asks for opinions by “Judge Luther A. Wilgarten.” Wilgarten is a fictional judge created as a prank in law reviews.

The AI, treating the law reviews as factual text, retrieves “cases” by this fake judge. It fails to distinguish between a real judicial opinion and a satirical article within its knowledge base. This illustrates the “garbage in, garbage out” risk even within RAG systems if the model cannot discern the nature of the source material.

19. Hallucination Rates in Legal AI

(Timestamp: 10:44)

The speaker references a Stanford paper analyzing hallucination rates across major legal AI tools (Lexis, Westlaw, GPT-4). The chart shows that these tools still hallucinate or provide incomplete answers 17% to 33% of the time.

This data point serves as a reality check: “These models hallucinate using real questions.” Despite marketing claims of being “hallucination-free,” the complexity of the domain means that errors are still frequent, posing significant risks for professional use.

20. Limits of RAG

(Timestamp: 11:03)

This slide summarizes the limitations discovered in the legal example. RAG works well when documents are “True, Authoritative, and Applicable.” However, in complex domains like law, these attributes are often contested.

“Sometimes all these things are very contested and it gets really hard to separate it.” If the underlying documents contain conflicting information, satire, or outdated facts, the RAG system (which assumes retrieved text is “truth”) will propagate those errors to the user.

21. Why Legal is Hard

(Timestamp: 11:28)

The speaker elaborates on the complexity of legal research. It involves navigating different specialties (Tort vs. Maritime), jurisdictions (Federal vs. State), and authorities (Supreme Court vs. Law Reviews). Furthermore, the element of time is crucial—knowing if a case has been overturned.

“You really have to have a lot of knowledge to be able to weave everything in and out.” An LLM often lacks the meta-knowledge to weigh these factors, treating a lower court opinion from 1950 with the same weight as a Supreme Court ruling from 2024.

22. Conclusion on Legal Chatbots

(Timestamp: 13:00)

The conclusion for the legal use case is that human expertise remains essential. While AI can “get you a stack of documents,” you still need “facts people to actually tease out the insights.”

The current state of technology is an aid, not a replacement. The speaker transitions away from the legal example to a new story about building a data application, suggesting that while law is hard, structured data might offer different challenges and solutions.

23. Building Generative AI (Text-to-SQL)

(Timestamp: 13:20)

The presentation shifts to the story of “Frosty,” a company building a Text-to-SQL application. The goal is to turn natural language questions (e.g., “How many orders do I have in each state?”) into SQL code that can query a database.

This is a “very common application” for Gen AI, allowing non-technical users to interact with data. This section will focus on the engineering steps required to build this system, moving beyond the simple RAG implementation discussed previously.

24. Evaluating SQL Queries

(Timestamp: 14:14)

The first challenge in building this app is evaluation. How do you know if the AI’s generated SQL is good? The slide shows a “Gold Standard” query (the correct answer) and a “Candidate SQL” (the AI’s attempt).

In this example, the AI added an extra column (“latitude”) that wasn’t requested. While the query might still work, it isn’t an exact match. The speaker notes, “We really need to have a way to give partial credit,” because simple string matching would mark this helpful addition as a failure.

25. Model Based Evaluation

(Timestamp: 15:45)

To solve the grading problem at scale, the speaker introduces Model-Based Evaluation. This involves using an LLM (like GPT-4) to act as the “judge” for the output of another model.

Instead of humans manually grading thousands of SQL queries, “we’re going to use a large language model to do this.” This allows for nuanced grading (partial credit) that strict code comparison cannot provide.

26. Skepticism of Model Evaluation

(Timestamp: 15:59)

The speaker acknowledges the common reaction to this technique: “Is that going to work? I mean that’s like the fox guarding the outhouse.” There is a fear of “model collapse” or circular logic when AI evaluates AI.

Despite this intuition, the speaker assures the audience that this is a standard and effective practice in modern AI development, and proceeds to explain how to implement it correctly.

27. The Evaluation Prompt

(Timestamp: 16:14)

This slide reveals the system prompt used for the model-based judge. It instructs the LLM to act as a “data quality analyst” and provides a specific grading rubric (0 to 3 scale).

By explicitly defining what constitutes a “Perfect Match,” “Good Match,” or “No Match,” the engineer can control how the AI judges the output. This turns a subjective assessment into a structured, automated process.

28. The “TX” vs “TEXAS” Problem

(Timestamp: 16:50)

A specific example of why strict matching fails. The user asked for data in “Texas.” The database uses the abbreviation ‘TX’, but the AI generated a query looking for ‘TEXAS’.

“It’s a natural mistake here to confuse TX and Texas… but if we go with the strict criteria of that exact match, we don’t get an exact match.” A standard code test would fail this, even though the intent is correct and easily fixable.

29. Execution Accuracy Failure

(Timestamp: 17:16)

This slide confirms that under “Execution Accuracy” (strict matching), the query is a failure (“No Match”). This metric is too harsh for development because it obscures progress; a model that gets the logic right but misses an abbreviation is much better than one that writes gibberish.

30. Execution Score Success

(Timestamp: 17:30)

Using the Model-Based Evaluation, the same ‘TX’ vs ‘TEXAS’ error is graded differently. The “Execution Score” is a “Perfect Match” because the judge recognizes the semantic intent was captured.

“It captures the user’s intent… the user could easily fix this.” This allows developers to optimize the model for logic and reasoning first, handling minor syntax issues separately.

31. Correlation with Other Metrics

(Timestamp: 17:50)

The speaker presents data showing a strong correlation between the model-based scores and other evaluation methods. When the model judge gives a 5/5, other metrics generally agree.

This validation step is crucial. The engineer in the story checked her results and found “80% were the exact same when she scored them.” This high level of agreement gives confidence in automating the evaluation pipeline.

32. Research on Model Evaluation

(Timestamp: 18:19)

Supporting the anecdote, this slide references broader research indicating that LLMs correlate with human judges about 80% of the time regarding correctness and readability.

“I got tired of adding research sites here… universally we see that often in many contexts that these large language models correlate about 80% of the time to humans.” This establishes model-based evaluation as an industry standard.

33. Initial Benchmark Results

(Timestamp: 18:50)

After setting up the evaluation pipeline and creating an internal enterprise benchmark (not a public dataset), the initial results are poor: only 33% accuracy.

The speaker emphasizes the importance of using internal data for benchmarks: “You can’t trust those public data sets… they’re far too easy.” The low score sets the stage for the iterative engineering process required to improve the application.

34. Using Multiple Models

(Timestamp: 19:15)

The first improvement strategy is Ensembling. The engineer noticed different models had different strengths, so she combined them.

“In traditional machine learning, we often Ensemble models… she decided to try the same thing here.” By using multiple Text-to-SQL models and combining their outputs, performance improved.

35. Error Correction (Self-Reflection)

(Timestamp: 19:40)

The next optimization is Error Correction via self-reflection. When the model generates an error, the system asks the model to “reflect upon it” or think “step-by-step.”

“That actually makes the model spend more time thinking about it… and actually they can use all of that to get a better answer.” This technique, often called Chain of Thought, leverages the model’s ability to debug its own output when prompted correctly.

36. Screening Inputs

(Timestamp: 20:30)

Improving the input data is just as important as improving the model. The engineer adds a Screening layer to filter out questions that are ambiguous or irrelevant (non-SQL questions).

“She noticed that a lot of what the users were typing in just didn’t make sense.” By catching bad queries early and asking the user for clarification, the system avoids processing garbage data, thereby increasing overall success rates.

37. Feature Extraction

(Timestamp: 21:40)

Recognizing that different questions require different handling, the engineer implements Feature Extraction. A time-series question needs different context than a ranking question.

“If I’m cooking macaroni and cheese I need different ingredients than if I’m making tacos.” The system now identifies the type of question and extracts the specific features (metadata, table schemas) relevant to that type before generating SQL.

38. The Semantic Layer

(Timestamp: 22:50)

To bridge the gap between messy enterprise databases and user language, a Semantic Layer is added. This involves human experts defining the data structure in business terms.

“We’re going to use the expertise… to give us details of the data structure in a way that deals with all this confusing structure.” This layer translates business logic (e.g., what defines a “churned customer”) into a schema the AI can understand, significantly boosting accuracy.

39. Generative AI Decision App

(Timestamp: 24:10)

This flowchart represents the final, production-grade system. It is no longer just a prompt sent to a model. It includes classification, feature extraction, multiple SQL generation agents, error correction, and a semantic layer.

The lesson is that “Generative AI is not about a data scientist sitting out an Island by themselves… instead it’s building a system like this.” It requires a cross-functional team of analysts, engineers, and domain experts to build a reliable application.

40. The Future of AI

(Timestamp: 25:00)

The speaker pivots to the future, acknowledging the rapid pace of innovation from companies like OpenAI and Google DeepMind. He addresses the audience’s potential skepticism: “The future is you’re just going to be able to just take all your data cram it into one thing it’s just going to solve it all for you.”

This sets up the final section on Reasoning and Planning, moving beyond simple retrieval and text generation.

41. Can LLMs Reason and Plan? (Block World)

(Timestamp: 25:38)

To test reasoning, the speaker introduces the Block World benchmark. The task is to stack colored blocks in a specific order. This requires multi-step planning.

“You have to logically think and plan for maybe five, six, ten, even 20 steps to be able to solve it.” This tests the model’s ability to handle dependencies and sub-tasks, rather than just predicting the next word.

42. GPT-4 Planning Performance

(Timestamp: 26:40)

The results for GPT-4 are shown. While it achieves 34% on the standard Block World, its performance collapses to 3% in “Mystery World.” Mystery World is the same problem, but the block names are randomized (e.g., obfuscated).

“What you call them doesn’t matter [to a human]… but for a large language model, what you does call them matters a lot.” The collapse in performance proves the model was relying on memorized patterns (approximate reasoning) rather than true logical planning.

43. o1 Models and Progress

(Timestamp: 27:50)

The speaker updates the data with the very latest OpenAI o1 model results. This model uses “Chain of Thought on steroids” (reinforcement learning). It shows a massive improvement, jumping to nearly 100% on Block World and significantly higher on Mystery World (around 37-53%).

While this is “solid progress,” the speaker notes it “still has a ways to go.” The models are getting better at approximate reasoning, but they are not infallible logic engines yet.

44. Be Skeptical of Benchmarks

(Timestamp: 29:36)

A warning accompanies the new capabilities: Be skeptical. As models get better at approximating reasoning, their mistakes will become harder to spot. They will sound incredibly convincing even when they are logically flawed.

“You’re going to have to have an expert to be able to tell when this models are going off the rails… because the Baseline for these models is so good.” Just as legal experts were needed for RAG, domain experts are needed to verify AI reasoning.

45. Common Gen AI Use Cases Summary

(Timestamp: 30:13)

The speaker summarizes the key technical concepts covered: Hallucinations, RAG, Reasoning, Evaluation, Model as a Judge, and Data Enrichment.

These pillars form the basis of current Gen AI development. The presentation has moved from the simple idea of “asking a chatbot” to the complex reality of building systems that manage retrieval, evaluation, and reasoning.

46. Project Reality

(Timestamp: 30:35)

The final takeaway emphasizes the organizational aspect. “Generative AI is like any other project and doesn’t go as planned.” It is not magic; it is engineering.

Success requires a diverse team (“system of people”) including evaluators, analysts, and technical builders. It is an iterative process that involves handling messy data and managing expectations.

47. Closing Title

(Timestamp: 30:53)

The presentation concludes. The speaker thanks the audience, hoping the stories of the law firm and the data company provided a realistic “Practical Perspective” on the current state of Generative AI.

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

Also, you can use ChatGPT to generate the code for you.