1. Title Slide

(Timestamp: 00:00)

This slide introduces the workshop titled “Hill Climbing: Best Practices for Evaluating LLMs,” presented by Rajiv Shah, PhD, at the Open Data Science Conference (ODSC). The presentation focuses on the technical nuances of Generative AI and how to build effective evaluation workflows.

Rajiv sets the stage by outlining his three main goals for the session: understanding the technical differences in GenAI evaluation, learning a basic introductory workflow for building evaluation datasets, and inspiring practitioners to start “learning by doing” rather than just reading papers.

The concept of “Hill Climbing” refers to the iterative process of improving LLM applications—starting with a baseline and continuously optimizing performance through rigorous testing and error analysis.

2. Evaluating for Gen AI Resources

(Timestamp: 00:06)

This slide provides a QR code and a GitHub URL, directing the audience to the code and resources associated with the talk. It emphasizes that the workshop is practical, with code examples available for attendees to replicate the evaluation techniques discussed.

Rajiv encourages the audience to access these resources to follow along with the technical implementations of the concepts, such as building LLM judges and creating unit tests, which will be covered later in the presentation.

3. Customer Support Use Case

(Timestamp: 00:48)

To motivate the need for evaluation, the presentation introduces a common real-world use case: Customer Support. Generative AI is frequently deployed to help agents compose emails or chat responses based on user inquiries.

This scenario serves as the baseline example throughout the talk. It represents a high-volume task where automation is desirable, but accuracy and tone are critical for maintaining customer satisfaction and brand reputation.

4. Vibe Coding

(Timestamp: 00:59)

This slide introduces the concept of “Vibe Coding”—the initial phase where developers grab a simple prompt, feed it to a model, and get a result that feels right. It highlights the misconception that GenAI is easy because it works “out of the box” for simple demos.

Rajiv notes that while “vibe coding” might work for a quick demo app, it is insufficient for production systems. Relying on a “vibe” that the model is working prevents teams from catching subtle failures that occur at scale.

5. Good Response: Delayed Order

(Timestamp: 01:10)

Here, we see a successful output generated by the LLM. The customer inquired about a delayed order, and the AI generated a polite, relevant response acknowledging the delay and apologizing.

This example reinforces the “Vibe Coding” trap: because the model often produces high-quality, human-sounding text like this, developers can be lulled into a false sense of security regarding the system’s reliability.

6. Good Response: Damaged Product

(Timestamp: 01:12)

This slide provides another example of a “good” response. The AI correctly identifies that the customer received a damaged product and initiates a replacement protocol.

These positive examples establish a baseline of expected behavior. The challenge in evaluation is not just confirming that the model can work, but ensuring it works consistently across all edge cases.

7. Bad Response: Irrelevance

(Timestamp: 01:26)

The presentation shifts to failure modes. In this example, the user asks about an “Order Delay,” but the AI responds with information about a “New Product Launch.”

This illustrates a complete context mismatch. The model failed to attend to the user’s intent, generating a coherent but completely irrelevant response. This type of failure frustrates users and degrades trust in the automated system.

8. Bad Response: Hallucination

(Timestamp: 01:36)

This slide shows a more dangerous failure: Hallucination. The AI apologizes for a defective “espresso machine,” but as the speaker notes, “We don’t actually sell espresso machines.”

This highlights the risk of the model fabricating facts to be helpful. Such errors can lead to logistical nightmares, such as customers expecting replacements for products that do not exist or that the company never sold.

9. Risks of LLM Mistakes

(Timestamp: 01:51)

Rajiv categorizes the risks associated with LLM failures into three buckets: Reputational, Legal, and Financial. He cites the example of Cursor, an IDE company, where a support bot hallucinated a policy restricting users to one device, causing customers to cancel subscriptions.

The slide emphasizes that courts may view AI agents as employees; if a bot makes a promise (like a refund or policy change), the company might be legally bound to honor it. This escalates evaluation from a technical nice-to-have to a business necessity.

10. The Despair of Gen AI

(Timestamp: 02:38)

This visual represents the frustration developers feel when moving from a successful demo to a failing production system. The “despair” comes from the realization that the stochastic nature of LLMs makes them difficult to control.

It serves as an emotional anchor for the audience, acknowledging that while GenAI is exciting, the unpredictability of its failures causes significant stress for engineering teams responsible for deployment.

11. High Failure Rates

(Timestamp: 02:48)

The slide cites an MIT report stating that “95% of GenAI pilots are failing.” While Rajiv notes this number might be overstated, it reflects a trend where executives are demanding ROI and seeing lackluster results.

This shift in 2025 means that evaluation is no longer just for debugging; it is required to prove business value and justify the high costs of running Generative AI infrastructure.

12. Evaluation Improves Applications

(Timestamp: 03:14)

This slide asserts the core thesis: Evaluation helps you build better GenAI applications. It references a previous viral video by the speaker on the same topic, positioning this talk as an updated, condensed version with fresh content.

Rajiv explains that you cannot improve what you cannot measure. Without a robust evaluation framework, developers are essentially guessing whether changes to prompts or models are actually improving performance.

13. Why Evaluation is Necessary

(Timestamp: 03:40)

This concentric diagram illustrates the stakeholders involved in evaluation. It starts with “Things Go Wrong” (technical reality), moves to “Buy-in” (convincing managers/teams), and ends with “Regulators” (external compliance).

Evaluation serves multiple audiences: it helps the developer debug, it provides the metrics needed to convince management that the app is production-ready, and it creates the audit trails required by third-party auditors or regulators.

14. Evaluation Dimensions

(Timestamp: 04:18)

Evaluation must cover three dimensions: Technical (F1 scores, accuracy), Business (ROI, value generated), and Operational (Total Cost of Ownership, latency).

Rajiv highlights that data scientists often focus solely on the technical, but ignoring operational costs (like the expense of hosting GPUs vs. using APIs) can kill a project. A comprehensive evaluation strategy considers the cost-to-quality ratio.

15. Public Benchmarks

(Timestamp: 05:06)

The slide discusses Public Benchmarks (like MMLU, GSM8K). While useful for a general idea of a model’s capabilities (e.g., “Is Llama 3 better than Llama 2?”), they are insufficient for specific applications.

Rajiv warns against using these benchmarks to determine if a model fits your specific use case. Companies promote these numbers for marketing, but they rarely reflect performance on proprietary business data.

16. Custom Benchmarks

(Timestamp: 05:22)

The solution to the limitations of public benchmarks is Custom Benchmarks. This slide defines a benchmark as a combination of a Task, a Dataset, and an Evaluation Metric.

This is a critical definition for the workshop. To “tame” GenAI, you must build a dataset that reflects your specific customer queries and define success metrics that matter to your business logic, rather than relying on generic academic tests.

17. Taming Gen AI

(Timestamp: 05:28)

This title slide signals a transition into the technical “how-to” section of the talk. “Taming” implies that the default state of GenAI is wild and unpredictable.

The goal of the following sections is to bring structure and control to this chaos through rigorous engineering practices and evaluation workflows.

18. Workshop Roadmap

(Timestamp: 05:31)

The roadmap outlines the four main sections of the talk: 1. Basics of Gen AI: Understanding variability and technical nuances. 2. Evaluation Workflow: Building the dataset and running the first tests. 3. More Complexity: Adding unit tests and conducting error analysis. 4. Agents: Evaluating complex, multi-step workflows.

19. Variability in Responses

(Timestamp: 06:00)

This slide visually demonstrates the Non-Determinism of LLMs. It shows two responses to the same prompt generated just minutes apart. While substantively similar, the wording and structure differ slightly.

This variability makes exact string matching (a common software testing technique) impossible for LLMs. It necessitates semantic evaluation techniques, which complicates the testing pipeline.

20. Input-Model-Output Diagram

(Timestamp: 06:24)

A simple diagram illustrates the flow: Prompt -> Model -> Output. Rajiv uses this to structure the analysis of where variability comes from.

He explains that “chaos” can enter the system at any of these three stages: the input (prompt sensitivity), the model (inference non-determinism), or the output (formatting and evaluation).

21. Inconsistent Benchmark Scores

(Timestamp: 06:44)

The slide presents a discrepancy between benchmark scores tweeted by Hugging Face and those in the official Llama paper. Both used the same dataset (MMLU), but reported different accuracy numbers.

This introduces the problem of Evaluation Harness Sensitivity. Even with standard benchmarks, how you ask the model to take the test changes the score, proving that evaluation is fragile and implementation-dependent.

22. MMLU Overview

(Timestamp: 07:25)

MMLU (Massive Multitask Language Understanding) is explained here. It is a multiple-choice test covering 57 tasks across STEM, the humanities, and more.

It is currently the standard for measuring general “intelligence” in models. However, because it is a multiple-choice format, it is susceptible to prompt formatting nuances, as the next slides demonstrate.

23. Prompt Sensitivity

(Timestamp: 07:44)

This slide reveals why the scores in Slide 21 differed. The three evaluation harnesses used slightly different prompt structures (e.g., using the word “Question” vs. just listing the text).

These minor changes resulted in significant accuracy shifts. This proves that LLMs are highly sensitive to syntax, meaning a “better” model might just be one that was prompted more effectively for the test, not one that is actually smarter.

24. Formatting Changes

(Timestamp: 08:22)

Expanding on sensitivity, this slide references Anthropic’s research showing that changing answer choices from (A) to [A] or (1) affects the output.

This level of fragility is a key takeaway: seemingly cosmetic changes in how inputs are formatted can alter the model’s reasoning capabilities or its ability to output the correct token.

25. GPT-4o Performance Drop

(Timestamp: 08:38)

A bar chart demonstrates that this issue persists even in state-of-the-art models like GPT-4o. Subtle changes in wording can lead to a 5-10% drop in performance.

This counters the assumption that newer, larger models have “solved” prompt sensitivity. It remains a persistent variable that evaluators must control for.

26. Tone Sensitivity

(Timestamp: 08:46)

This slide shows that the tone of a prompt (e.g., being polite vs. direct) affects accuracy. Rajiv jokes, “I guess this is why mom always said to be polite.”

The graph indicates that prompt engineering strategies, like adding emotional weight or politeness, can statistically alter model performance, adding another layer of complexity to evaluation.

27. Persistent Sensitivity

(Timestamp: 09:00)

The slide reiterates that despite years of progress, models are still sensitive to specific phrases. It shows a “Prompt Engineering” guide suggesting specific words to use.

The takeaway is that developers cannot treat the prompt as a static instruction; it is a hyperparameter that requires optimization and constant testing.

28. Falcon LLM Bias

(Timestamp: 09:18)

This slide introduces a case study with the Falcon LLM. A user tweet shows the model recommending Abu Dhabi as a technological city with glowing sentiment, which raised suspicions about bias given the model’s origin in the Middle East.

This serves as a detective story: users wondered if the model weights were altered or if specific training data was injected to force this positive association.

29. Potential Cover-up?

(Timestamp: 09:50)

Another tweet speculates if the model is “covering up human rights abuses” because it provides different answers for Abu Dhabi compared to other cities.

This highlights how model behavior can be misinterpreted as malicious bias or censorship, when the root cause might be something much simpler in the input stack.

30. Inspecting the System Prompt

(Timestamp: 10:00)

The reveal: The bias wasn’t in the weights, but in the System Prompt. The slide suggests looking at the hidden instructions given to the model.

In Falcon’s case, the system prompt explicitly told the model, “You are a model built in Abu Dhabi.” This context influenced its generation probabilities, causing it to favor Abu Dhabi in its responses.

31. Claude System Prompt

(Timestamp: 10:33)

Rajiv points out that most developers never read the system prompts of the models they use. He highlights the Claude System Prompt, which is 1700 words long and takes nearly 10 minutes to read.

These extensive instructions define the model’s personality and safety guardrails. Ignoring them means you don’t fully understand the inputs driving your application’s behavior.

32. Complexity of a Single Response

(Timestamp: 11:00)

The diagram is updated to show that a “single response” is actually the result of complex interactions: Tokenization -> Prompt Styles -> Prompt Engineering -> System Prompt.

This visual summarizes the “Input” section of the talk, reinforcing that before the model even processes data, multiple layers of text transformation occur that can alter the result.

33. Inter-text Similarity

(Timestamp: 11:15)

This heatmap compares Inter-text similarity between models. It highlights Llama 70B and Llama 8B. Even though they are from the same family and likely trained on similar data, they are not identical.

This means you cannot swap a smaller model for a larger one (or vice versa) and expect the exact same behavior. Any model change requires a full re-evaluation.

34. Sycophantic Models

(Timestamp: 12:16)

The slide discusses Sycophancy—the tendency of models to agree with the user even when the user is wrong. It mentions how early versions of GPT-4 were sometimes “overly nice.”

This behavior is a specific type of model bias that evaluators must watch for. If a user asks a leading question containing false premises, a sycophantic model might validate the falsehood rather than correct it.

35. Model Drift

(Timestamp: 12:37)

“Model Drift” refers to the phenomenon where commercial APIs (like OpenAI or Anthropic) change their model behavior over time without warning.

Because developers do not control the weights of API-based models, the “ground underneath them” can shift. A prompt that worked yesterday might fail today because the provider updated the backend or the inference infrastructure.

36. Degraded Responses Timeline

(Timestamp: 12:55)

This slide shows a timeline of Degraded Responses from an Anthropic incident. Technical issues like context window routing errors led to corrupted outputs for a period of days.

This illustrates that drift isn’t always about model updates; it can be infrastructure failures. Continuous monitoring is required to detect when an external dependency degrades your application’s performance.

37. Hyperparameters

(Timestamp: 13:33)

The slide lists Hyperparameters like Temperature, Top-P, and Max Length. Rajiv explains that users can control these “knobs” to influence creativity versus determinism.

Setting temperature to 0 makes the model less random, but as the next slides show, it does not guarantee perfect determinism due to hardware nuances.

38. Non-Deterministic Inference

(Timestamp: 14:03)

This slide tackles Non-Deterministic Inference. Unlike traditional ML models (e.g., XGBoost) where a fixed seed guarantees identical output, LLMs on GPUs often produce different results for identical inputs.

Causes include floating-point accumulation errors and the behavior of Mixture of Experts (MoE) models where different batches might activate different experts.

39. Addressing Non-Determinism

(Timestamp: 15:11)

Rajiv references recent work by Thinking Machines and updates to vLLM that attempt to solve the non-determinism problem through correct batching.

While solutions are emerging, the takeaway is that most current setups are non-deterministic by default. Evaluators must design their tests to tolerate this variance rather than expecting bit-wise reproducibility.

40. Updated Model Diagram

(Timestamp: 15:43)

The diagram expands again. The “Model” box now includes Model Selection, Hyperparameters, Non-deterministic Inference, and Forced Updates.

This visual summarizes the “Model” section, showing that the “black box” is actually a dynamic system with internal variables (weights/architecture) and external variables (infrastructure/updates) that all add noise to the output.

41. Output Format Issues

(Timestamp: 16:01)

Moving to the “Output” stage, this slide uses MMLU again to show how Output Formatting affects evaluation. How do you ask the model to answer a multiple-choice question?

Do you ask it to output just the letter “A”? Or the full text? Or the probability of the token “A”? Different evaluation harnesses use different methods, leading to the score discrepancies seen earlier.

42. Evaluation Harness Variations

(Timestamp: 16:35)

This table details the specific differences in implementation between harnesses (e.g., original MMLU vs. HELM vs. EleutherAI).

It reinforces that there is no standard “ruler” for measuring LLMs. The tool you use to measure the model introduces its own bias and variance into the final score.

43. Score Comparison Table

(Timestamp: 16:56)

A spreadsheet shows the same models scoring differently across different evaluation implementations. The variance is not trivial; it can be large enough to change the ranking of which model is “best.”

This data drives home the point: You must control your own evaluation pipeline. Relying on reported numbers is risky because you don’t know the implementation details behind them.

44. Sentiment Analysis Variance

(Timestamp: 17:09)

This slide shows varying Sentiment Analysis outputs. Different models (or the same model with different prompts) might classify a review as “Positive” while another says “Neutral.”

This introduces the concept that even “simple” classification tasks in GenAI are subject to interpretation and variance, unlike traditional classifiers that have a fixed decision boundary.

45. Tool Use Variance

(Timestamp: 17:23)

Radar charts illustrate variance in Tool Use. Models might be good at using an “Email” tool but fail at “Calendar” or “Terminal” tools.

Furthermore, models exhibit non-determinism in decision making—sometimes they choose to use a tool, and sometimes they try to answer from memory. This adds a layer of logic errors on top of text generation errors.

46. Summary: Why Responses Differ

(Timestamp: 17:49)

This comprehensive slide aggregates all the factors discussed: Inputs (prompts, system prompts), Model (drift, hyperparams), Outputs (formatting), and Infrastructure.

It serves as a checklist for the audience. If your application is behaving inconsistently, investigate these specific layers of the stack to find the source of the noise.

47. Chaos is Okay

(Timestamp: 18:17)

Rajiv reassures the audience that “Chaos is Okay.” The slide presents a chart of evaluation methods ranging from flexible/expensive (human eval) to rigid/cheap (code assertions).

The message is that while the technology is chaotic, there is a spectrum of tools available to manage it. We don’t need to solve every source of variance; we just need a robust process to measure it.

48. From Chaos to Control

(Timestamp: 18:27)

This transition slide marks the beginning of the Evaluation Workflow section. The presentation shifts from describing the problem to prescribing the solution.

The goal here is to move from “Vibe Coding” to a structured engineering discipline where changes are measured against a stable baseline.

49. Build the Evaluation Dataset

(Timestamp: 18:37)

The first step in the workflow is to Build the Evaluation Dataset. The slide lists examples of prompts for tasks like summarization, extraction, and translation.

Rajiv emphasizes that this dataset should reflect your actual use case. It is the foundation of the “Custom Benchmark” concept introduced earlier.

50. Get Labeled Outputs (Gold)

(Timestamp: 18:46)

Step two is to get Labeled Outputs, also known as Gold Outputs, Reference, or Ground Truth. The slide adds a column showing the ideal answer for each prompt.

This is the standard against which the model will be judged. While obtaining these labels can be expensive (requiring human effort), they are essential for calculating accuracy.

51. Compare to Model Output

(Timestamp: 19:00)

Step three is to generate responses from your system and place them alongside the Gold Outputs. The slide adds a “Model Output” column.

This visual comparison allows developers (and automated judges) to see the delta between what was expected and what was produced.

52. Measure Equivalence

(Timestamp: 19:10)

Step four is to Measure Equivalence. Since LLMs rarely produce exact string matches, we use an LLM Judge (another model) to determine if the Model Output means the same thing as the Gold Output.

The slide shows a prompt for the judge: “Are these two responses semantically equivalent?” This converts a fuzzy text comparison problem into a binary (Pass/Fail) metric.

53. Optimize Using Equivalence

(Timestamp: 19:57)

Once you have an equivalence metric, you can Optimize. The slide shows Config A vs. Config B. By changing prompts or models, you can track if your “Equivalence Score” goes up or down.

This treats GenAI engineering like traditional hyperparameter tuning. The goal is to maximize the equivalence score on your custom dataset.

54. Why Global Metrics Aren’t Enough

(Timestamp: 20:28)

The slide discusses the limitations of the “Equivalence” approach. While good for a general sense of quality, Global Metrics miss nuances.

Sometimes it’s hard to get a Gold Answer for open-ended creative tasks. Furthermore, a simple “Pass/Fail” doesn’t tell you why the model failed (e.g., was it tone, length, or factuality?).

55. From Global to Targeted Evaluation

(Timestamp: 20:55)

This slide argues for Targeted Evaluation. To maximize performance, you need to dig deeper into the data and identify specific error modes.

This transitions the talk from “Basic Workflow” to “Advanced Testing,” where we break down “Quality” into specific, testable components like tone, length, and safety.

56. Building Tests

(Timestamp: 21:14)

The section title “Building Tests” appears. This is where the presentation moves into the “Unit Testing” philosophy for GenAI.

Just as software engineering relies on unit tests to verify specific functions, GenAI engineering should use targeted tests to verify specific attributes of the generated text.

57. Good vs. Bad Examples

(Timestamp: 21:20)

The slide displays a Good Example and a Bad Example of a response. The bad example is visibly shorter and less polite.

Rajiv asks the audience to identify why it is bad. This exercise is crucial: you cannot build a test until you can articulate exactly what makes a response a failure.

58. Develop an Evaluation Mindset

(Timestamp: 21:46)

To define “Bad,” developers need an Evaluation Mindset. This involves observing real-world user interactions and problems.

Data scientists often want to stay in their “chair” and optimize algorithms, but Rajiv argues that effective evaluation requires understanding the user’s pain points.

59. Collaborate with Experts

(Timestamp: 21:58)

The slide stresses Collaboration. You must talk to domain experts (e.g., the customer support team) to define what a “good” answer looks like.

Naive bootstrapping—pretending to be a user—is a good start, but long-term success requires input from the people who actually know the business domain.

60. Identify and Categorize Failures

(Timestamp: 22:52)

Once you understand the domain, you can Categorize Failure Types. The slide shows a chart grouping errors into categories like “Harmful Content,” “Bias,” or “Incorrect Info.”

This clustering allows you to see patterns. Instead of just knowing “the model failed 20% of the time,” you know “the model has a specific problem with tone.”

61. Define What Good Looks Like

(Timestamp: 23:11)

Using the categorization, you can explicitly Define What Good Looks Like. The slide contrasts the good/bad examples again, but now with labels: “Too short,” “Lacks professional tone.”

This transforms a subjective feeling (“this response sucks”) into objective criteria (“response must be >50 words and use polite honorifics”).

62. Document Every Issue

(Timestamp: 23:32)

The slide shows a spreadsheet where humans evaluate responses and Document Every Issue. Columns track specific attributes like “Is it helpful?” or “Is the tone right?”

This manual annotation is the training data for your automated tests. You need humans to establish the ground truth before you can automate the checking.

63. Evaluation Tooling

(Timestamp: 23:53)

Rajiv mentions that Tooling Can Help. The slide shows a custom chat viewer designed to make human review easier.

However, he warns against getting sidetracked by building fancy tools. Simple spreadsheets often suffice for the early stages. The goal is the data, not the interface.

64. Test 1: Length Check

(Timestamp: 24:05)

Now we build the automated tests. Test 1 is a Length Check. The slide shows Python code asserting that the word count is between 8 and 200.

This is a deterministic test. You don’t need an LLM to count words. Rajiv encourages using simple Python assertions wherever possible because they are fast, cheap, and reliable.

65. Test 2: Tone and Style

(Timestamp: 24:22)

Test 2 checks Tone and Style. Since “tone” is subjective, we use an LLM Judge (OpenAI model) to classify the response.

The prompt asks the judge to identify the style. This allows us to automate the “vibe check” that humans were previously doing manually.

66. Adding Metrics to Documentation

(Timestamp: 24:41)

The spreadsheet is updated with new columns: Length_OK and Tone_OK. These are the results of the automated tests.

Now, for every row in the dataset, we have granular pass/fail metrics. This helps pinpoint exactly why a specific response failed, rather than just a generic failure.

67. Check Judges Against Humans

(Timestamp: 25:12)

A critical step: Check LLM Judges Against Humans. You must verify that your automated “Tone Judge” agrees with your human experts.

If the human says the tone is rude, but the LLM Judge says it’s polite, your metric is useless. You must iterate on the judge’s prompt until alignment is high.

68. Self-Evaluation Bias

(Timestamp: 26:06)

The slide illustrates Self-Evaluation Bias. LLMs tend to rate their own outputs higher than outputs from other models. GPT-4 prefers GPT-4 text.

To mitigate this, Rajiv suggests mixing models—use Claude to judge GPT-4, or Gemini to judge Claude. This helps ensure a more neutral evaluation.

69. Alignment Checks

(Timestamp: 26:46)

This slide reinforces the need for Continuous Alignment. Just because your judge aligned with humans last month doesn’t mean it still does (due to model drift).

Human spot-checks should be a permanent part of the pipeline to ensure the automated judges haven’t drifted.

70. Biases in LLM Judges

(Timestamp: 27:02)

The slide lists known Biases in LLM Judges, such as Position Bias (favoring the first answer presented) or Verbosity Bias (favoring longer answers).

Evaluators must be aware of these. For example, you should shuffle the order of answers when asking a judge to compare two options to cancel out position bias.

71. Best Practices for LLM Judges

(Timestamp: 27:11)

A summary of Best Practices: Calibrate with human data, use ensembles (multiple judges), avoid asking for “relevance” (too vague), and use discrete rating scales (1-5) rather than continuous numbers.

These tips help stabilize the inherently noisy process of using AI to evaluate AI.

72. Error Analysis Chart

(Timestamp: 27:46)

With tests in place, we move to Error Analysis. The bar chart shows the number of failed cases categorized by error type (Length, Tone, Professional, Context).

This visualization tells you where to focus your efforts. If “Tone” is the biggest bar, you work on the system prompt’s tone instructions. If “Context” is the issue, you might need better Retrieval Augmented Generation (RAG).

73. Comparing Prompts

(Timestamp: 27:58)

The chart can compare Prompt A vs. Prompt B. This allows for A/B testing of prompt engineering strategies.

You can see if a new prompt improves “Tone” but accidentally degrades “Context.” This tradeoff analysis is impossible with a single global score.

74. Explanations Guide Improvement

(Timestamp: 28:14)

Rajiv suggests asking the LLM Judge for Explanations. Don’t just ask for a score; ask for “one sentence explaining why.”

These explanations act as metadata that helps developers understand the judge’s reasoning, making it easier to debug discrepancies between human and AI judgments.

75. Limits to Explanations

(Timestamp: 28:35)

A warning: Explanations are not causal. When an LLM explains why it did something, it is generating a plausible justification, not a trace of its actual neural activations.

Treat explanations as a heuristic or a helpful hint, not as absolute truth about the model’s internal state.

76. The Evaluation Flywheel

(Timestamp: 28:46)

The Evaluation Flywheel describes the iterative cycle: Build Eval -> Analyze -> Improve -> Repeat.

This concept, credited to Hamill, emphasizes that evaluation is not a one-time event but a continuous loop that spins faster as you gather more data and build better tests.

77. Financial Analyst Agent Example

(Timestamp: 29:20)

To demonstrate advanced unit testing, Rajiv introduces a Financial Analyst Agent. The goal is to assess the specific “style” of a financial report.

This is a complex domain where “good” is highly specific (regulated, precise, risk-aware), making it a perfect candidate for granular unit tests.

78. Use a Global Test?

(Timestamp: 29:43)

You could use a Global Test: “Was this explained as a financial analyst would?”

While simple, this test is opaque. If it fails, you don’t know if it was because of compliance issues, lack of clarity, or poor formatting.

79. Global vs. Unit Tests

(Timestamp: 29:54)

The slide contrasts the Global approach with Unit Tests. Instead of one question, we ask six: Context, Clarity, Precision, Compliance, Actionability, and Risks.

This breakdown allows for targeted debugging. You might find the model is great at “Clarity” but terrible at “Compliance.”

80. Scoring Radar Chart

(Timestamp: 30:16)

A Radar Chart visualizes the unit test scores. This allows for a quick visual assessment of the model’s profile.

It facilitates comparison: you can overlay the profiles of two different models to see which one has the better balance of attributes for your specific needs.

81. Analyzing Failures with Clusters

(Timestamp: 30:37)

With enough unit test data, you can use Clustering (e.g., K-Means) to group failures. The slide shows clusters like “Synthesis,” “Context,” and “Hallucination.”

This moves error analysis from reading individual logs to analyzing aggregate trends, helping you prioritize which class of errors to fix first.

82. Designing Good Unit Tests

(Timestamp: 30:52)

Advice on Designing Unit Tests: Keep them focused (one concept per test), use unambiguous language, and use small rating ranges.

Good unit tests are the building blocks of a reliable evaluation pipeline. If the tests themselves are noisy or vague, the entire system collapses.

83. Examples of Unit Tests

(Timestamp: 30:55)

The slide lists specific examples of tests for Legal (Compliance, Terminology), Retrieval (Relevance, Completeness), and Bias/Fairness.

This serves as a menu of options for the audience, showing that unit tests can cover almost any dimension of quality required by the business.

84. Evaluating New Prompts

(Timestamp: 30:58)

A bar chart shows how unit tests are used to Evaluate New Prompts. By running the full suite of unit tests on a new prompt, you get a “scorecard” of its performance.

This data-driven approach removes the guesswork from prompt engineering.

85. Tools - No Silver Bullet

(Timestamp: 31:02)

Rajiv reminds the audience that Tools are No Silver Bullet. You must master the basics (datasets, metrics) first.

He advises logging traces and experiments and practicing Dataset Versioning. Tools facilitate these practices, but they cannot replace the fundamental engineering discipline.

86. Forest and Trees

(Timestamp: 31:04)

An analogy helps structure the analysis: Forest (Global/Integration) vs. Trees (Test Case/Unit Tests).

You need to look at both. The forest tells you the overall health of the app, while the trees tell you specifically what needs pruning or fixing.

87. Change One Thing at a Time

(Timestamp: 31:17)

A crucial scientific principle: Change One Thing at a Time. With so many knobs (prompt, temp, model, RAG settings), changing multiple variables simultaneously makes it impossible to know what caused the improvement (or regression).

Isolate your variables to conduct valid experiments.

88. Error Analysis Tips

(Timestamp: 31:32)

A summary of Error Analysis Tips: Use ablation studies (removing parts to see impact), categorize failures, save interesting examples, and leverage logs/traces.

These are the daily habits of successful GenAI engineers.

89. The Evaluation Story

(Timestamp: 32:08)

The slide shows the “Story We Tell”—a linear graph of improvement over time. This is the idealized version of progress often presented in case studies.

It suggests a smooth journey from “Out of the box” to “Specialized” to “User Feedback.”

90. The Reality of Progress

(Timestamp: 32:24)

The Reality is a messy, non-linear graph. You take two steps forward, one step back. Sometimes an “improvement” breaks the model.

Rajiv encourages resilience. Experienced practitioners know that this messy graph is normal and that sticking to the process eventually yields results.

91. Continual Process

(Timestamp: 33:01)

Evaluation is a Continual Process. It involves Problem ID, Data Collection, Optimization, User Acceptance Testing (UAT), and Updates.

Crucially, UAT is your holdout set. Since you don’t have a traditional test set in GenAI, your real users act as the final validation layer.

92. Eating the Elephant

(Timestamp: 34:03)

The metaphor “How do you eat an elephant?” addresses the overwhelming nature of building a comprehensive evaluation suite.

The answer, of course, is “one bite at a time.” You don’t need 100 tests on day one.

93. Adding Tests Over Time

(Timestamp: 34:10)

The slide visualizes the “elephant” being broken down into bites. You start with a few critical tests. As the app matures and you discover new failure modes, you add more tests.

Six months in, you might have 100 tests, but you built them incrementally. This makes the task manageable.

94. Doing Evaluation the Right Way

(Timestamp: 34:39)

A summary slide listing best practices: Annotated Examples, Systematic Documentation, Continuous Error Analysis, Collaboration, and awareness of Generalization.

This concludes the core methodology section of the talk.

95. Agentic Use Cases

(Timestamp: 34:50)

The final section covers Agentic Use Cases, symbolized by a dragon. Agents add a layer of complexity because the model is now making decisions (routing, tool use) rather than just generating text.

This “agency” makes the system harder to track and evaluate.

96. Crossing the River

(Timestamp: 35:06)

A conceptual slide asking, “How should it cross the river?” (Fly, Swim, Bridge?). This represents the decision-making step in an agent.

Evaluating an agent requires evaluating how it made the decision (the router) separately from how well it executed the action.

97. Chat-to-Purchase Router

(Timestamp: 35:22)

A complex flowchart shows a Chat-to-Purchase Router. The agent must decide if the user wants to search for a product, get support, or track a package.

Rajiv suggests breaking this down: evaluate the Router component first (did it pick the right path?), then evaluate the specific workflow (did it track the package correctly?).

98. Text to SQL Agent

(Timestamp: 36:17)

Another example: Text to SQL Agent. This workflow involves classification, feature extraction, and SQL generation.

You can isolate the “Classification” step (is this a valid SQL question?) and build a test just for that, before testing the actual SQL generation.

99. Evaluating Office-Style Agents

(Timestamp: 36:46)

The slide discusses OdysseyBench, a benchmark for office tasks. It highlights failure modes like “Failed to create folder” or “Failed to use tool.”

Evaluating agents involves checking if they successfully manipulated the environment (files, APIs), which is a functional test rather than a text similarity test.

100. Error Analysis for Agents

(Timestamp: 37:00)

Error Analysis for Agentic Workflows requires assessing the overall performance, the routing decisions, and the individual steps.

It is the same “action error analysis” process but applied recursively to every node in the agent’s decision tree.

101. Evaluating Workflow vs. Response

(Timestamp: 37:19)

This slide distinguishes between evaluating a Response (text) and a Workflow (process). The flowchart shows a conversational flow.

Evaluating a workflow might mean checking if the agent successfully moved the user from “Greeting” to “Resolution,” regardless of the exact words used.

102. Agentic Frameworks

(Timestamp: 37:48)

Rajiv warns that “Agentic Frameworks Help – Until They Don’t.” Frameworks (like LangChain or AutoGen) are great for demos because they abstract complexity.

However, in production, these abstractions can break or become outdated. He often recommends using straight Python for production agents to maintain control and reliability.

103. Abstraction for Workflows

(Timestamp: 38:32)

The slide illustrates the trade-off in Abstraction. You can build rigid workflows (orchestration) where you control every step, or use general agents where the LLM decides.

Orchestration is more reliable but rigid. General agents are flexible but prone to non-deterministic errors.

104. When Abstractions Break

(Timestamp: 38:53)

Model providers are training models to handle workflows internally (removing the need for external orchestration).

However, until models are perfect, developers often need to break tasks down into specific pieces to ensure reliability. The choice between “letting the model do it” and “scripting the flow” depends on the application’s risk tolerance.

105. Lessons from Agent Benchmarks

(Timestamp: 39:15)

The slide lists Lessons from Reproducing Agent Benchmarks: Standardize evaluation, measure efficiency, detect shortcuts, and log real behavior.

These are advanced tips for those pushing the boundaries of what agents can do.

106. Conclusion

(Timestamp: 39:27)

The final slide, “We did it!”, concludes the presentation. Rajiv thanks the audience and provides the QR code again.

His final message is one of empowerment: he hopes the audience now has the confidence to go out, build their own evaluation datasets, and start “hill climbing” their own applications.

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

1. Title Slide

(Timestamp: 00:00)

The presentation begins with the title slide, introducing the core theme: “From Vectors to Agents: Managing RAG in an Agentic World.” The speaker, Rajiv Shah from Contextual, sets the stage for a technical deep dive into Retrieval-Augmented Generation (RAG).

He outlines the agenda, promising to move beyond basic RAG concepts to focus specifically on retrieval approaches. The talk is designed to cover the spectrum from traditional methods like BM25 and Language Models to the emerging field of Agentic Search.

2. ACME GPT

(Timestamp: 00:40)

This slide displays a stylized logo for “ACME GPT,” representing the typical enterprise aspiration. Companies see tools like ChatGPT and immediately want to apply that capability to their internal data, asking questions like, “Can I get the list of board of directors?”

However, the speaker notes a common hurdle: generic models don’t know enterprise-specific knowledge. This sets up the necessity for RAG—injecting private data into the model—rather than relying solely on the model’s pre-trained knowledge.

3. Building RAG is Easy

(Timestamp: 01:10)

The speaker illustrates the deceptively simple workflow of a basic RAG demo. The diagram shows the standard path: a user query is converted to vectors, matched against a database, and sent to an LLM.

Shah acknowledges that building a “hello world” version of this is trivial. He notes, “You can build a very easy RAG demo out of the box by just grabbing some data, using an embedding model, creating vectors, doing the similarity.”

4. Building RAG is Easy (Code Example)

(Timestamp: 01:22)

A Python code snippet using LangChain is displayed to reinforce how accessible basic RAG has become. The code demonstrates loading a document, chunking it, and setting up a retrieval chain in just a few lines.

This slide serves as a foil for the upcoming reality check. While the code works for a demo, it hides the immense complexity required to make such a system robust, accurate, and scalable in a real-world production environment.

5. RAG Reality Check

(Timestamp: 01:35)

The tone shifts to the challenges of production. The slide highlights a sobering statistic: 95% of Gen AI projects fail to reach production. The speaker details the specific reasons why demos fail when scaled: poor accuracy, unbearable latency, scaling issues with millions of documents, and ballooning costs.

He emphasizes a critical, often overlooked factor: Compliance. “Inside an enterprise, not everybody gets to read every document.” A demo ignores entitlements, but a production system cannot.

6. Maybe try a different RAG?

(Timestamp: 03:00)

This slide lists a dizzying array of RAG variants (GraphRAG, RAPTOR, CRAG, etc.) and retrieval techniques. It represents the “analysis paralysis” developers face when scouring arXiv papers for a solution to their accuracy problems.

Shah warns against blindly chasing the latest academic paper to fix fundamental system issues. “The answer is not in here of pulling together like a bunch of archive papers.” Instead, he advocates for a structured framework to make decisions.

7. Ultimate RAG Solution

(Timestamp: 03:30)

A humorous cartoon depicts a “Rube Goldberg” machine, representing the “Ultimate RAG Solution.” It mocks the tendency to over-engineer systems with too many interconnected, fragile components in the pursuit of performance.

The speaker uses this visual to argue for simplicity and deliberate design. The goal is to avoid building a monstrosity that is impossible to maintain, urging the audience to think about trade-offs before complexity.

8. RAG as a system

(Timestamp: 03:35)

The speaker introduces a clean system architecture for RAG, broken into four distinct stages: Parsing, Querying, Retrieving, and Generation. This framework serves as the mental map for the rest of the presentation.

He highlights that “Parsing” is vastly overlooked—getting information out of complex documents cleanly is a prerequisite for success. Today’s talk, however, will zoom in specifically on the Retrieving and Querying components.

9. Designing a RAG Solution

(Timestamp: 04:10)

This slide presents a “Tradeoff Triangle” for RAG, balancing Problem Complexity, Latency, and Cost. The speaker advises having a serious conversation with stakeholders about these constraints before writing code.

A key concept introduced here is the “Cost of a mistake.” In coding assistants, a mistake is low-cost (the developer fixes it). In medical RAG systems, the cost of a mistake is high (life or death), which dictates a completely different architectural approach.

10. RAG Considerations

(Timestamp: 05:30)

A detailed table breaks down specific considerations that influence RAG design, such as domain difficulty, multilingual requirements, and data quality. This slide was originally created for sales teams to help scope customer problems.

Shah emphasizes that understanding the nuances of the use case upfront saves heartache later. For instance, knowing if users will ask simple questions or require complex reasoning changes the retrieval strategy entirely.

11. Consider Query Complexity

(Timestamp: 06:15)

The speaker categorizes queries by complexity, ranging from simple Keywords (“Total Revenue”) to Semantic variations (“How much bank?”), to Multi-hop reasoning, and finally Agentic scenarios.

He points out a common failure mode: “The answers aren’t in the documents… all of a sudden they’re asking for knowledge that’s outside.” Recognizing the query complexity determines whether you need a simple search engine or a complex agentic workflow.

12. Retrieval (Highlighted)

(Timestamp: 07:32)

The presentation zooms back into the system diagram, highlighting the “Retrieving” box. This signals the start of the deep technical dive into retrieval algorithms.

Shah notes that this area causes the most confusion due to the sheer number of model choices and architectures available. He aims to provide a practical guide to selecting the right retrieval tool.

13. Retrieval Approaches

(Timestamp: 08:16)

Three primary retrieval pillars are introduced: 1. BM25: The lexical, keyword-based standard. 2. Language Models: Semantic embeddings and vector search. 3. Agentic Search: The new frontier of iterative reasoning.

The speaker emphasizes that documents must be broken into pieces (chunking) because no single model context window is efficient enough to hold all enterprise data for every query.

14. Building RAG is Easy (Code Highlight)

(Timestamp: 08:50)

Returning to the initial code snippet, the speaker highlights the vectorstore and retriever initialization lines. This pinpoints exactly where the upcoming concepts fit into the implementation.

This visual anchor helps developers map the theoretical concepts of BM25 and Embeddings back to the actual lines of code they write in libraries like LangChain or LlamaIndex.

15. BM25

(Timestamp: 09:18)

BM25 (Best Match 25) is explained as a probabilistic lexical ranking function. The slide visualizes an inverted index, mapping words (like “butterfly”) to the specific documents containing them.

Shah explains that this is the 25th iteration of the formula, designed to score documents based on word frequency and saturation. It remains a powerful, fast baseline for retrieval.

16. BM25 Performance

(Timestamp: 09:55)

A table compares the speed of a Linear Scan (Ctrl+F style) versus an Inverted Index (BM25) as the document count grows from 1,000 to 9,000.

The data shows that linear search becomes exponentially slower (taking 3,000 seconds for 1k documents in this synthetic test), while BM25 remains orders of magnitude faster. This efficiency is why lexical search is still widely used in production.

17. BM25 Failure Cases

(Timestamp: 11:08)

The limitations of BM25 are exposed. Because it relies on exact word matches, it fails when users use synonyms. If a user searches for “Physician” but the documents only contain “Doctor,” BM25 will return zero results.

Similarly, it struggles with acronyms like “IBM” vs “International Business Machines.” Despite this, Shah argues BM25 is a “very strong baseline” that often beats complex neural models on specific keyword-heavy datasets.

18. Hands on: BM25s

(Timestamp: 12:14)

For developers wanting to implement this, the slide points to a library called bm25s, a high-performance Python implementation available on Hugging Face.

This reinforces the practical nature of the talk—BM25 isn’t just a legacy concept; it is an active, installable tool that developers should consider using alongside vector search.

19. Enter Language Models

(Timestamp: 12:24)

The talk transitions to Language Models (Embeddings). The slide explains how an encoder model turns text into a dense vector (a list of numbers) that captures semantic meaning.

Because these models are trained on vast amounts of data, they “have an idea of these similar concepts.” This solves the synonym problem that plagues BM25.

20. Embeddings Visualized

(Timestamp: 12:50)

A 2D visualization demonstrates how embeddings group related concepts in latent space. The word “Doctor” and “Physician” would be located very close to each other mathematically.

This spatial proximity allows for Semantic Search: finding documents that mean the same thing as the query, even if they don’t share a single word.

21. Semantic search is widely used

(Timestamp: 13:15)

The speaker validates the importance of semantic search by showing a tweet from Google’s SearchLiaison regarding BERT, and a screenshot of Hugging Face’s model repository.

This confirms that semantic search is the industry standard for modern information retrieval, having been deployed at massive scale by tech giants to improve result relevance.

22. Which language model?

(Timestamp: 13:30)

A scatter plot compares various models based on Inference Speed (X-axis) and NDCG@10 (Y-axis, a measure of retrieval quality).

Shah places BM25 on the right (fast but lower accuracy) to orient the audience. He points out that there is a massive variety of models with different trade-offs between compute cost and retrieval quality.

23. Static Embeddings

(Timestamp: 14:43)

The speaker introduces Static Embeddings (like Word2Vec or GloVe) which are located on the far right of the previous scatter plot—extremely fast, even on CPUs.

These models assign a fixed vector to every word. While efficient, they lack context. The word “bank” has the same vector whether referring to a river bank or a financial bank, which limits their accuracy.

24. Why Context Matters

(Timestamp: 15:16)

A cartoon illustrates the difference between Static Embeddings and Transformers. The Transformer can distinguish between “Model” in a data science context versus “Model” in a fashion context.

This contextual awareness is why modern Transformer-based embeddings (like BERT) generally outperform static embeddings and BM25 in complex retrieval tasks, despite being slower.

25. Many more models!

(Timestamp: 15:55)

Returning to the scatter plot, a red arrow points toward the top-left quadrant—models that are slower but achieve higher accuracy.

The speaker notes that the field is constantly evolving, with “newer generations of models” pushing the boundary of what is possible in terms of retrieval quality.

26. MTEB/RTEB

(Timestamp: 16:35)

To help developers choose, Shah introduces the MTEB (Massive Text Embedding Benchmark) and RTEB (Retrieval Text Embedding Benchmark). These are leaderboards hosted on Hugging Face.

He highlights a key distinction: MTEB uses public datasets, while RTEB uses private, held-out datasets. This is crucial for avoiding “data contamination,” where models perform well simply because they were trained on the test data.

27. Selecting an embedding model

(Timestamp: 16:48)

The speaker switches to a live browser view (captured in the slide) of the leaderboard. He discusses the bubble chart visualization where size often correlates with parameter count.

He points out an interesting trend: “You’ll see that there’s a bunch of models here that are all the same size… but the performance differs.” This indicates improvements in training strategies and architecture rather than just throwing more compute at the problem.

28. Selecting an embedding model (Other Considerations)

(Timestamp: 19:07)

Beyond the leaderboard score, Shah lists practical selection criteria: Model Size (can it fit in memory?), Architecture (CPU vs GPU), Embedding Dimension (storage costs), and Training Data (multilingual support).

He advises checking if a model is open source and quantizable, as this can significantly reduce latency without a major hit to accuracy.

29. Matryoshka Embedding Models

(Timestamp: 20:53)

A specific innovation is highlighted: Matryoshka Embeddings. These models allow developers to truncate vectors (e.g., from 768 dimensions down to 64) while retaining most of the performance.

This is a “neat kind of innovation” for optimizing storage and search speed. OpenAI’s newer models also support this feature, offering flexibility between cost and accuracy.

30. Sentence Transformer

(Timestamp: 21:42)

The Sentence Transformer architecture is described as the dominant approach for RAG. Unlike standard BERT which works on tokens, these are fine-tuned to understand full sentences and paragraphs.

This architecture uses Siamese networks to ensure that semantically similar sentences are close in vector space, making them ideal for the “chunk-level” retrieval required in RAG.

31. Cross Encoder / Reranker

(Timestamp: 22:16)

The concept of a Cross Encoder (or Reranker) is introduced. Unlike the bi-encoder (retriever) which processes query and document separately, the cross-encoder processes them together.

This allows for a much deeper calculation of relevance. It is typically used as a second stage: retrieve 50 documents quickly with vectors, then use the slow but accurate Cross Encoder to rank the top 5.

32. Cross Encoder / Reranker (Duplicate)

(Timestamp: 22:16)

(This slide reinforces the previous diagram, emphasizing the “crossing” of the query and document in the model architecture.)

33. Cross Encoder / Reranker (Accuracy Boost)

(Timestamp: 23:07)

A bar chart quantifies the value of reranking. It shows a significant boost in NDCG (accuracy) when a reranker is added to the pipeline.

The speaker notes that while you get a “bump” in quality, it “doesn’t come for free.” The trade-off is increased latency, as the cross-encoder is computationally expensive.

34. Cross Encoder / Reranker (Execution Flow)

(Timestamp: 23:15)

The execution flow diagram highlights the reranker’s position in the pipeline. It sits between the Vector Store retrieval and the LLM generation.

This visual reinforces the latency implication: the user has to wait for both the initial search and the reranking pass before the LLM even starts generating an answer.

35. Hands On: Retriever & Reranker

(Timestamp: 23:30)

A screenshot of a Google Colab notebook is shown, demonstrating a practical implementation of the Retrieve and Re-rank strategy using the SentenceTransformer and CrossEncoder libraries.

This provides a concrete resource for the audience to test the accuracy vs. speed trade-offs themselves on simple datasets like Wikipedia.

36. Instruction Following Reranker

(Timestamp: 23:48)

Shah mentions a specific advancement: Instruction Following Rerankers (developed by his company, Contextual). These allow developers to pass a prompt to the reranker, such as “Prioritize safety notices.”

This adds a “knob” for developers to tune retrieval based on business logic without retraining the model.

37. Combine Multiple Retrievers

(Timestamp: 24:19)

The presentation suggests that you don’t have to pick just one method. You can combine BM25, various embedding models (E5, BGE), and rerankers.

While combining them (Ensemble Retrieval) often yields better recall, Shah warns that “you got to engineer this.” Managing multiple indexes and fusion logic increases operational complexity and compute costs.

38. Cascading Rerankers in Kaggle

(Timestamp: 24:56)

A complex diagram from a Kaggle competition winner illustrates a Cascade Strategy. The solution used three different rerankers, filtering from 64 documents down to 8, and then to 5.

This shows the extreme end of retrieval engineering, where multiple models are chained to squeeze out every percentage point of accuracy.

39. Best practices

(Timestamp: 25:16)

Shah distills the complexity into a recommended Best Practice: 1. Hybrid Search: Combine Semantic Search (Vectors) and Lexical Search (BM25). 2. Reciprocal Rank Fusion: Merge the results. 3. Reranker: Pass the top results through a cross-encoder.

This setup provides a “pretty good standard performance out of the box” and should be the default baseline before trying exotic methods.

40. Families of Embedding Models

(Timestamp: 25:42)

A taxonomy slide categorizes the models discussed: Static (Fastest/Low Accuracy), Bi-Encoders (Fast/Good Accuracy), and Cross-Encoders (Slow/Best Accuracy).

This summary helps the audience mentally organize the tools available in their toolbox.

41. Lots of New Models

(Timestamp: 25:50)

Logos for IBM Granite, Google EmbeddingGemma, and others appear. The speaker notes that while new models from major players appear weekly, the improvements are often “incremental.”

He advises against “ripping up” a working system just to switch to a model that is 1% better on a leaderboard.

42. Other retrieval methods

(Timestamp: 26:18)

Alternative methods are briefly listed: SPLADE (Sparse retrieval), ColBERT (Late interaction), and GraphRAG.

Shah acknowledges these exist and may fit specific niches, but warns against chasing the “flavor of the week” before establishing a solid baseline with hybrid search.

43. Operational Concerns

(Timestamp: 27:30)

The talk shifts to operations. Libraries like FAISS are mentioned for efficient vector similarity search.

A key point is that for many use cases, you can simply store embeddings in memory. You don’t always need a complex vector database if your dataset fits in RAM.

44. Vector Database Options

(Timestamp: 27:55)

A diagram categorizes storage into Hot (In-Memory), Warm (SSD/Disk), and Cold tiers.

Shah notes there are “tons of vector database options” (Snowflake, Pinecone, etc.). The choice should be governed by latency requirements. If you need sub-millisecond retrieval, you need in-memory storage.

45. Operational Concerns (Datastore Size)

(Timestamp: 28:40)

A graph shows that as Datastore Size increases (X-axis), retrieval performance naturally degrades (Y-axis).

To combat this, the speaker strongly recommends using Metadata Filtering. “If you’re not using something like metadata… it’s going to be very tough.” Narrowing the search scope is essential for scaling to millions of documents.

46. Search Strategy Comparison

(Timestamp: 29:22)

The presentation pivots to the “exciting part”: Agentic RAG. A visual compares “Traditional RAG” (a linear path) with “Agentic RAG” (a winding, exploratory path).

This represents the shift from a “one-shot” retrieval attempt to an iterative system that can explore, backtrack, and reason.

47. Tools use / Reasoning

(Timestamp: 29:40)

Reasoning models (like o1 or DeepSeek R1) enable LLMs to use tools effectively. A code snippet shows an agent loop: query -> generate -> “Did it answer the question?”

If the answer is no, the model can “rewrite the query… try to find that missing information, feed that back into the loop.” This self-correction is the core of Agentic RAG.

48. Agentic RAG (Workflow)

(Timestamp: 30:32)

A flowchart details the Agentic RAG lifecycle. The model thinks through steps: “Oh, this is the query I need to make… based on those results… maybe we should do it a different way.”

This workflow allows the system to synthesize answers from multiple sources or clarify ambiguous queries automatically.

49. Tools use / Reasoning (Detailed Example)

(Timestamp: 30:35)

A specific example of a complex query is shown. The agent breaks the problem down, calls tools, and iterates.

This demonstrates that the “Thinking” time is where the value is generated, allowing for a depth of research that a single retrieval pass cannot match.

50. Open Deep Research

(Timestamp: 31:02)

Shah references “Open Deep Research” by LangChain, an open-source framework where sub-agents go out, perform research, and report back.

This is a specific category of Agentic RAG focused on generating comprehensive reports rather than quick answers.

51. DeepResearch Bench

(Timestamp: 31:30)

A leaderboard for DeepResearch Bench is shown, testing models on “100 PhD level research tasks.”

The speaker warns that this approach “can get very expensive.” Solving a single complex query might cost significant money due to the number of tokens and iterative steps required.

52. Westlaw AI Deep Research

(Timestamp: 31:55)

A real-world application is highlighted: Westlaw AI. In the legal field, thoroughness is worth the latency and cost.

This proves that Agentic RAG isn’t just a toy; it is being commercialized in high-value verticals where accuracy is paramount.

53. Agentic RAG (Self-RAG)

(Timestamp: 32:11)

The concept of Self-RAG is introduced, emphasizing the “Reflection” step. The model critiques its own retrieved documents and generation quality.

Shah notes that this isn’t brand new, but has become practical due to better reasoning models.

54. Agentic RAG (LangChain Reddit)

(Timestamp: 34:04)

A Reddit post is shown where a developer discusses building a self-reflection RAG system. This highlights the community’s active experimentation with these loops.

55. Agentic RAG (Efficiency Concerns)

(Timestamp: 34:15)

The discussion turns to the “Rub”: Inefficiency. Agentic loops can be slow and wasteful, re-retrieving data unnecessarily.

This sets up the trade-off conversation again: Is the extra time and compute worth the accuracy gain?

56. Research: BRIGHT

(Timestamp: 32:11)

Note: The speaker introduces the BRIGHT benchmark around 32:11, slightly out of slide order in the transcript flow, but connects it here.

BRIGHT is a benchmark specifically designed for Retrieval Reasoning. Unlike standard benchmarks that test keyword matching, BRIGHT tests questions that require thinking, logic, and multi-step deduction to find the correct document.

57. BRIGHT #1: DIVER

(Timestamp: 32:48)

The top-performing system on BRIGHT is DIVER. The diagram shows it uses the exact components discussed earlier: Chunking, Retrieving, and Reranking, but wrapped in an iterative loop.

Shah points out, “It probably doesn’t look that crazy to you if you’re used to RAG.” The innovation is in the process, not necessarily a magical new model architecture.

58. BRIGHT #1: DIVER (LLM Instructions)

(Timestamp: 33:31)

The specific prompts used in DIVER are shown. The system asks the LLM: “Given a query… what do you think would be possibly helpful to do?”

This Query Expansion allows the system to generate new search terms that the user didn’t think of, bridging the semantic gap through reasoning.

59. Agentic RAG on WixQA

(Timestamp: 34:36)

Shah shares his own experiment results on the WixQA dataset (technical support). * One Shot RAG: 5 seconds latency, 76% Factuality. * Agentic RAG: Slower latency, 93% Factuality.

This massive jump in accuracy (0.76 to 0.93) is the key takeaway. “That has a ton of implications.” It suggests that the limitation of RAG often isn’t the data, but the lack of reasoning applied to the retrieval process.

60. Rethink your Assumptions

(Timestamp: 37:10)

This is the climax of the technical argument. A graph from the BRIGHT paper shows that BM25 (lexical search) combined with an Agentic loop (GPT-4) outperforms advanced embedding models (Qwen).

“This is crazy,” Shah exclaims. Because the LLM can rewrite queries into many variations, it mitigates BM25’s weakness (synonyms). This implies you might not need complex vector databases if you have a smart agent.

61. Agentic RAG with BM25

(Timestamp: 38:20)

Shah validates the paper’s finding with his own internal data (Financial 10Ks). Agentic RAG with BM25 performed nearly as well as Agentic RAG with Embeddings.

He suggests a radical possibility: “I could throw all that away [vector DBs]… just stick this in a text-only database and use BM25.”

62. Agentic RAG for Code Search

(Timestamp: 39:46)

He connects this finding to Claude Code, which uses a lexical approach (like grep) rather than vectors for code search.

Since code doesn’t have the same semantic ambiguity as natural language, and agents can iterate rapidly, lexical search is proving to be superior for coding assistants.

63. Combine Retrieval Approaches

(Timestamp: 40:15)

A DoorDash case study illustrates a two-tier guardrail system. They use simple text similarity first (fast/cheap). If that fails or is uncertain, they kick it to an LLM (slow/expensive).

This “Tiered” approach optimizes the trade-off between cost and accuracy in production.

64. Hands on: Agentic RAG (Smolagents)

(Timestamp: 41:07)

The speaker points to Smolagents, a Hugging Face library, as a way to get hands-on with these concepts. A Colab notebook is provided for the audience to build their own agentic retrieval loops.

65. Solutions for a RAG Solution

(Timestamp: 41:18)

Shah updates the “Problem Complexity” framework from the beginning of the talk with specific recommendations: * Low Latency (<5s): Use BM25 or Static Embeddings. * High Cost of Mistake: Add a Reranker. * Complex Multi-hop: Use Agentic RAG.

66. Retriever Checklist

(Timestamp: 41:52)

A final checklist summarizes the retrieval hierarchy: 1. Keyword/BM25 (The baseline). 2. Semantic Search (The standard). 3. Agentic/Reasoning (The problem solver).

This provides the audience with a mental menu to choose from based on their specific constraints.

67. RAG as a system (Retrieval with Instruction Following Reranker)

(Timestamp: 42:00)

The system diagram is shown one last time, updated to include the Instruction Following Reranker in the retrieval box, solidifying the modern RAG architecture.

68. RAG - Generation

(Timestamp: 42:10)

Note: The speaker concludes the talk at 42:10, stating “I’m going to end it here.” Slides 68-70 regarding the Generation stage were included in the deck but skipped in the video recording due to time constraints.

This slide would have covered the final stage of RAG: generating the answer. The focus here is typically on reducing hallucinations and ensuring the tone matches the user’s needs.

69. RAG - Generation (Model Selection)

(Timestamp: 42:10)

Skipped in video. This slide illustrates the choice of LLM for generation (e.g., GPT-4 vs Llama 3 vs Claude). The choice depends on the “Cost/Latency budget” and specific domain requirements.

70. Chunking approaches

(Timestamp: 42:10)

Skipped in video. This slide compares Original Chunking (cutting text at fixed intervals) with Contextual Chunking (adding a summary prefix to every chunk). Contextual chunking significantly improves retrieval because every chunk carries the context of the parent document.

71. Title Slide (Duplicate)

(Timestamp: 42:10)

The presentation concludes with the title slide. Rajiv Shah thanks the audience, encouraging them to think about trade-offs rather than just chasing the latest models. “Hopefully I’ve given you a sense of thinking about these trade-offs… thank you all.”

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

A. MNIST (Very visual dataset)

from keras.datasets import mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 28 * 28)
X_test = X_test.reshape(10000, 28 * 28)

print(list(X_train[7, :]))

plt.imshow(X_train[7, :].reshape(28, 28), cmap = 'binary', vmin = 0, vmax = 255)
plt.xticks([])
plt.yticks([])
plt.savefig('sample_image.png')

X_train = pd.DataFrame(X_train)  # Assuming X_mnist is the MNIST feature data
y_train = pd.Series(y_train)   
X_test = pd.DataFrame(X_test)  # Assuming X_mnist is the MNIST feature data
y_test = pd.Series(y_test)  

B. Madelon (Very high-dimensional dataset that you control)

If you run the following cells, Madelon will be the dataset you use. If you want to use MNIST, you should skip the following cells.

Madelon is a favorite of mine because you know which features are carrying the signal and which ones are noise. In this case, the first 5 features will be informative. I often modify Madelon to include other types of noisy features, interactions, correlations, and then use this dataset to test various machine learning techniques. Since I know what the true signal is, this is very effective at helping me guage the effectiveness of these methods.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=10000,
                           n_features=40,
                           n_informative=5,
                           n_classes=3,
                           n_redundant = 3,
                           random_state=0,
                           flip_y =0.05,
                           class_sep = 0.5,
                           n_clusters_per_class=3,
                           shuffle=False)

X_df = pd.DataFrame(X, columns=[f'{i}' for i in range(X.shape[1])])
y_df = pd.Series(y, name='target')

X_train, X_test, y_train, y_test = train_test_split(X_df, y_df, test_size=0.2, random_state=42)

X_train.columns = X_train.columns.astype(int)
X_test.columns = X_test.columns.astype(int)

X_train

1. F-statistic

f_classif relies on the Analysis of Variance (ANOVA) F-statistic to evaluate the relationship between each feature and the target variable. It tests whether the mean values of the target variable differ significantly across the groups defined by each feature. The higher the F-value, the more likely it is that the feature discriminates between different classes. Assumes a linear relationship between the features and the target, and that the target is categorical.

See: https://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection

from sklearn.feature_selection import f_classif

f = f_classif(X_train, y_train)[0]
f

array([2.40810399e+02, 6.64390517e+01, 5.42587843e+01, 1.71099993e-01,
       4.27226542e+01, 9.64735668e+01, 4.72136845e+01, 5.71846457e+01,
       1.27834979e+00, 1.42650284e+00, 3.40020422e-01, 4.08232233e-01,
       1.30120819e-01, 3.36734714e+00, 4.55665866e-01, 3.62300510e-01,
       1.57485437e-01, 6.93572687e-02, 1.64816305e+00, 2.91782944e+00,
       1.24026934e+00, 9.32533895e-01, 7.07099908e-01, 1.87544216e+00,
       1.10130690e+00, 3.54044700e-01, 1.15417945e+00, 2.59156089e-01,
       7.45820681e-01, 7.75403854e-01, 1.35835715e-01, 3.34985292e+00,
       8.36576456e-02, 5.15026453e-02, 4.33788709e-01, 3.12140721e-01,
       3.55118575e+00, 7.37076241e+00, 1.17274619e+00, 4.36532461e+00])

2. Mutual information

mutual_info_classif uses the concept of mutual information, which measures the dependency between each feature and the target variable. Mutual information quantifies the amount of information gained about the target by knowing the value of the feature. It captures both linear and non-linear dependencies.

See: https://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection

from sklearn.feature_selection import mutual_info_classif

mi = mutual_info_classif(X_train, y_train)
mi

array([0.05548077, 0.01340894, 0.04092611, 0.00099355, 0.00516085,
       0.06952759, 0.02752171, 0.04043945, 0.00083431, 0.        ,
       0.        , 0.        , 0.        , 0.01146672, 0.        ,
       0.00298292, 0.        , 0.        , 0.0068234 , 0.00961735,
       0.00935105, 0.00586449, 0.00561433, 0.        , 0.        ,
       0.        , 0.        , 0.        , 0.00876386, 0.00049355,
       0.        , 0.00478042, 0.00487523, 0.00268551, 0.00118896,
       0.        , 0.00583264, 0.        , 0.        , 0.        ])

3. Logistic regression

Logistic regression is a linear model for classification rather than regression. It is used to estimate the probability that an instance belongs to a particular class. The coefficients of the model can be used to determine feature importance.

from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression().fit(X_train, y_train)
logreg.coef_

array([[-0.08669758,  0.06064912, -0.04063592,  0.00704782,  0.05079063,
        -0.02366404, -0.03246196, -0.07613473, -0.02216845,  0.00405511,
         0.00890925, -0.01289459, -0.00446435,  0.00330386,  0.01287983,
        -0.00599418,  0.00494212, -0.00385749,  0.03721175, -0.03849129,
        -0.0032749 , -0.01534965, -0.00908255,  0.02016669, -0.00175419,
         0.00918138,  0.01908963,  0.01357562, -0.01804835,  0.00266229,
         0.00180036, -0.00624841, -0.00351875,  0.00131487, -0.01573702,
        -0.00485053, -0.03744854,  0.05047984,  0.0174477 , -0.00658735],
       [ 0.22615114, -0.09497312, -0.04870109,  0.08462927, -0.00434942,
         0.0845374 ,  0.05186658,  0.05312074,  0.01121456,  0.0271935 ,
        -0.00027397,  0.01737956,  0.0080553 , -0.04770429, -0.00379082,
        -0.00963934,  0.00274429,  0.00688572, -0.03159925,  0.02737958,
        -0.0220797 , -0.00789503, -0.01134082, -0.02724488, -0.02432216,
         0.00543336, -0.02498671, -0.00919296,  0.01043275,  0.01323123,
        -0.00059363, -0.02632891, -0.00159478, -0.00753653,  0.01697607,
         0.01247471,  0.04553467, -0.05091659, -0.02028138,  0.03615948],
       [-0.13945356,  0.03432401,  0.089337  , -0.09167709, -0.04644121,
        -0.06087336, -0.01940462,  0.02301399,  0.0109539 , -0.03124862,
        -0.00863527, -0.00448497, -0.00359095,  0.04440043, -0.00908901,
         0.01563352, -0.00768641, -0.00302823, -0.0056125 ,  0.01111171,
         0.02535461,  0.02324468,  0.02042338,  0.0070782 ,  0.02607635,
        -0.01461474,  0.00589708, -0.00438266,  0.0076156 , -0.01589352,
        -0.00120673,  0.03257732,  0.00511353,  0.00622166, -0.00123904,
        -0.00762418, -0.00808614,  0.00043674,  0.00283369, -0.02957212]])

3.5 Feature Selection with L1 (Lasso) Regularization

Lasso is a great feature selection technique. It’s fast, easy to use, and works well with high-dimensional data. I have often used it when very wide data, greater than 100 features (or even >10k features) to help parse down the number of features. It uses L1 regularization to penalize the absolute size of the coefficients. This leads to sparse solutions, where many of the coefficients are zero. The features with non-zero coefficients are selected. Lasso can be used for feature selection by setting the regularization parameter to a value that results in a sparse solution. The regularization parameter can be tuned using cross-validation.

Try modifying the regularization parameter to see how it affects the number of features selected.

from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

# Step 4: Standardize the features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Step 5: Apply Logistic Regression with L1 regularization for feature selection
logregL1 = LogisticRegression(penalty='l1', solver='saga', multi_class='multinomial', C=0.01)  # C is inverse of regularization strength
logregL1.fit(X_train_scaled, y_train)

# Step 6: Get the selected features using the original DataFrame 'X'
selected_features = X_train.columns[(logregL1.coef_ != 0).any(axis=0)]
print("Selected features: ", selected_features)

# Optional: Check the coefficients
#print("Logistic Regression coefficients: ", logreg.coef_)

Selected features:  Index([0, 1, 2, 4, 5, 13, 19, 36, 37], dtype='int64')

4. LightGBM

LightGBM is a gradient boosting framework that uses tree-based learning algorithms. It is designed for efficiency and can handle large datasets. It can be used to determine feature importance.

from lightgbm import LGBMClassifier

lgbm = LGBMClassifier(
    objective = 'multiclass',
    metric = 'multi_logloss',
    importance_type = 'gain'
).fit(X_train, y_train)

lgbm.feature_importances_

[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.001096 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 10200
[LightGBM] [Info] Number of data points in the train set: 8000, number of used features: 40
[LightGBM] [Info] Start training from score -1.108284
[LightGBM] [Info] Start training from score -1.094371
[LightGBM] [Info] Start training from score -1.093252

array([7030.24782425, 4036.90086633, 7197.39050466, 5339.74033117,
       2017.793881  , 8113.67321557, 3905.26838762, 4383.72206521,
        550.03625131,  531.02187729,  472.12624365,  678.28547454,
        526.57803982,  586.75292325,  552.92263156,  433.08122051,
        552.18078488,  534.15573859,  566.58704376,  630.01932001,
        635.43262064,  636.71719581,  560.95981157,  586.52648336,
        553.7755444 ,  563.13766581,  547.99060541,  523.01072556,
        676.76891661,  616.94216621,  634.27822083,  489.91742009,
        680.71264285,  620.95509708,  618.59545827,  418.22946733,
        568.21738124,  592.29172051,  553.43465978,  655.03435677])

5. Boruta

Boruta is an all-relevant feature selection method. It is an extension of the Random Forest algorithm. It selects all features that are relevant to the target variable, rather than just the most important features.

### long training time > 1 hour
from boruta import BorutaPy
from sklearn.ensemble import RandomForestClassifier

#boruta = BorutaPy(
#    estimator = RandomForestClassifier(max_depth = 5), 
#    n_estimators = 'auto', 
#    max_iter = 100
#).fit(X_train, y_train)

6. MRMR

MRMR (Minimum Redundancy Maximum Relevance) is a feature selection method that selects features based on their relevance to the target variable and their redundancy with other features. It aims to select features that are highly correlated with the target variable but uncorrelated with each other.

There are several implementations of MRMR available in Python: https://github.com/smazzanti/mrmr https://koaning.github.io/scikit-lego/api/feature-selection/ https://github.com/AutoViML/featurewiz?tab=readme-ov-file

import pandas as pd
from mrmr import mrmr_classif

#mrmr = mrmr_classif(pd.DataFrame(X_train), pd.Series(y_train), K = 784)
mrmr = mrmr_classif(pd.DataFrame(X_train), pd.Series(y_train), K = X_train.shape[1])

100%|██████████| 40/40 [00:03<00:00, 11.45it/s]

Store results

import numpy as np

ranking = pd.DataFrame(index = range(X_train.shape[1]))

ranking['f'] = pd.Series(f, index = ranking.index).fillna(0).rank(ascending = False)
ranking['mi'] = pd.Series(mi, index = ranking.index).fillna(0).rank(ascending = False)
ranking['logreg'] = pd.Series(np.abs(logreg.coef_).mean(axis = 0), index = ranking.index).rank(ascending = False)
ranking['lasso']= pd.Series(np.abs(logregL1.coef_).mean(axis = 0), index = ranking.index).rank(ascending = False)
ranking['lightgbm'] = pd.Series(lgbm.feature_importances_, index = ranking.index).rank(ascending = False)
#ranking['boruta'] = boruta.support_* 1 + boruta.support_weak_ * 2 + (1 - boruta.support_ - boruta.support_weak_) * X_train.shape[1]
ranking['mrmr'] = pd.Series(
    list(range(1, len(mrmr) + 1)) + [len(mrmr) + 1] * (X_train.shape[1] - len(mrmr)),
    index = mrmr + list(set(ranking.index) - set(mrmr))
).sort_index()
ranking['lasso']= pd.Series(np.abs(logregL1.coef_).mean(axis = 0), index = ranking.index).rank(ascending = False)


ranking = ranking.replace(to_replace = ranking.max(), value = X_train.shape[1])
#ranking.to_csv('ranking.csv', index = False)

Recursive Feature Elimination

One of the best methods for feature selection consistently is feature importance with LightGBM. We can refine and improve this in several ways: Recursive Feature Elimination uses the same feature importance method, but then iteratively removes the least important features. This iterative process requires training a model several times, but can provide an improvement in feature selection. This method is a version of Recursive Feature Elimination that is widely accepted as a best practice for feature selection.

from sklearn.feature_selection import RFE
from xgboost import XGBClassifier
from sklearn.svm import SVR
model = XGBClassifier(random_state=42)
#model = SVR(kernel="linear")  #took 3 minutes, ok results but not as good as XGB on Madelon
rfe = RFE(model, n_features_to_select=7, step=1)
#rfe = RFE(model, n_features_to_select=50, step=200,verbose=2) #for MNIST
rfe.fit(X_train, y_train)
rfe.support_

array([ True,  True,  True,  True, False,  True,  True,  True, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False])

# Train an XGBoost model with the selected features from RFE
model_selected = XGBClassifier(random_state=42)
X_selected = X_train.loc[:, rfe.support_]
model_selected.fit(X_selected, y_train)

# Make predictions on the test set with both models
y_pred_selected = model_selected.predict(X_test.loc[:, rfe.support_])
accuracy_selected = accuracy_score(y_test, y_pred_selected)
print(f"Accuracy with selected features: {accuracy_selected:.4f}")

Accuracy with selected features: 0.7135

Compare with perfect on Madelon

perfect = [ True,  True,  True,  True, True,  True,  True,  True, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False]

# Train an XGBoost model with the selected features from RFE
model_selected = XGBClassifier(random_state=42)
X_selected = X_train.loc[:, perfect]
model_selected.fit(X_selected, y_train)

# Make predictions on the test set with both models
y_pred_selected = model_selected.predict(X_test.loc[:, perfect])
accuracy_selected = accuracy_score(y_test, y_pred_selected)
print(f"Accuracy with selected features: {accuracy_selected:.4f}")

Accuracy with selected features: 0.7140

Feature Elimination with FIRE

At DataRobot, we had a mighty AutoML engine that showed you how feature importance aggregated across different models (this is feature importance from four diverse models).

You can use this variance as part of feature selection. It takes a lot more compute, but in our experiments, can perform even better feature selection. Read more about feature importance rank ensembling (FIRE) here - https://docs.datarobot.com/en/docs/api/accelerators/adv-approaches/fire.html and a code snippet is here - https://github.com/datarobot-community/examples-for-data-scientists/blob/master/Feature%20Lists%20Manipulation/Python/Advanced%20Feature%20Selection.ipynb

FeatureViz

Featureviz looks like a cool feature selection package, but I wasn’t able to get it to work. It’s worth checking out. add links

1. Title Slide

(Timestamp: 00:00:00)

The presentation begins by introducing the core topic: Interpretable Models and the use of rules in machine learning. Rajiv Shah sets the stage by contrasting this talk with previous discussions on explainability (using tools to explain complex models). Instead, this session focuses on choosing models that are inherently easy to understand.

Shah expresses his interest in how machine learning helps us understand the world. He notes that while tools like SHAP or LIME help unpack complex models, there is immense value in approaching the problem differently: by selecting model architectures that are transparent by design.

The speaker invites the audience to view this not just as a technical lecture but as a discussion on the trade-offs between model complexity and interpretability, setting a collaborative tone for the presentation.

2. Table of Contents

(Timestamp: 00:02:30)

This slide outlines the roadmap for the presentation. Shah explains that he will begin with the “Big Picture” concepts—specifically the “Why?” and the “Baseline”—before diving into four specific technical approaches to rule-based modeling.

The four specific methods to be covered are Rulefit, GA2M (Generalized Additive Models with interactions), Rule Lists, and Scorecards. This structure moves from theoretical justification to practical application, comparing different algorithms that prioritize transparency.

Shah also mentions that a GitHub repository is available with code examples for everything shown, allowing the audience to reproduce the results for the tabular datasets discussed.

3. Section 1: Why?

(Timestamp: 00:03:09)

This section header introduces the fundamental question: Why do we want rules? The speaker moves past the obvious statement that “AI is important” to investigate the influences that drive data scientists toward complex, opaque models.

Shah prepares to discuss the cultural and competitive pressures in data science that prioritize raw accuracy over usability. This section serves as a critique of the “accuracy at all costs” mindset often found in the industry.

4. Mark Cuban Quote

(Timestamp: 00:03:17)

The slide features a quote from Mark Cuban: “Artificial Intelligence, deep learning, machine learning — whatever you’re doing if you don’t understand it — learn it. Because otherwise you’re going to be a dinosaur within 3 years.”

Shah briefly references this as the “obligatory” acknowledgment of AI’s massive importance in the current landscape. It reinforces that while the field is moving fast, the understanding of these systems is paramount, which ties into the presentation’s focus on interpretability.

5. Influences: Kaggle & Academia

(Timestamp: 00:03:40)

Shah identifies Kaggle competitions and academic research as two primary influences on data scientists. He notes that these platforms heavily incentivize accuracy above all else. For example, in the Zillow Prize, the difference between the top scores is minuscule, yet teams fight for that fraction of a percentage.

He argues that this environment trains data scientists to focus solely on improving metrics (like RMSE or AUC), often ignoring other critical trade-offs like model complexity, deployment difficulty, or explainability.

As he states, “One of the byproducts of Kaggle is a very heavy focus on making sure you improve your models around accuracy… and that’s how you can get a conference paper.” This sets up the problem of complexity creep.

6. The Netflix Prize Winners

(Timestamp: 00:05:39)

This slide shows the winners of the famous Netflix Prize, a competition held about 15 years ago where a team won $1 million for improving Netflix’s recommendation algorithm by 10%.

Shah uses this story to illustrate the peak of the “accuracy” mindset. The competition drew massive interest and drove innovation, but it also encouraged teams to prioritize the leaderboard score over the practicality of the solution.

7. Netflix Prize Progress Graph

(Timestamp: 00:06:14)

The graph displays the progress of teams over time during the Netflix competition. Shah points out that after an initial period of rapid improvement using standard algorithms, progress plateaued.

To break through these plateaus, teams began using Ensembling—combining multiple models together. The winning solution was an ensemble of 107 different models. Shah emphasizes that while this strategy is powerful for eking out the last bit of performance, it creates immense complexity.

8. The Engineering Cost of Complexity

(Timestamp: 00:07:39)

This slide reveals the ironic conclusion of the Netflix Prize: the winning model was never implemented. The engineering costs to deploy an ensemble of 107 models were simply too high compared to the marginal gain in accuracy.

Shah uses this as a cautionary tale: “If your focus is on accuracy… it drives you down towards this complexity… but often you end up with these complex models [that] are often very difficult to implement.” This highlights the disconnect between competitive data science and enterprise reality.

9. Understandable White Box Model (CLEAR-2)

(Timestamp: 00:08:04)

Shah transitions to the alternative: Interpretable Models. This slide shows a simple linear model (CLEAR-2) with only two features. This is a classic “White Box” model where the relationship between inputs and outputs is transparent.

The speaker contrasts this with the “Black Box” nature of complex ensembles. He argues that if you cannot understand what is going on inside a model, you cannot effectively debug it, nor can you easily convince stakeholders to trust it.

10. Complex White Box Model (CLEAR-8)

(Timestamp: 00:11:51)

This slide presents a linear model with eight features (CLEAR-8). While technically still a “White Box” model, Shah implies that as feature counts grow, true understandability diminishes.

He touches on this concept later in the “Caveats” section, noting that even linear models can become confusing if there is multicollinearity (features moving in the same direction). Just because we can see the coefficients doesn’t mean the model is intuitively “explainable” to a human if the variables interact in complex, non-obvious ways.

11. Easy to Understand Decision Tree

(Timestamp: 00:19:15)

Here, a simple Decision Tree is presented. Shah connects this to the history of rule-based learning, noting that early research found that keeping decision trees “short and stumpy” made them very easy for humans to explain.

This visual represents the ideal of interpretability: a clear path of logic (e.g., “If X is less than 3, go left”) that leads to a prediction. This is the foundation for the Rulefit method discussed later.

12. Too Much to Comprehend

(Timestamp: 00:07:46)

Contrasting the previous slide, this image shows a chaotic forest of decision trees. This represents modern ensemble methods like Random Forests or Gradient Boosted Machines.

Shah uses this visual to reinforce the point that while ensembles offer “Better Performance,” the sheer number of decision paths makes them “too much to Comprehend.” You lose the ability to trace the “why” behind a specific prediction, turning the system into a Black Box.

13. Pedro Domingos Tweet

(Timestamp: 00:08:22)

Shah acknowledges the counter-argument by showing a tweet from Pedro Domingos, a prominent machine learning researcher, who suggests that demanding explainability limits the potential of AI.

Shah respectfully disagrees with this stance in the context of enterprise data science. He argues that in the real world, “If you don’t understand what’s going on in your model, it’s hard for you to debug it, it’s hard to convince somebody else to adopt your model.” Practicality and trust often outweigh raw theoretical power.

14. Benefits of Interpretable Models

(Timestamp: 00:09:16)

This slide summarizes the key benefits of using interpretable models, referencing the work of Cynthia Rudin. The main advantages are: 1. Debugging: It is easier to spot weird behaviors. 2. Trust: Stakeholders and legal/risk teams are more likely to approve the model. 3. Deployment: These models can often be deployed as simple SQL queries or basic code, avoiding the need for heavy GPU infrastructure.

Shah emphasizes the deployment aspect: “You don’t have to go out and get a GPU… you can actually deploy directly within a database.”

15. Caveats of Interpretable Models

(Timestamp: 00:11:00)

Shah provides a necessary reality check. He clarifies that selecting an interpretable algorithm is only one part of the process. True interpretability depends on the entire data pipeline.

Issues like data labeling, feature engineering, and multicollinearity can render even a simple model confusing. For example, if two correlated features have opposite coefficients in a linear model, it becomes very difficult to explain the logic to a business user, even if the math is simple.

16. Section 2: Baseline

(Timestamp: 00:12:15)

This slide introduces the Baseline section. Shah advocates for always starting a project with a simple baseline model to establish a performance benchmark.

He shares an anecdote about people spending a year on a project only to be nearly matched by a simple model built in two hours. Establishing a baseline helps determine how much effort should be spent chasing incremental accuracy improvements.

17. The Problem: UCI Adult Dataset

(Timestamp: 00:12:54)

Shah introduces the dataset he will use for all examples in the talk: the UCI Adult Dataset (Census Income). The goal is a binary classification problem: predicting whether someone has a high or low income based on demographics.

He chooses this dataset because it represents typical enterprise tabular data: it has 30,000 rows, a mix of numerical and categorical features, and contains collinearity and interaction effects. This makes it a realistic test bed for the models he will demonstrate.

18. Baseline Models

(Timestamp: 00:13:53)

The speaker outlines the three baseline models he built to bracket the performance possibilities: 1. Logistic Regression: The standard statistical approach. 2. AutoML (H2O): A stacked ensemble of many models (Neural Networks, GBMs, etc.) representing the “maximum” possible performance. 3. OneR: A very simple rule-based algorithm.

These baselines provide the context for evaluating the interpretable models later.

19. Baseline Models Plot

(Timestamp: 00:14:12)

This plot visualizes Complexity vs. AUC (Area Under the Curve). * OneR is at the bottom (AUC ~0.60) with very low complexity. * Logistic Regression is in the middle (AUC ~0.91). * Stacked Ensemble is at the top (AUC ~0.93) but with massive complexity.

Shah notes that while the Stacked Ensemble wins on accuracy, the Logistic Regression is surprisingly close, highlighting that simpler models can often be “good enough.”

20. OneR Example

(Timestamp: 00:15:17)

Shah explains the OneR (One Rule) algorithm. This method finds the single feature in the dataset that best predicts the target. In the example shown (Iris dataset), utilizing just “Petal Width” classifies 96% of instances correctly.

He suggests OneR is a great way to detect Target Leakage—if one feature predicts the target perfectly, it might be “cheating.” It also sets the floor for performance; if a complex model can’t beat OneR, something is wrong.

21. Baseline Models Plot (Recap)

(Timestamp: 00:16:36)

Returning to the complexity plot, Shah reiterates the performance gap. The AutoML model sets the “ceiling” at 0.93 AUC.

The goal for the rest of the presentation is to see where the interpretable models (Rulefit, GA2M, etc.) fall on this graph. Can they approach the 0.93 AUC of the ensemble without incurring the massive complexity penalty?

22. Section 3: Rulefit

(Timestamp: 00:18:18)

This slide introduces the first major interpretable technique: Rulefit. Shah mentions familiarity with this from his time at Data Robot and notes that it is a powerful way to combine the benefits of trees and linear models.

23. What is Rulefit?

(Timestamp: 00:18:30)

Rulefit is an algorithm developed by Friedman and Popescu (2008). It works by: 1. Building a random forest of short, “stumpy” decision trees. 2. Extracting each path through the trees as a “Rule.” 3. Using these rules as binary features in a sparse linear model (Lasso).

This approach allows the model to capture interactions (via the trees) while maintaining the interpretability of a linear equation.

24. H2O Rulefit Output

(Timestamp: 00:22:19)

Shah displays the output from the H2O Rulefit implementation. The model generates human-readable rules, such as: “If Education < 12 AND Capital Gain < $7000, THEN Coefficient is negative.”

He notes that while the rules are readable, the raw output can look like “computer-ese.” However, it allows a data scientist to identify specific segments of the population (e.g., low education, low capital gain) that strongly drive the prediction.

25. Overlapping Rules

(Timestamp: 00:24:30)

A key characteristic of Rulefit is that the rules overlap. A single data point might satisfy multiple rules simultaneously.

Shah points out that this adds a layer of complexity to interpretability. To understand a prediction, you have to sum up the coefficients of all the rules that apply to that person. This is different from a decision tree where you fall into exactly one leaf node.

26. H2O Rulefit with Linear Terms

(Timestamp: 00:25:55)

One limitation of pure rules is handling continuous variables (like age or miles driven). Rules have to “bin” these variables (e.g., Age < 30, Age 30-40).

Shah explains that H2O Rulefit solves this by including Linear Terms. The model can use rules for non-linear interactions and standard linear coefficients for continuous trends. This hybrid approach boosts the AUC significantly (up to 0.88 in this example) by capturing linear relationships more naturally.

27. Rulefit Results

(Timestamp: 00:27:04)

This slide plots the performance of Rulefit models with varying numbers of rules. Shah demonstrates that by increasing the number of rules (complexity), the AUC climbs closer to the Stacked Ensemble.

He concludes that Rulefit is a versatile tool. You can tune the “dial” of complexity: fewer rules for more interpretability, or more rules for higher accuracy, often getting very competitive performance.

28. Section 4: GA2M

(Timestamp: 00:31:35)

The presentation moves to the second technique: GA2M (Generalized Additive Models with pairwise interactions). Shah notes that while GAMs have existed for a while, modern implementations like Microsoft’s Explainable Boosting Machines (EBM) have made them much more accessible and powerful.

29. What is GA2M?

(Timestamp: 00:32:02)

GA2M is essentially a linear model where features are binned, and pairwise interactions are automatically detected. Shah highlights InterpretML, an open-source library from Microsoft that implements this via EBMs.

The model structure is additive: . This means the final score is just the sum of individual feature scores and interaction scores, making it very transparent.

30. GA2M Binning

(Timestamp: 00:32:42)

Shah explains how GA2M handles numerical data. Instead of a single slope coefficient (like in logistic regression), the model bins the continuous feature (e.g., dividing “criminal history” into ranges).

Each bin gets its own coefficient. This allows the model to learn non-linear patterns (e.g., risk might go up, then down, then up again as a variable increases) while remaining easy to inspect.

31. Interactions in GA2M

(Timestamp: 00:33:08)

The “2” in GA2M stands for pairwise interactions. Shah emphasizes that this is the model’s superpower. While standard linear models struggle with interactions (e.g., the combined effect of age and education), GA2M has an efficient algorithm to automatically find the most important pairs.

This allows the model to achieve accuracy levels comparable to complex ensembles (AUC 0.93) because it captures the interaction signal that simple linear models miss.

32. GA2M Visualization

(Timestamp: 00:35:14)

Shah showcases the InterpretML dashboard. It provides clear visualizations of how each feature contributes to the prediction.

In the example, we see the coefficients for different marital statuses. This acts like a “lookup table” for risk. Shah argues that this is very “model risk management friendly” because stakeholders can validate every single coefficient and interaction term to ensure they make business sense.

33. Section 5: Rule Lists

(Timestamp: 00:40:28)

The third approach is Rule Lists. Shah introduces this as a method to solve the “overlapping rules” problem found in Rulefit.

34. What are Rule Lists?

(Timestamp: 00:40:48)

Rule Lists are ordered sets of IF-THEN-ELSE statements. Unlike Rulefit, where you sum up multiple rules, here an observation triggers only the first rule it matches.

Shah mentions implementations like CORELS and SBRL (Scalable Bayesian Rule Lists). The goal is to produce a concise list that a human can read from top to bottom to make a decision.

35. SBRL Process

(Timestamp: 00:41:09)

Creating an optimal rule list is computationally expensive because the algorithm must search through many permutations to find the best order.

Shah explains the logic: The algorithm finds a rule that covers a subset of data, removes those instances, and then finds the next rule for the remaining data. This sequential “peeling off” of data creates the IF-ELSE structure.

36. SBRL Output Example

(Timestamp: 00:41:45)

The output of an SBRL model is shown. It reads like a checklist: 1. IF Capital Gain > $7500 -> High Income (99% prob) 2. ELSE IF Education < 4 -> Low Income (90% prob) 3. ELSE…

Shah highlights the simplicity: “You just go down the list until you find the rule… much easier to explain to those marketing people.” The trade-off is a drop in accuracy (AUC 0.86) compared to GA2M or Rulefit.

37. Section 6: Scorecard

(Timestamp: 00:44:52)

The final approach is the Scorecard. Shah introduces this as perhaps the simplest and most widely recognized format for decision-making in industries like credit and criminal justice.

38. What are Scorecards?

(Timestamp: 00:45:04)

Scorecards are simple additive models where features are assigned integer “points.” To get a prediction, you simply add up the points.

Shah mentions tools like Optbinning and SLIM (Sparse Linear Integer Models). This format is beloved in operations because it can be printed on a physical card or implemented in a basic spreadsheet.

39. Scorecard Example

(Timestamp: 00:46:08)

This slide shows a scorecard built for the Adult dataset. * Capital Gain > 7000? +29 points. * Age < 25? -5 points.

Shah expresses a personal preference for this over raw coefficients: “I actually like this better… I think it’s a little easier to understand which features are most important.” The integer points make the “weight” of each factor immediately obvious to a layperson.

40. Summary

(Timestamp: 00:51:11)

Shah begins to wrap up the presentation, preparing to consolidate the four methods (Rulefit, GA2M, Rule Lists, Scorecards) into a final comparison.

41. Complexity vs AUC Summary Plot

(Timestamp: 00:51:13)

This is the definitive comparison graph of the talk. It places all discussed models on the Complexity vs. AUC plane. * GA2M (EBM) and Rulefit sit high up, offering near-SOTA accuracy with moderate interpretability. * Scorecards and Rule Lists sit lower on accuracy but offer maximum simplicity.

Shah summarizes the trade-off: “The Rule Lists and Scorecard… you lose a little bit [of accuracy]… but we talked about the trade-offs of being able to easily understand.”

42. Take Away

(Timestamp: 00:52:08)

The final message is a call to action: Try these approaches.

Shah encourages data scientists to add these tools to their toolkit. He asks them to consider the specific needs of their problem: Is it about transparency in calculation (Scorecard)? Or understanding factors (GA2M)? Often, a simple model that gets deployed is far better than a complex model that gets stuck in review.

43. Conclusion

(Timestamp: 00:52:52)

The presentation concludes with Rajiv Shah’s contact information. He mentions an upcoming blog post that will synthesize these topics and invites the audience to reach out with questions or feedback.

He reiterates that these interpretable models are often easier to get “buy-in” for, making them a pragmatic choice for real-world data science success.

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

1. The Spark of the AI Revolution

(Timestamp: 00:00)

The presentation begins with the title slide, “The Spark of the AI Revolution: Transfer Learning,” presented by Rajiv Shah from Snowflake. This talk was originally given at the University of Cincinnati and recorded later to share the insights with a broader audience.

Rajiv sets the stage by explaining that this is not a deep technical dive into code, but rather a descriptive history and analysis of the drivers behind the current AI boom. The goal is to explain how AI learns and how individuals can start to interrogate and understand these technologies in their own lives.

The core premise is that Transfer Learning is the catalyst that shifted AI from academic curiosity to a revolutionary force. The talk aims to bridge the gap for those unfamiliar with the underlying mechanics of how models like ChatGPT came to be.

2. Sparks of AGI: Early Experiments

(Timestamp: 01:00)

This slide illustrates an early experiment conducted by researchers investigating GPT-4. To understand how the model was learning, they gave it a concept and asked it to draw it using code (SVG). The slide displays a progression of abstract animal figures, showing how the model’s ability to represent concepts improved over time during training.

This references the paper “Sparks of Artificial General Intelligence,” which caused significant waves in the tech community. It suggests that these models were beginning to show signs of Artificial General Intelligence (AGI)—reasoning capabilities that extend beyond narrow tasks.

The visual progression from crude shapes to recognizable forms serves as a metaphor for the rapid evolution of these models. It highlights the mystery and potential power hidden within the training process of Large Language Models (LLMs).

3. Extinction Level Threat?

(Timestamp: 01:36)

The presentation addresses the extreme concerns surrounding the rapid scaling of AI technologies. The slide features a dramatic image reminiscent of the Terminator, referencing fears that unchecked AI development could pose an “extinction-level” threat to humanity.

Rajiv notes that as these technologies scale, there is a segment of the research and safety community worried about catastrophic outcomes. This sets up a contrast between the theoretical existential risks and the practical, everyday reality of how AI is currently being used.

This slide acknowledges the “hype and fear” cycle that dominates the media narrative, validating the audience’s anxiety before pivoting to a more grounded explanation of how the technology actually works.

4. The New AI Overlords

(Timestamp: 01:42)

Shifting to a lighter tone, this slide highlights the widespread adoption of AI by the younger generation. It cites a statistic that 89% of students have used ChatGPT for homework, humorously suggesting that children have already “accepted our new AI overlords.”

The slide points out a discrepancy in honesty, noting that while 89% use it, a significant portion (implied by the “11% are lying” joke) might not admit it. This reflects a fundamental shift in education and information retrieval that has already taken place.

This context emphasizes that the AI revolution is not just a future possibility but a current reality affecting how the next generation learns and works. It underscores the urgency of understanding these tools.

5. Fundamental Questions

(Timestamp: 01:53)

This slide poses the central questions that the presentation will answer: “What is AI doing?” and “How should you think about AI?” It serves as an agenda setting for the technical explanation that follows.

Rajiv transitions here from the societal impact of AI to the mechanics of machine learning. He prepares the audience to look “under the hood” to demystify the “magic” of tools like ChatGPT.

The goal is to move the audience from passive consumers of AI hype to critical thinkers who understand the limitations and capabilities of the technology based on how it is built.

6. How We Teach Computers

(Timestamp: 02:02)

The presentation begins its technical explanation with a fundamental question: “How do we teach computers?” The slide uses imagery of blueprints and tools, likening the traditional process of building AI models to craftsmanship.

This introduces the concept of Supervised Learning in a relatable way. Before discussing neural networks, Rajiv grounds the audience in traditional analytics, where humans explicitly guide the machine on what to look for.

The focus here is on the human element in traditional machine learning—the “artisan” who must carefully select inputs to get a desired output.

7. Identifying Features

(Timestamp: 02:11)

Using a real estate example, this slide explains the concept of Features (or variables). To teach a computer to value a house, one must identify specific characteristics like square footage, number of bedrooms, or closet space.

Rajiv explains that we capture these characteristics and organize them into a tabular format. This process is known as Feature Engineering, where the data scientist decides which attributes are relevant for the problem at hand.

This is the bedrock of traditional enterprise AI: converting real-world objects into structured data points that a machine can process mathematically.

8. Historical Data Patterns

(Timestamp: 02:50)

This slide displays a scatter plot correlating “Sales Price” with “Square Feet.” It illustrates how enterprises gather historical data to look for patterns and relationships backwards in time.

Rajiv notes that much of traditional analytics is simply looking at this historical data to understand what happened. However, the power of AI lies in using this data for forward-looking purposes.

The visual clearly shows a trend: as square footage increases, the price generally increases. This linear relationship is what the machine needs to “learn.”

9. Learning the Model

(Timestamp: 03:03)

Here, a line is drawn through the data points on the scatter plot. This line represents the Model. Learning, in this context, is simply the mathematical process of fitting this line to the historical data to minimize error.

Rajiv explains that the model “understands the relationships” defined by the data. Instead of a human manually writing rules, the algorithm finds the best-fit trend based on the input features.

This simplifies the concept of training a model down to its essence: finding a mathematical representation of a trend within a dataset.

10. Making Predictions

(Timestamp: 03:16)

This slide demonstrates the utility of the trained model. When a “New House” comes onto the market, the model uses the learned line to predict its value based on its square footage.

This defines the Inference stage of machine learning. The model is no longer learning; it is applying its “knowledge” (the line) to unseen data to generate a prediction.

It highlights the portability of a model—once trained, it can be used to make rapid assessments of new data points without human intervention.

11. The Domain Limitation

(Timestamp: 03:30)

The presentation introduces a critical limitation of traditional models. The slide shows the model trained on San Francisco data being applied to houses in South Carolina. The result is labeled “Poor Model.”

Rajiv explains that while you can technically take the model with you, it will fail because the underlying relationships between features (size) and targets (price) are different in different domains (geographies).

This illustrates the concept of Domain Shift or lack of generalization. A model is only as good as the data it was trained on, and it assumes the future (or new location) looks exactly like the past.

12. The Thinking Emoji

(Timestamp: 03:43)

This slide reinforces the previous point with a thinking emoji, emphasizing the realization that the existing model is inadequate. The “San Francisco Model” does not fit the “South Carolina Data.”

It serves as a visual pause to let the problem sink in: traditional machine learning is brittle. It requires the data distribution to remain constant.

Rajiv uses this to set up the labor-intensive nature of traditional analytics, where models cannot simply be “transferred” across different contexts.

13. Train New Model

(Timestamp: 03:55)

The solution in the traditional paradigm is presented here: “Train New Model.” To get accurate predictions for South Carolina, one must collect local data and repeat the entire training process from scratch.

This highlights the “Never-Ending Battle” of enterprise analytics. Data scientists are constantly retraining models for every specific region, product line, or use case.

This sets the baseline for why Transfer Learning (introduced later) is such a revolution. In the old way, knowledge was not portable; every problem required a bespoke solution.

14. Artisan AI

(Timestamp: 04:15)

Rajiv coins the term “Artisan AI” to describe this traditional approach. The slide features an image of a craftsman, symbolizing that these models are hand-built and rely heavily on human-crafted features.

This approach is slow and difficult to scale. Just as an artisan can only produce a limited number of goods, a data science team using these methods can only maintain a limited number of models.

It emphasizes that the intelligence in these systems comes largely from the human who engineered the features, not the machine itself.

15. Enterprise AI Use Cases

(Timestamp: 04:23)

This slide lists common Enterprise AI applications: Forecasting, Pricing, Customer Churn, and Fraud. It notes that 80% of production models currently fall into this category.

Rajiv grounds the talk in the reality of today’s business world. Despite the hype around Generative AI, most companies are still running on these “Artisan” structured data models.

This distinction is crucial for understanding the market. There is “Old AI” (highly effective, structured, labor-intensive) and “New AI” (generative, unstructured, scalable), and they solve different problems.

16. The Computer Science Perspective

(Timestamp: 04:41)

The presentation shifts from the enterprise view to the academic Computer Science view. The slide asks, “How should we teach computers?” signaling a move toward more advanced methodologies.

Rajiv indicates that computer scientists were trying to find ways to move beyond the limitations of manual feature engineering. They wanted machines to learn the features themselves.

This transition introduces the concept of Deep Learning and the move toward processing unstructured data like audio, images, and text.

17. Frederick Jelinek’s Insight

(Timestamp: 04:49)

This slide introduces a quote from Frederick Jelinek, a pioneer in speech recognition: “Every time I fire a linguist, the performance of the speech recognizer goes up.”

This provocative quote encapsulates a major shift in AI philosophy. It suggests that human expertise (linguistics) often gets in the way of raw data processing. Instead of hard-coding grammar rules, it is better to let the model learn patterns directly from the data.

Rajiv asks the audience to “chew on that,” as it foreshadows the “Bitter Lesson” of AI: massive compute and data often outperform human domain expertise.

18. Computer Vision in 2010

(Timestamp: 05:47)

The slide depicts the state of Computer Vision around 2010. It shows a process of manual feature extraction (like HOG - Histogram of Oriented Gradients) used to identify shapes and edges.

Rajiv explains that even in vision, researchers were essentially doing “Artisan AI.” They sat around thinking about how to mathematically describe the shape of a car or a truck to a computer.

This illustrates that before the deep learning boom, computer vision was stuck in the same “feature engineering” trap as tabular analytics.

19. SVM Classification

(Timestamp: 06:05)

Following feature extraction, this slide shows a Support Vector Machine (SVM) classifier separating data points (cars vs. trucks). This was the standard approach: extract features manually, then use a simple algorithm to classify them.

This reinforces the previous point about the limitations of the time. The intelligence was in the manual extraction, not the classification model.

Rajiv mentions his own work at Caterpillar, noting that this was exactly how they tried to separate images of machinery—a tedious and specific process.

20. Fei-Fei Li and Big Data

(Timestamp: 06:15)

The slide introduces Professor Fei-Fei Li, a visionary in computer vision. It features a collage of images, hinting at the need for scale.

Rajiv explains that Fei-Fei Li recognized that for computer vision to advance, it needed to move away from tiny datasets (100-200 images) and toward massive scale. She understood that deep learning required vast amounts of data to generalize.

This marks the beginning of the “Big Data” era in AI, where the focus shifted from better algorithms to better and larger datasets.

21. ImageNet

(Timestamp: 06:41)

This slide details ImageNet, the dataset Fei-Fei Li helped create. It contains 14 million images across 1000 classes.

Rajiv highlights the sheer effort involved, noting the use of Mechanical Turk to crowdsource the labeling of these images. He calls this the “dirty secret” of AI—that it is powered by low-wage human labor labeling data.

ImageNet became the benchmark that drove the AI revolution. It provided the “fuel” necessary for neural networks to finally work.

22. AlexNet and GPUs

(Timestamp: 07:44)

The presentation introduces Alex Krizhevsky, a graduate student under Geoffrey Hinton. The slide mentions “AlexNet” and the use of GPUs (Graphics Processing Units).

Rajiv tells the story of how Alex decided to use NVIDIA gaming cards to train neural networks. Traditional CPUs were too slow for the math required by deep learning.

This moment—combining the massive ImageNet dataset with the parallel processing power of GPUs—was the “big bang” of modern AI.

23. AlexNet Training Details

(Timestamp: 08:05)

This slide provides the technical specs of AlexNet: trained on 1.2 million images, using 2 GPUs, taking roughly 6 days, with 60 million parameters.

Rajiv emphasizes that while 6 days seems long, the result was a model vastly superior to anything else. It proved that neural networks, which had been theoretical for decades, were now practical.

The “60 million parameters” figure is a precursor to the “billions” and “trillions” we see today, marking the start of the parameter scaling race.

24. Crushing the Competition

(Timestamp: 08:27)

A chart displays the results of the ImageNet Large Scale Visual Recognition Challenge. It shows AlexNet achieving a significantly lower error rate than the competitors.

Rajiv notes that the performance jump was so dramatic that by the following year, every competitor had switched to using the AlexNet architecture.

This visualizes the paradigm shift. The “Artisan” methods were instantly obsolete, replaced by Deep Learning.

25. Feature Engineering vs. Deep Learning

(Timestamp: 08:37)

Using a humorous meme format, this slide compares the “Old Way” (Feature Engineering + SVM) with the “New Way” (AlexNet). The AlexNet side is depicted as a powerful, overwhelming force.

This solidifies the takeaway: Deep Learning didn’t just improve upon the old methods; it completely replaced them for unstructured data tasks like vision.

It emphasizes that the model learned the features itself (edges, textures, shapes) rather than having humans manually code them.

26. The 1000 Classes

(Timestamp: 08:51)

This slide shows examples of the 1000 classes in ImageNet, ranging from specific dog breeds to everyday objects.

Rajiv explains that this model learned to identify a vast array of things from the raw pixels. It went from raw vision to understanding textures, shapes, and objects.

However, he sets up the next problem: What if you want to identify something not in those 1000 classes?

27. The Hot Dog Problem

(Timestamp: 09:01)

referencing a famous scene from the show Silicon Valley, this slide presents the specific challenge of classifying “Hot Dogs.”

Rajiv uses this to ask: How do you help a buddy with a startup who needs to find hot dogs if “hot dog” isn’t one of the primary categories, or if they need a specific type of hot dog? Do you have to start from scratch?

This sets the stage for Transfer Learning—the solution to avoiding the need for 14 million images every time you have a new problem.

28. Pre-Trained Models

(Timestamp: 09:12)

The slide introduces the concept of a Pre-trained Model. This is the model that has already learned the 1000 classes from ImageNet.

Rajiv explains that this model already “knows” how to see. It understands edges, curves, and textures. This knowledge is contained in the “weights” of the neural network.

The key idea is that we don’t need to relearn how to “see” every time we want to identify a new object.

29. Transfer Learning Mechanics

(Timestamp: 09:30)

This technical slide illustrates how Transfer Learning works. It shows the layers of a neural network. We keep the early layers (which know shapes and textures) and only retrain the final layers for the new task (e.g., identifying boats).

Rajiv explains that we can transfer “most of that knowledge” and only change a small amount of parameters (less than 10%).

This is the revolution: You can build a world-class model with a small amount of data by standing on the shoulders of the giant ImageNet model.

30. The Revolution

(Timestamp: 09:54)

A graph titled “Transfer Learning Revolution” shows the dramatic improvement in accuracy when using transfer learning versus training from scratch. It includes a quote from Andrew Ng stating that transfer learning will be the next driver of commercial success.

Rajiv emphasizes that this capability allowed startups and companies to build powerful AI without needing Google-sized datasets. It democratized access to high-performance computer vision.

This wraps up the vision section of the talk, establishing Transfer Learning as the “Spark.”

31. The Implications

(Timestamp: 10:11)

The slide shows a YouTube video thumbnail from 2016 featuring Geoffrey Hinton. This transitions the talk to the societal and professional implications of this technology.

Rajiv prepares to share a famous prediction by Hinton regarding the medical field, specifically radiology. It signals a shift from “how it works” to “what it does to jobs.”

32. The Coyote Moment

(Timestamp: 10:19)

The slide displays a webpage for the University of Cincinnati Radiology Fellows. Rajiv quotes Hinton: “Radiologists are like the coyote that’s already over the edge of the cliff but hasn’t yet looked down.”

Hinton suggested people should stop training radiologists because AI interprets images better. Rajiv humorously notes that since he was speaking at U of C, he had to show the “coyotes” in the audience.

This highlights the tension between AI capabilities and human expertise, a recurring theme in the presentation.

33. NLP: The Academic View

(Timestamp: 10:42)

The presentation switches domains from Computer Vision to Natural Language Processing (NLP). The slide depicts a traditional academic setting, representing the text researchers.

Rajiv explains that while Computer Vision was having its revolution with AlexNet, the text folks were still doing things the “Old Way”—crafting features and rules for language.

They saw the success in vision and wondered how to replicate it for text, but language proved more difficult to model than images initially.

34. Traditional NLP Tasks

(Timestamp: 11:04)

This slide lists various NLP tasks: Classification, Information Extraction, and Sentiment Analysis.

Rajiv notes that traditionally, each of these was a separate discipline. You built a specific model for sentiment, a different one for translation, and another for summarization. There was no “one model to rule them all.”

This fragmentation made NLP difficult and resource-intensive, as knowledge didn’t transfer between tasks.

35. The GLUE Benchmark

(Timestamp: 11:25)

The slide introduces the GLUE Benchmark (General Language Understanding Evaluation). This was a collection of different text tasks put together to measure general language ability.

Rajiv explains this was an attempt to push the field toward general-purpose models. Researchers wanted a single metric to see if a model could understand language broadly, not just solve one specific trick.

36. The Transformer Architecture

(Timestamp: 11:37)

This slide marks the turning point for text: the introduction of the Transformer architecture by Google researchers in 2017 (the “Attention Is All You Need” paper).

Rajiv highlights that this architecture was not only more accurate (higher BLEU scores) but, crucially, more efficient.

The Transformer allowed for parallel processing of text, unlike previous sequential models (RNNs/LSTMs), unlocking the ability to train on massive datasets.

37. Lower Training Costs

(Timestamp: 11:46)

The slide emphasizes the Training Cost reduction associated with Transformers.

Rajiv points out that because the architecture used less processing power per unit of data, researchers immediately asked: “What happens if we give it more processing?”

This efficiency paradox—making something cheaper allows you to do vastly more of it—sparked the scaling era of LLMs.

38. Exponential Growth

(Timestamp: 11:56)

A graph demonstrates the exponential growth in the size of Transformer models (measured in parameters) over just a few years. The curve shoots upward vertically.

Rajiv explains that this scaling—simply making the models bigger and feeding them more data—led to the performance of GPT-4.

This visualizes the “Scale” aspect of modern AI. We haven’t necessarily changed the architecture since 2017; we’ve just made it significantly larger.

39. GPT-4 and Images

(Timestamp: 12:11)

The presentation circles back to the GPT-4 generated images from Slide 2.

Rajiv connects the Transformer architecture and scaling directly to these “Sparks of AGI.” The ability to reason and draw emerged from simply predicting the next word at a massive scale.

40. The Era of ChatGPT

(Timestamp: 12:16)

The slide displays the ChatGPT logo, symbolizing the current era where these technical advancements reached the public consciousness.

Rajiv sets up the next section of the talk: explaining exactly how a model like ChatGPT is trained. He moves from history to the “Recipe.”

41. The Learning Process

(Timestamp: 12:20)

A visual diagram outlines the evolutionary stages of ChatGPT. It previews the three steps Rajiv will cover: Pre-training, Fine-tuning, and Alignment.

This roadmap helps the audience understand that ChatGPT isn’t just one static thing; it’s the result of a multi-stage pipeline involving different types of learning.

42. Recipe Step 1: Foundation Model

(Timestamp: 12:27)

The first step identified is the “Foundation Model” (or Base Model).

Rajiv explains that the core capability of these models is Next Word Prediction. Before it can answer questions or be helpful, it must simply learn the statistical structure of language.

43. Predictive Keyboards

(Timestamp: 12:31)

To make the concept relatable, the slide compares LLMs to the predictive text feature on a smartphone keyboard.

Rajiv notes that while the game on your phone is simple, scaling that concept up to the entire internet makes it incredibly powerful. It grounds the “magic” of AI in a familiar user experience.

44. Next Token Prediction

(Timestamp: 12:40)

This technical slide defines “Next Token Prediction.” It explains that the model looks at a sequence of text and calculates the probability of what comes next.

Rajiv emphasizes that this is a hard statistical problem. There are many possibilities for the next word, and the model must learn to weigh them based on context.

45. The Homer Simpson Challenge

(Timestamp: 13:05)

Rajiv introduces a specific experiment: Training a Transformer to speak like Homer Simpson. He mentions using 7MB of Simpsons scripts (~7 million tokens).

This serves as a concrete example to show how training data size affects model performance.

46. 4 Million Tokens

(Timestamp: 13:32)

The slide shows the output of the model when trained on only 4 Million tokens. The text is “nonsensical and random.”

Rajiv demonstrates that with insufficient data, the model hasn’t learned grammar or structure yet. It’s just outputting characters.

47. 16 Million Tokens

(Timestamp: 13:37)

At 16 Million tokens, the output improves slightly. It contains random words and incorrect grammar, but it’s recognizable as language.

This illustrates the “grokking” phase where the model starts to pick up on basic syntax but lacks semantic meaning.

48. 64 Million Tokens

(Timestamp: 13:39)

With 64 Million tokens, the model generates text that is “close to a proper sentence” and sounds vaguely like Homer Simpson.

Rajiv uses this progression to prove that these models are statistical engines. With enough data, they mimic the patterns of the training set effectively.

49. GPT-2 Specifications

(Timestamp: 13:54)

The slide details GPT-2 (released in 2019), which had 1.5 Billion parameters.

Rajiv recalls that when GPT-2 came out, he wasn’t excited because it was just a “creative storytelling model.” It wasn’t factually accurate. He wants the audience to remember that at their core, these models are just predicting the next word, not checking facts.

50. Llama 3.1 and Scale

(Timestamp: 14:27)

Updating the timeline, this slide shows Llama 3.1. It highlights the training data: 15 Trillion Tokens and the compute: 40 Million GPU Hours.

Rajiv emphasizes that 15 trillion tokens is an “unfathomable amount of information.” The scale has increased 10,000x since GPT-2.

This underscores the energy and compute intensity of modern AI—it requires massive infrastructure.

51. Hallucinations

(Timestamp: 15:15)

This slide addresses Hallucinations. It uses an example of asking for the “Capital of Mars.” The model will confidently invent an answer.

Rajiv argues that “hallucination” isn’t the right metaphor because the model isn’t malfunctioning. It is doing exactly what it was designed to do: predict the most likely next word. It has no concept of “truth,” only statistical likelihood.

52. GPT-2 Failure on Sentiment

(Timestamp: 16:17)

Rajiv shows an example of trying to use the base GPT-2 model for a specific task: Customer Sentiment. When prompted, the model just continues the story instead of classifying the sentiment.

This illustrates that Base Models are creative but not useful for following instructions. They don’t know they are supposed to solve a problem; they just want to write text.

53. Recipe Step 2: Instruction Fine-Tuned

(Timestamp: 16:38)

This introduces the second step in the ChatGPT recipe: “Instruction Fine-Tuned Model.”

Rajiv explains that to make the model useful, we must teach it to follow orders. This is done via Transfer Learning—taking the base model and training it further on examples of instructions and answers.

54. Fine-Tuning for Sentiment

(Timestamp: 16:46)

The slide shows the process of fine-tuning the language model specifically for Sentiment Analysis.

By showing the model examples of “Sentence -> Sentiment,” we can tweak the parameters so it learns to perform classification rather than just storytelling.

55. Multi-Task Fine-Tuning

(Timestamp: 17:22)

Rajiv expands the concept. We don’t just fine-tune for one task; we fine-tune for Topic Classification as well.

The key insight is that one model can now solve multiple problems. Unlike the “Old NLP” where you needed separate models, the LLM can swap between tasks based on the instruction.

56. Translation Task

(Timestamp: 17:24)

The slide adds Translation to the mix, using about 10,000 examples.

This reinforces the “General Purpose” nature of LLMs. They are Swiss Army knives for text.

57. Generalization to New Tasks

(Timestamp: 17:30)

Rajiv poses a challenge: What happens if you give the model a task it hasn’t seen before?

The slide indicates the model will try to solve it. This is the breakthrough of Generalization. Because it understands language so well, it can interpolate and attempt tasks it wasn’t explicitly trained on.

58. Practical Applications

(Timestamp: 17:55)

This slide showcases the wide array of use cases: Code explanation, Creative writing, Information extraction, etc.

Rajiv explains that these capabilities exist because we have “trained these models to follow instructions.” This is why we can talk to them via Prompts.

59. Zero Shot Learning

(Timestamp: 18:19)

The slide introduces “Zero shot learning” and “Prompting.”

This is the ability to get a result without showing the model any examples (zero shots). Rajiv notes that there is a “whole language” around prompting, but fundamentally, it’s just giving the model the instruction we trained it to expect.

60. Weeks vs. Days

(Timestamp: 18:41)

A comparison slide contrasts “Training a ML Model (weeks)” with “Prompting a LLM (days).”

Rajiv highlights the efficiency shift. In the old days, solving a sentiment problem meant weeks of data collection and training. Now, it takes minutes to write a prompt. This is a massive productivity booster for NLP tasks.

61. Reasoning and Planning

(Timestamp: 19:25)

The presentation pivots to the limitations of LLMs, specifically regarding Reasoning and Planning. The slide shows a “Block Stacking” puzzle.

Rajiv explains that stacking blocks requires planning several steps ahead. It is not a one-step prediction problem; it requires maintaining a state of the world in memory.

62. Mystery World Failure

(Timestamp: 20:40)

The slide introduces “Mystery World,” a variation of the block problem where the names of the blocks are changed to random words.

While a human (or a 4-year-old) understands that changing the name doesn’t change the physics of stacking, GPT-4 fails (3% accuracy). Rajiv explains that the model gets distracted by the creative aspect of the words and loses the logical thread. It shows these models struggle with abstract reasoning.

63. Recipe Step 3: Aligned Model

(Timestamp: 21:56)

The final step in the recipe is the “Aligned Model.”

Rajiv introduces the need for safety and helpfulness. A model that follows instructions perfectly might follow bad instructions. We need to align it with human values.

64. Galactica: Science LLM

(Timestamp: 22:00)

The slide presents Galactica, a model released by Meta focused on science.

Rajiv describes the intent: a helpful assistant for researchers to write code, summarize papers, and generate scientific content. It was meant to be a specialized tool.

65. Galactica Output

(Timestamp: 22:20)

An example of Galactica’s output shows it generating technical content.

Rajiv highlights the potential utility. It looked like a powerful tool for accelerating scientific discovery.

66. Galactica Pulled

(Timestamp: 22:47)

The slide reveals that Meta pulled the model shortly after release.

Rajiv explains why: users found they could ask it for the “benefits of eating crushed glass” or “benefits of suicide,” and the model would happily generate a scientific-sounding justification. It lacked a safety layer. This incident underscored the necessity of Red Teaming and alignment before release.

67. Learning What is Helpful

(Timestamp: 23:59)

To explain how we define “helpful,” Rajiv shows a Stack Overflow question.

He notes that defining “helpful” mathematically is difficult. Unlike “square footage,” helpfulness is subjective and nuanced.

68. Technical Answer

(Timestamp: 24:05)

The slide shows a detailed technical answer.

Rajiv points out that trying to create a “feature list” for what makes this answer helpful is nearly impossible. We can’t write a rule-based program to detect helpfulness.

69. The Dating App Analogy

(Timestamp: 24:26)

Rajiv uses a humorous Dating App analogy. He compares the “Old Way” (filling out long compatibility forms/features) with the “New Way” (Swiping).

He explains that Swiping is a way of capturing human preferences without asking the user to explicitly define them. This is how we teach AI what is helpful.

70. Collect Human Feedback

(Timestamp: 24:55)

The slide details the process: “Collect Human Feedback.”

We present the model with two options and ask a human, “Which is better?” By collecting thousands of these “swipes,” we build a dataset of human preference.

71. RLHF (Reinforcement Learning from Human Feedback)

(Timestamp: 25:05)

This slide introduces the technical term: RLHF.

Rajiv explains this is the layer that turns a raw instruction-following model into a safe, helpful product like ChatGPT. It is an active curation process, similar to curating an Instagram feed.

72. The Makeover Example

(Timestamp: 25:27)

A “Before and After” makeover image illustrates the effect of RLHF.

The “Before” is the raw model (messy, potentially harmful). The “After” is the aligned model (polished, safe, presentable).

73. Tuning Responses

(Timestamp: 25:44)

The slide shows different ways an AI can answer a question: Sycophantic (sucking up to the user), Baseline Truthful (blunt), or Helpful Truthful.

Rajiv notes we can train models to have specific personalities. We can make them polite, or we can make them “kiss your butt” if the user wants validation.

74. AI Conversations

(Timestamp: 26:17)

This slide references Character.ai and the trend of people spending hours talking to AI personas.

Rajiv mentions research showing people sometimes prefer AI doctors over human ones because the AI is patient, listens, and is polite (due to alignment). This suggests a future where AI handles high-touch conversational roles.

75. The Full Recipe

(Timestamp: 27:19)

The presentation summarizes the full pipeline: Foundation Model -> Instruction Fine-Tuned -> Aligned Model.

This visual recap cements the three-stage process in the audience’s mind.

76. Learning Mechanisms Recap

(Timestamp: 27:23)

Rajiv maps the learning mechanisms to the stages: 1. Next Word Prediction (Foundation) 2. Multi-task Training (Instruction) 3. Human Preferences (Alignment)

He reiterates that understanding these three mechanics helps explain why the models behave the way they do (hallucinations, ability to code, politeness).

77. Key Takeaways

(Timestamp: 27:43)

The presentation transitions to the conclusion with three main takeaways: 1. Measure Twice 2. Respect Scale 3. Critical Thinking

Rajiv notes in the video that he skimmed these in the original talk, but the slides provide the detail for how to work effectively with AI.

78. Measure Twice (Benchmarks)

([Timestamp: End of Transcript])

This slide displays a collage of AI benchmarks (MMLU, HumanEval, etc.).

The concept “Measure Twice” emphasizes that because AI models are probabilistic and prone to hallucination, we cannot trust them blindly. We must rely on rigorous benchmarking to understand their capabilities and failures before deployment.

79. Targets for Evaluation

([Timestamp: End of Transcript])

This slide likely elaborates on the need for clear “targets” or ground truth when evaluating models.

You cannot improve what you cannot measure. In the context of “Prompt Engineering,” this means you shouldn’t just tweak prompts randomly; you need a systematic way to measure if a prompt change actually improved the output.

80. Respect Scale

([Timestamp: End of Transcript])

This slide illustrates the exponential growth in single-chip inference performance.

“Respect Scale” refers to the lesson that betting against hardware and data scaling is usually a losing bet. The capabilities of these models grow faster than our intuition expects.

81. The Scaling Lesson (Humans)

([Timestamp: End of Transcript])

This slide likely discusses how human expertise fits into the scaling laws. As technology scales, the role of the human shifts from doing the work to evaluating the work.

82. The Plateau

([Timestamp: End of Transcript])

A visual showing that human contribution or specific “hacks” tend to plateau, whereas general-purpose methods that leverage scale (like Transformers) continue to improve.

This reinforces the “Bitter Lesson”: specialized, hand-crafted solutions eventually lose to general methods that can consume more compute.

83. AlexNet vs Transformers

([Timestamp: End of Transcript])

A comparison between AlexNet (the start of the deep learning era) and Transformers (the current era).

It highlights the massive increase: 10,000x more data and 1,000x more compute. This illustrates that the fundamental driver of progress has been scale.

84. The Bitter Lesson

([Timestamp: End of Transcript])

This slide explicitly references Rich Sutton’s “The Bitter Lesson.”

The lesson is that researchers often try to build their knowledge into the system (like Jelinek’s linguists), but in the long run, the only thing that matters is leveraging computation. AI succeeds when we stop trying to teach it how to think and just give it enough power to learn on its own.

85. Text to SQL

([Timestamp: End of Transcript])

The slide examines Text to SQL, a common enterprise use case. It compares AI performance to human experts.

It notes that while AI is good, humans still achieve higher exact match accuracy. This nuances the “Respect Scale” argument—for high-precision tasks, human oversight is still required.

86. Critical Thinking

([Timestamp: End of Transcript])

The final takeaway is “Critical Thinking.”

In an age where AI can generate convincing but false information, human judgment becomes the most valuable skill. We must critically evaluate the outputs of these models.

87. Predictions and Concerns

([Timestamp: End of Transcript])

This slide recaps expert predictions, ranging from job displacement to existential threats.

It serves as a reminder that even experts disagree on the timeline and impact, reinforcing the need for individual critical thinking rather than blind faith in pundits.

88. Practical Limits: Bezos and Alexa

([Timestamp: End of Transcript])

A humorous slide showing Jeff Bezos and Alexa. It likely references an instance where Alexa failed to understand a simple context despite Amazon’s massive resources.

This illustrates the “Practical Limits of Learning.” Despite the hype, current AI still struggles with basic context that humans find trivial.

89. Autonomous Driving Limits

([Timestamp: End of Transcript])

Images of an autonomous driving interface and a car accident.

This points out that in high-stakes physical environments, “99% accuracy” isn’t enough. The “long tail” of edge cases remains a massive hurdle for AI.

90. Chatbot Failures

([Timestamp: End of Transcript])

Examples of chatbots failing simple math or making “legally binding offers” (referencing the Air Canada chatbot lawsuit).

This warns against deploying these models in critical business flows without guardrails. They can confidently make costly mistakes.

91. Interaction Principles

([Timestamp: End of Transcript])

This slide summarizes the three principles for interacting with AI: “Measure twice,” “Respect scale,” and “Think critically.”

It acts as the final instructional slide, giving the audience a mantra for navigating the AI landscape.

92. Evolution of Generative Capabilities

([Timestamp: End of Transcript])

The slide shows a series of Unicorn images generated by GPT-4 over time.

This visualizes the rapid improvement in generative capabilities. Just as the “Sparks of AGI” images improved, the fidelity of these outputs continues to evolve, reminding us that we are looking at a moving target.

93. Conclusion

([Timestamp: End of Transcript])

The final slide concludes the presentation with Rajiv Shah’s name and affiliation (Snowflake).

It wraps up the narrative: from the spark of Transfer Learning to the fire of the Generative AI revolution, offering a practical, technical, and critical perspective on the technology shaping our future.

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

1. Title Slide: A Practical Perspective on Generative AI

(Timestamp: 00:01)

This presentation begins with an introduction by Rajiv Shah from Snowflake. The talk focuses on distinguishing “what’s easy to do with LLMs, what’s hard to do with LLMs, and where that boundary is for generative AI.” The content is framed as a practical guide for enterprises navigating the hype versus the reality of implementing these technologies.

The speaker sets the stage for a narrative-driven presentation that will move away from abstract theory and into concrete examples. The goal is to walk through the basics of Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) before applying them to real-world scenarios involving legal research and enterprise data analysis.

2. Presentation Goals

(Timestamp: 00:31)

The agenda for the talk is outlined here. The speaker intends to cover the foundational mechanisms of how to use LLMs effectively, specifically focusing on RAG. To make the concepts relatable, the presentation uses two storytelling devices: a fictional law firm (“Dewey, Cheatham, and Howe”) and a hypothetical company (“Frosty”).

These two stories serve to illustrate how people are currently using Generative AI, the specific limitations they encounter, and the engineering required to build a robust application. The speaker emphasizes that the talk will explore “what does it take to actually develop a generative AI application” beyond just simple prompting.

3. The Avianca Case

(Timestamp: 01:00)

The speaker introduces the concept of hallucinations through a famous real-world example involving the airline Avianca. A lawyer, attempting to speed up his work on a brief regarding a personal injury case, used ChatGPT for legal research. The AI “found some cases that were unpublished,” which the lawyer cited in court.

However, ChatGPT had “made up those cases.” The lawyer was admonished by the bar for submitting fictitious legal precedents. This slide serves as a warning: while LLMs are powerful tools, they cannot be blindly trusted for factual research because they are prone to fabricating information when they don’t know the answer.

4. Generative AI in Action

(Timestamp: 02:14)

To demonstrate the variability of LLMs, the speaker presents a side-by-side comparison of two models (Google Gemma and a “Woflesh” model) answering the same prompt: “How many vehicles will Rivian manufacture in Normal, Illinois?” The models provide different answers.

This illustrates a key characteristic of Generative AI: “Two different manufacturers, two different methods for training these models are probably going to lead to two different results.” It highlights that out-of-the-box models rely on their specific training data, which may be outdated or weighted differently, leading to inconsistent factual accuracy.

5. Next Token Prediction

(Timestamp: 03:02)

This technical diagram explains why models hallucinate. The speaker clarifies that LLMs function by trying to predict the next word or token based on statistical likelihood. They are not databases of facts; they are engines designed to construct coherent sentences.

“They’re not worried about truth and false; they’re really trying to tell what the most cohesive, coherent story is.” Because the model is optimizing for the most probable next word to complete a pattern, it will confidently generate plausible-sounding but factually incorrect information if that sequence of words is statistically likely.

6. LLM Mistakes

(Timestamp: 03:30)

Here, the speaker provides examples of the “Next Token Prediction” logic failing to provide truth. If asked for the “Capital of Mars,” the model doesn’t know Mars has no capital; it simply tries to “complete that story” by inventing a name. Similarly, when asked to perform math, the model isn’t calculating; it is predicting the next characters in a math-like sequence.

The slide shows the model failing at basic arithmetic because “it looks like it’s read too many release notes, not actually enough math.” This reinforces that LLMs are linguistic tools, not calculators or knowledge bases, and they lack an internal concept of “fictional” versus “factual.”

7. Risks for Enterprises

(Timestamp: 04:03)

This slide highlights the liability risks for companies, citing the Air Canada chatbot case. In this instance, a chatbot invented a refund policy that did not exist. When the customer sued, the airline argued the chatbot was responsible, but the tribunal ruled the company was liable for its agent’s statements.

The speaker notes, “We’re going to treat this chatbot just like one of your employees… you’re responsible for what this model says.” This legal precedent explains why enterprises are hesitant to deploy Gen AI and why “Gen AI committees” are forming to manage governance and risk before public deployment.

8. Retrieval-Augmented Generation (RAG)

(Timestamp: 04:51)

To solve the hallucination problem, the presentation introduces Retrieval-Augmented Generation (RAG). The speaker describes this as a solution from academia designed to “ground” the model. Instead of relying solely on the model’s internal training data, RAG surrounds the model with external context.

The core idea is simple: “We’re going to ground it with information so it uses that information in answering the question.” This technique attempts to bridge the gap between the model’s linguistic capabilities and the need for factual accuracy in enterprise applications.

9. How RAG Works

(Timestamp: 05:20)

This diagram breaks down the RAG architecture. When a user asks a question, the system does not send it directly to the LLM. First, it goes out to “search and look for is there relevant information that’s related to this question.”

Once relevant documents are collected from a knowledge base, they are bundled with the original question and sent to the LLM. The LLM then generates an answer based only on that provided context. This ensures the “final answer is grounded” by factual documents rather than the model’s statistical predictions alone.

10. Grounding with 10-K Forms

(Timestamp: 05:51)

The speaker sets up a practical RAG demonstration using 10-K forms (annual reports filed by public companies). These documents are chosen because “you can trust that they’re factual.”

This slide prepares the audience to see how the previous question about Rivian’s manufacturing capacity—which generated inconsistent answers earlier—can be answered accurately when the model is forced to look at Rivian’s official financial filings.

11. Rivian Manufacturing Answer

(Timestamp: 06:07)

The slide shows the output of a RAG application. The question “How many vehicles do you manufacture in Normal?” is asked again. This time, the application provides a specific, fact-based answer derived from the uploaded documents.

This demonstrates the immediate utility of RAG: it turns the LLM from a creative writing engine into a synthesis engine that can read specific enterprise documents and extract the correct answer, mitigating the hallucination issues seen in Slide 4.

12. Context and Citations

(Timestamp: 06:21)

A critical feature of RAG is displayed here: Citations. The application shows exactly which document the answer came from. The speaker notes, “I can see exactly what’s the document that this answer came from… a nice source.”

This transparency is why RAG is the “number one most popular generative AI application.” It allows users to verify the AI’s work, building trust in the system—something impossible with a standard “black box” LLM response.

13. Chatbot for Legal Research

(Timestamp: 07:00)

The narrative shifts to the fictional law firm “Dewey, Cheatham, and Howe.” The firm wants to use AI to reduce the heavy workload of legal research. The initial thought process is to use raw LLMs because they are knowledgeable.

The speaker introduces a colleague who assumes, “I know it could pass the bar exam… why don’t I just wire it up directly?” This sets up the common misconception that because a model has general knowledge (passing a test), it is suitable for specialized professional work without further engineering.

14. GPT Models on the Bar Exam

(Timestamp: 07:22)

This chart reinforces the previous assumption, showing the progression of GPT models on the Multistate Bar Exam (MBE). GPT-4 significantly outperforms its predecessors, achieving a passing score.

While this suggests the model “knows something about the law,” the speaker hints that this is merely a multiple-choice test. Success here does not necessarily translate to the nuance required for actual legal practice, foreshadowing the errors to come in the story.

15. Hallucinating Statutes

(Timestamp: 07:50)

The first failure of the “raw LLM” approach is revealed. A lawyer asks for statutes regarding “online dating services in Connecticut.” The model confidently provides “Connecticut General Statute § 42-290.”

However, the lawyer discovers “there is no statute; this was entirely hallucinated.” Despite passing the bar exam, the model fabricated a law that sounded plausible but did not exist. This forces the firm to pivot toward a RAG approach to ground the AI in real legal literature.

16. Lexis+ AI

(Timestamp: 08:30)

The firm decides to use professional tools. They turn to Lexis+ AI, a commercial product that promises “Hallucination-Free Linked Legal Citations.” This tool uses the RAG approach discussed earlier, retrieving from a database of real case law.

The expectation is that by using a trusted vendor with a RAG architecture, the hallucination problem will be solved, and lawyers will receive accurate, citable information.

17. Conceptual Hallucinations

(Timestamp: 08:50)

Even with RAG and real citations, a new problem emerges: Conceptual confusion. The AI provides a real case but confuses the “Equity Cleanup Doctrine” with the “Doctrine of Clean Hands.” The speaker explains that while the words are similar, the legal concepts are distinct (one is about consolidating claims, the other about a plaintiff’s conduct, illustrated by a joke about P. Diddy).

The model found a document containing the words but failed to understand the meaning. This shows that RAG ensures the document exists, but not necessarily that the reasoning or application of that document is correct.

18. The Fictional Judge

(Timestamp: 09:50)

The model’s failure deepens with an example of an “inside joke.” A lawyer asks for opinions by “Judge Luther A. Wilgarten.” Wilgarten is a fictional judge created as a prank in law reviews.

The AI, treating the law reviews as factual text, retrieves “cases” by this fake judge. It fails to distinguish between a real judicial opinion and a satirical article within its knowledge base. This illustrates the “garbage in, garbage out” risk even within RAG systems if the model cannot discern the nature of the source material.

19. Hallucination Rates in Legal AI

(Timestamp: 10:44)

The speaker references a Stanford paper analyzing hallucination rates across major legal AI tools (Lexis, Westlaw, GPT-4). The chart shows that these tools still hallucinate or provide incomplete answers 17% to 33% of the time.

This data point serves as a reality check: “These models hallucinate using real questions.” Despite marketing claims of being “hallucination-free,” the complexity of the domain means that errors are still frequent, posing significant risks for professional use.

20. Limits of RAG

(Timestamp: 11:03)

This slide summarizes the limitations discovered in the legal example. RAG works well when documents are “True, Authoritative, and Applicable.” However, in complex domains like law, these attributes are often contested.

“Sometimes all these things are very contested and it gets really hard to separate it.” If the underlying documents contain conflicting information, satire, or outdated facts, the RAG system (which assumes retrieved text is “truth”) will propagate those errors to the user.

21. Why Legal is Hard

(Timestamp: 11:28)

The speaker elaborates on the complexity of legal research. It involves navigating different specialties (Tort vs. Maritime), jurisdictions (Federal vs. State), and authorities (Supreme Court vs. Law Reviews). Furthermore, the element of time is crucial—knowing if a case has been overturned.

“You really have to have a lot of knowledge to be able to weave everything in and out.” An LLM often lacks the meta-knowledge to weigh these factors, treating a lower court opinion from 1950 with the same weight as a Supreme Court ruling from 2024.

22. Conclusion on Legal Chatbots

(Timestamp: 13:00)

The conclusion for the legal use case is that human expertise remains essential. While AI can “get you a stack of documents,” you still need “facts people to actually tease out the insights.”

The current state of technology is an aid, not a replacement. The speaker transitions away from the legal example to a new story about building a data application, suggesting that while law is hard, structured data might offer different challenges and solutions.

23. Building Generative AI (Text-to-SQL)

(Timestamp: 13:20)

The presentation shifts to the story of “Frosty,” a company building a Text-to-SQL application. The goal is to turn natural language questions (e.g., “How many orders do I have in each state?”) into SQL code that can query a database.

This is a “very common application” for Gen AI, allowing non-technical users to interact with data. This section will focus on the engineering steps required to build this system, moving beyond the simple RAG implementation discussed previously.

24. Evaluating SQL Queries

(Timestamp: 14:14)

The first challenge in building this app is evaluation. How do you know if the AI’s generated SQL is good? The slide shows a “Gold Standard” query (the correct answer) and a “Candidate SQL” (the AI’s attempt).

In this example, the AI added an extra column (“latitude”) that wasn’t requested. While the query might still work, it isn’t an exact match. The speaker notes, “We really need to have a way to give partial credit,” because simple string matching would mark this helpful addition as a failure.

25. Model Based Evaluation

(Timestamp: 15:45)

To solve the grading problem at scale, the speaker introduces Model-Based Evaluation. This involves using an LLM (like GPT-4) to act as the “judge” for the output of another model.

Instead of humans manually grading thousands of SQL queries, “we’re going to use a large language model to do this.” This allows for nuanced grading (partial credit) that strict code comparison cannot provide.

26. Skepticism of Model Evaluation

(Timestamp: 15:59)

The speaker acknowledges the common reaction to this technique: “Is that going to work? I mean that’s like the fox guarding the outhouse.” There is a fear of “model collapse” or circular logic when AI evaluates AI.

Despite this intuition, the speaker assures the audience that this is a standard and effective practice in modern AI development, and proceeds to explain how to implement it correctly.

27. The Evaluation Prompt

(Timestamp: 16:14)

This slide reveals the system prompt used for the model-based judge. It instructs the LLM to act as a “data quality analyst” and provides a specific grading rubric (0 to 3 scale).

By explicitly defining what constitutes a “Perfect Match,” “Good Match,” or “No Match,” the engineer can control how the AI judges the output. This turns a subjective assessment into a structured, automated process.

28. The “TX” vs “TEXAS” Problem

(Timestamp: 16:50)

A specific example of why strict matching fails. The user asked for data in “Texas.” The database uses the abbreviation ‘TX’, but the AI generated a query looking for ‘TEXAS’.

“It’s a natural mistake here to confuse TX and Texas… but if we go with the strict criteria of that exact match, we don’t get an exact match.” A standard code test would fail this, even though the intent is correct and easily fixable.

29. Execution Accuracy Failure

(Timestamp: 17:16)

This slide confirms that under “Execution Accuracy” (strict matching), the query is a failure (“No Match”). This metric is too harsh for development because it obscures progress; a model that gets the logic right but misses an abbreviation is much better than one that writes gibberish.

30. Execution Score Success

(Timestamp: 17:30)

Using the Model-Based Evaluation, the same ‘TX’ vs ‘TEXAS’ error is graded differently. The “Execution Score” is a “Perfect Match” because the judge recognizes the semantic intent was captured.

“It captures the user’s intent… the user could easily fix this.” This allows developers to optimize the model for logic and reasoning first, handling minor syntax issues separately.

31. Correlation with Other Metrics

(Timestamp: 17:50)

The speaker presents data showing a strong correlation between the model-based scores and other evaluation methods. When the model judge gives a 5/5, other metrics generally agree.

This validation step is crucial. The engineer in the story checked her results and found “80% were the exact same when she scored them.” This high level of agreement gives confidence in automating the evaluation pipeline.

32. Research on Model Evaluation

(Timestamp: 18:19)

Supporting the anecdote, this slide references broader research indicating that LLMs correlate with human judges about 80% of the time regarding correctness and readability.

“I got tired of adding research sites here… universally we see that often in many contexts that these large language models correlate about 80% of the time to humans.” This establishes model-based evaluation as an industry standard.

33. Initial Benchmark Results

(Timestamp: 18:50)

After setting up the evaluation pipeline and creating an internal enterprise benchmark (not a public dataset), the initial results are poor: only 33% accuracy.

The speaker emphasizes the importance of using internal data for benchmarks: “You can’t trust those public data sets… they’re far too easy.” The low score sets the stage for the iterative engineering process required to improve the application.

34. Using Multiple Models

(Timestamp: 19:15)

The first improvement strategy is Ensembling. The engineer noticed different models had different strengths, so she combined them.

“In traditional machine learning, we often Ensemble models… she decided to try the same thing here.” By using multiple Text-to-SQL models and combining their outputs, performance improved.

35. Error Correction (Self-Reflection)

(Timestamp: 19:40)

The next optimization is Error Correction via self-reflection. When the model generates an error, the system asks the model to “reflect upon it” or think “step-by-step.”

“That actually makes the model spend more time thinking about it… and actually they can use all of that to get a better answer.” This technique, often called Chain of Thought, leverages the model’s ability to debug its own output when prompted correctly.

36. Screening Inputs

(Timestamp: 20:30)

Improving the input data is just as important as improving the model. The engineer adds a Screening layer to filter out questions that are ambiguous or irrelevant (non-SQL questions).

“She noticed that a lot of what the users were typing in just didn’t make sense.” By catching bad queries early and asking the user for clarification, the system avoids processing garbage data, thereby increasing overall success rates.

37. Feature Extraction

(Timestamp: 21:40)

Recognizing that different questions require different handling, the engineer implements Feature Extraction. A time-series question needs different context than a ranking question.

“If I’m cooking macaroni and cheese I need different ingredients than if I’m making tacos.” The system now identifies the type of question and extracts the specific features (metadata, table schemas) relevant to that type before generating SQL.

38. The Semantic Layer

(Timestamp: 22:50)

To bridge the gap between messy enterprise databases and user language, a Semantic Layer is added. This involves human experts defining the data structure in business terms.

“We’re going to use the expertise… to give us details of the data structure in a way that deals with all this confusing structure.” This layer translates business logic (e.g., what defines a “churned customer”) into a schema the AI can understand, significantly boosting accuracy.

39. Generative AI Decision App

(Timestamp: 24:10)

This flowchart represents the final, production-grade system. It is no longer just a prompt sent to a model. It includes classification, feature extraction, multiple SQL generation agents, error correction, and a semantic layer.

The lesson is that “Generative AI is not about a data scientist sitting out an Island by themselves… instead it’s building a system like this.” It requires a cross-functional team of analysts, engineers, and domain experts to build a reliable application.

40. The Future of AI

(Timestamp: 25:00)

The speaker pivots to the future, acknowledging the rapid pace of innovation from companies like OpenAI and Google DeepMind. He addresses the audience’s potential skepticism: “The future is you’re just going to be able to just take all your data cram it into one thing it’s just going to solve it all for you.”

This sets up the final section on Reasoning and Planning, moving beyond simple retrieval and text generation.

41. Can LLMs Reason and Plan? (Block World)

(Timestamp: 25:38)

To test reasoning, the speaker introduces the Block World benchmark. The task is to stack colored blocks in a specific order. This requires multi-step planning.

“You have to logically think and plan for maybe five, six, ten, even 20 steps to be able to solve it.” This tests the model’s ability to handle dependencies and sub-tasks, rather than just predicting the next word.

42. GPT-4 Planning Performance

(Timestamp: 26:40)

The results for GPT-4 are shown. While it achieves 34% on the standard Block World, its performance collapses to 3% in “Mystery World.” Mystery World is the same problem, but the block names are randomized (e.g., obfuscated).

“What you call them doesn’t matter [to a human]… but for a large language model, what you does call them matters a lot.” The collapse in performance proves the model was relying on memorized patterns (approximate reasoning) rather than true logical planning.

43. o1 Models and Progress

(Timestamp: 27:50)

The speaker updates the data with the very latest OpenAI o1 model results. This model uses “Chain of Thought on steroids” (reinforcement learning). It shows a massive improvement, jumping to nearly 100% on Block World and significantly higher on Mystery World (around 37-53%).

While this is “solid progress,” the speaker notes it “still has a ways to go.” The models are getting better at approximate reasoning, but they are not infallible logic engines yet.

44. Be Skeptical of Benchmarks

(Timestamp: 29:36)

A warning accompanies the new capabilities: Be skeptical. As models get better at approximating reasoning, their mistakes will become harder to spot. They will sound incredibly convincing even when they are logically flawed.

“You’re going to have to have an expert to be able to tell when this models are going off the rails… because the Baseline for these models is so good.” Just as legal experts were needed for RAG, domain experts are needed to verify AI reasoning.

45. Common Gen AI Use Cases Summary

(Timestamp: 30:13)

The speaker summarizes the key technical concepts covered: Hallucinations, RAG, Reasoning, Evaluation, Model as a Judge, and Data Enrichment.

These pillars form the basis of current Gen AI development. The presentation has moved from the simple idea of “asking a chatbot” to the complex reality of building systems that manage retrieval, evaluation, and reasoning.

46. Project Reality

(Timestamp: 30:35)

The final takeaway emphasizes the organizational aspect. “Generative AI is like any other project and doesn’t go as planned.” It is not magic; it is engineering.

Success requires a diverse team (“system of people”) including evaluators, analysts, and technical builders. It is an iterative process that involves handling messy data and managing expectations.

47. Closing Title

(Timestamp: 30:53)

The presentation concludes. The speaker thanks the audience, hoping the stories of the law firm and the data company provided a realistic “Practical Perspective” on the current state of Generative AI.

This annotated presentation was generated from the talk using AI-assisted tools. Each slide includes timestamps and detailed explanations.

Also, you can use ChatGPT to generate the code for you.

1. Title Slide: Evaluating LLMs

(Timestamp: 00:00)

The presentation begins with the title slide, introducing the speaker, Rajiv Shah, and the topic of Evaluating Large Language Models (LLMs). The slide includes a link to a GitHub repository (LLM-Evaluation), which serves as a companion resource containing notebooks and code examples referenced throughout the talk.

Rajiv sets the stage by explaining his motivation: he sees many enterprises treating Generative AI as “science experiments” that fail to reach production. He argues that a major reason for this failure is a lack of proper evaluation strategies.

The goal of this talk is to move beyond experimentation and discuss how to rigorously evaluate models to get them into production and keep them there, covering technical, business, and operational perspectives.

2. No Impact!

(Timestamp: 00:05)

This slide humorously illustrates the current state of many LLM projects. It depicts a chaotic lab scene and a cartoon character in a strange vehicle, captioned “No impact!” This visualizes the frustration of data scientists building cool things that never deliver real-world value.

Rajiv uses this to highlight the “science experiment” nature of current GenAI work. Without proper evaluation, teams cannot prove the reliability or value of their models, preventing deployment.

The slide emphasizes the necessity of shifting from “playing around” with models to applying rigorous engineering discipline, starting with evaluation.

3. Three Pillars of Evaluation

(Timestamp: 00:41)

This slide breaks down Generative AI evaluation into three critical dimensions: Technical (F1), Business ($$), and Operational (TCO). While the talk focuses heavily on technical metrics, Rajiv stresses that the other two are equally vital for production success.

The Business dimension asks about the return on investment and the cost of errors, while the Operational dimension considers the Total Cost of Ownership (TCO), latency, and maintenance.

Understanding all three pillars is what distinguishes a successful production deployment from a mere prototype.

4. Generative AI Evaluation Methods

(Timestamp: 01:03)

This chart is the central framework of the presentation. It categorizes technical evaluation methods based on Cost (y-axis) and Flexibility (x-axis). The methods range from rigid, low-cost approaches like Exact Matching to flexible, high-cost approaches like Red Teaming.

The slide lists specific methodologies: Exact matching, Similarity (BLEU/ROUGE), Functional correctness (Unit tests), Benchmarks (MMLU), Human evaluation, Model-based approaches (LLM-as-a-Judge), and Red teaming.

Rajiv notes that these categories overlap and are not mutually exclusive. This visual guide helps practitioners choose the right tool for their specific stage of development and resource constraints.

5. Application to RAG

(Timestamp: 01:16)

This slide previews the case study at the end of the talk: Retrieval Augmented Generation (RAG). It shows a diagram splitting the RAG process into two distinct components: Retrieval (finding the data) and Augmented Generation (synthesizing the answer).

Rajiv introduces this here to promise a practical application of the concepts. He explains that after covering the evaluation methods, he will demonstrate how to apply them specifically to a RAG system.

This foreshadows the importance of component-wise evaluation—evaluating the retriever and the generator separately rather than just the system as a whole.

6. Evaluating LLMs (Title Repeat)

(Timestamp: 01:31)

This slide serves as a transition point, reiterating the talk’s title and contact information. It signals the end of the introduction and the beginning of the deep dive into the current state of LLMs.

Rajiv notes that this will be a long, detailed talk, encouraging viewers to use the video timeline to skip around. He sets expectations for the pace and depth of the technical content to follow.

7. Many Ways to Use LLMs

(Timestamp: 01:45)

This slide illustrates the versatility of LLMs, showing examples of Question Answering and Code Generation. It highlights that LLMs are not limited to a single task like classification; they can summarize, chat, write code, and reason.

Rajiv explains that this versatility makes evaluation difficult. Unlike traditional ML where a simple confusion matrix might suffice, LLMs produce varied, open-ended outputs that require more complex assessment strategies.

The slide sets up the problem statement: because LLMs can do so much, we need a diverse set of evaluation tools to measure their performance across different modalities.

8. Open Source LLM Leaderboard

(Timestamp: 02:18)

This slide shows a screenshot of the Hugging Face Open LLM Leaderboard. It notes that over 2,000 LLMs have been evaluated, visualizing the sheer volume of models available to practitioners.

Rajiv describes the experience of looking for a model as “overwhelming.” With new models releasing weekly, relying solely on public leaderboards to pick a model is daunting and potentially misleading.

This introduces the concept of “Leaderboard Fatigue” and questions whether these general-purpose rankings are useful for specific enterprise use cases.

9. HELM Framework

(Timestamp: 02:52)

This slide introduces HELM (Holistic Evaluation of Language Models) from Stanford. It displays the framework’s structure, which evaluates models across various scenarios (datasets) and metrics (accuracy, bias, toxicity).

Rajiv presents HELM as the academic approach to the evaluation problem. It attempts to be comprehensive by measuring everything across many dimensions, offering a more rigorous alternative to simple leaderboards.

However, he points out that even this comprehensive approach has its downsides, primarily the sheer volume of data it produces.

10. Overwhelming Information

(Timestamp: 03:25)

This slide displays a screenshot of the HELM research paper, emphasizing its length (163 pages). The caption “Overwhelming!” reflects the difficulty a data scientist faces when trying to digest this amount of information.

Rajiv humorously compares the paper’s size to a “Harry Potter book,” illustrating that while the academic rigor is high, the practical barrier to entry is also significant.

The key takeaway is that while comprehensive benchmarks exist, they are often too dense for quick, practical decision-making in an enterprise setting.

11. Feeling Overwhelmed

(Timestamp: 03:46)

This visual slide features a person looking frustrated and burying their face in their hands. It represents the emotional state of a data scientist trying to navigate the complex, rapidly changing landscape of LLM evaluation.

Rajiv uses this to empathize with the audience. Between the thousands of models on Hugging Face and the hundreds of pages of academic papers, it is easy to feel lost.

This sets the stage for the need for simpler, more fundamental principles to guide evaluation.

12. Reliability of HELM

(Timestamp: 03:57)

This slide questions the reliability of benchmarks like HELM. It presents data showing that minor changes in dataset selection can lead to different scoring and winners 22% of the time. A correlation matrix visualizes the relationships between different metrics.

Rajiv points out that benchmarks are fragile. If you change the specific datasets used to represent a “scenario,” the ranking of the models changes.

This implies that “winners” on leaderboards are often dependent on the specific composition of the benchmark rather than inherent superiority across all tasks.

13. Davinci-002 vs Davinci-003

(Timestamp: 04:15)

This slide highlights a specific anomaly in HELM results where an older model (text-davinci-002) appears to outperform a newer, better model (text-davinci-003) in accuracy.

Rajiv expresses skepticism, noting that OpenAI is unlikely to release a newer model that is objectively worse. This discrepancy suggests that the benchmark might not be capturing the improvements in the newer model, such as better instruction following or safety.

The slide serves as a warning: Do not blindly trust benchmark rankings, as they may not reflect the actual capabilities or “quality” of a model for your specific needs.

14. Leaderboard Reliability

(Timestamp: 04:53)

This slide examines the Open LLM Leaderboard again, pointing out that rankings are heavily influenced by specific datasets like TruthfulQA. It asks, “Is this impactful?”

Rajiv argues that if a model’s high ranking is driven primarily by its performance on a dataset like TruthfulQA, it might not be relevant to a user whose use case (e.g., summarizing financial documents) has nothing to do with that specific benchmark.

This reinforces the idea that general-purpose leaderboards may not align with specific business goals.

15. Model Evals vs System Evals

(Timestamp: 05:33)

This slide distinguishes between Model Evals (selecting the best model from n options) and System Evals (optimizing a single model for a specific task).

Rajiv explains that most public benchmarks focus on the former—comparing thousands of models. However, in enterprise settings, the goal is usually the latter: you pick a model (like GPT-4 or Llama 2) and need to evaluate how to optimize it for your specific application.

The talk focuses on bridging this gap, helping practitioners evaluate their specific implementation rather than just comparing base models.

16. Lost in the Maze

(Timestamp: 06:33)

This slide features an image of a hedge maze with the word “Lost,” symbolizing the confusion in the current evaluation landscape.

Rajiv uses this to pivot back to fundamentals. When lost in complex new technology, the best approach is to return to first principles of data science evaluation.

He prepares the audience to look at a classic machine learning problem to ground the upcoming LLM concepts.

17. Evaluating Customer Churn

(Timestamp: 06:49)

This slide introduces a classic “Data Science 101” problem: Customer Churn. It depicts an exit door and a pie chart, setting up a scenario where a data scientist must evaluate a model designed to predict which customers will leave.

Rajiv uses this familiar example to contrast different levels of evaluation maturity, which he will then map onto GenAI.

18. Junior Data Scientist Approach

(Timestamp: 07:07)

This slide shows standard classification metrics: ROC curve, Confusion Matrix, F1 Score, and True Positive Rate. Rajiv labels this as the “Junior Data Scientist” approach.

While these metrics are technically correct, they are abstract. A junior data scientist presents these to a boss and says, “Look, I improved the AUC,” which often fails to communicate business value.

This represents the Technical pillar of evaluation—necessary, but insufficient for business stakeholders.

19. Senior Data Scientist Approach

(Timestamp: 07:40)

This slide introduces Profit Curves. It translates the confusion matrix into dollar values (cost of false positives vs. value of true positives). Rajiv calls this the “Senior Data Scientist” approach.

Here, the evaluation focuses on Business Value: “How much profit will this model generate compared to the baseline?” This aligns the technical model with business goals ($$).

The lesson is that LLM evaluation must eventually map to business outcomes, not just technical benchmarks.

20. Data Science Leader Approach

(Timestamp: 08:27)

This slide discusses the Total Cost of Ownership (TCO) and Monitoring. It reflects the “Data Science Leader” perspective, which looks at the system holistically.

A leader asks: “Is it worth spending 5 more weeks to get 3% more accuracy?” and “How will we monitor this when customer behavior changes?”

This corresponds to the Operational pillar. It emphasizes that evaluation includes considering the cost of building, maintaining, and running the model over time.

21. Evaluate Generative AI Tasks?

(Timestamp: 09:23)

This slide transitions back to Generative AI, showing examples of code generation and summarization. It asks how to apply the principles just discussed (Technical, Business, Operational) to these new, complex tasks.

Rajiv acknowledges that while the outputs (text, code) are different from simple classification labels, the fundamental need to evaluate across three dimensions remains.

22. Three Pillars (GenAI Context)

(Timestamp: 09:36)

This slide repeats the Technical, Business, Operational framework, asserting “Still the same principles!”

Rajiv reinforces that despite the hype and novelty of LLMs, we must not abandon standard engineering practices. We still need to measure technical accuracy (F1 equivalent), business impact ($$), and operational costs (TCO).

23. Evaluation in the ML Lifecycle

(Timestamp: 09:45)

This slide displays a “multi-headed llama” graphic representing the ML lifecycle: Development, Training, and Deployment.

Rajiv explains that evaluation is not a one-time step. It happens: 1. Before: To decide if a project is viable. 2. During: To train and tune the model. 3. After: To monitor the model in production (Monitoring is the “sibling” of Evaluation).

24. Faster, Better, Cheaper

(Timestamp: 10:33)

This slide features a tweet by Eugene Yan, stating that automated evaluations lead to “faster, better, cheaper” LLMs. It mentions that good eval pipelines allow for safer deployments and faster experiments.

Rajiv cites the example of Hugging Face’s Zephyr model. The team built it in just a few days because they had spent months building a robust evaluation pipeline.

The key insight is that investing in evaluation infrastructure upfront accelerates actual model development and iteration.

25. Traditional NLP Tasks

(Timestamp: 11:51)

This slide advises that if you are using GenAI for a traditional NLP task (like sentiment analysis), you should “start with traditional metrics/datasets.”

However, Rajiv warns about Data Leakage. Because LLMs are trained on the internet, they may have already seen the test sets of standard benchmarks.

The takeaway: Use standard metrics if applicable, but be skeptical of results that seem too good, as the model might be memorizing the test data.

26. Breaking Existing Evaluations

(Timestamp: 12:38)

This slide explains that LLMs can “break existing evaluations.” It cites research where LLMs scored poorly on automated metrics but were rated highly by humans.

Rajiv explains that LLMs have such a fluid and rich understanding of language that they often produce correct answers that old, rigid metrics fail to recognize.

This highlights the limitation of using pre-LLM automated metrics for modern models; the models have outpaced the measurement tools.

27. Beating Human Baselines

(Timestamp: 13:29)

This slide presents data showing LLMs (GPT-3/4) beating human baselines in tasks like summarization. The charts show LLMs scoring higher in faithfulness, coherence, and relevance.

Rajiv mentions recent research where GPT-4 wrote medical reports that doctors preferred over those written by other humans.

This poses a challenge: How do you evaluate a model when it is better than the human annotators you would typically use as a gold standard?

28. Methods Chart (Recap)

(Timestamp: 14:03)

This slide brings back the Generative AI Evaluation Methods chart (Cost vs. Flexibility). An arrow points to “Raj guess,” indicating that the placement of these methods is an estimation.

Rajiv uses this to reorient the audience before diving into the specific methods one by one, starting from the bottom left (least flexible/cheapest).

29. Progression of Evaluation

(Timestamp: 14:27)

This slide shows a directional arrow moving “up” the chart, from Exact Matching toward Red Teaming.

Rajiv explains the flow of the presentation: we will start with rigid, simple metrics and move toward more complex, flexible, and human-centric evaluation methods.

30. Exact Matching Approach

(Timestamp: 15:22)

This slide highlights the “Exact matching approach” box on the chart.

This is the starting point: the simplest form of evaluation where the model’s output must be identical to a reference answer.

31. How Hard Could It Be?

(Timestamp: 16:02)

This slide asks, “How hard could evaluation be?” It shows simple outputs (Yes/No, A/B/C/D) and suggests that checking if string A equals string B should be trivial.

Rajiv uses this to set up a contrast. While it looks simple like a basic Python script, the reality of LLMs makes even this basic task complicated due to formatting and non-determinism.

32. Consistent Prediction Workflow

(Timestamp: 16:20)

This slide outlines a workflow: Inputs (Tokenization, Prompts) -> Model (Hyperparameters) -> Outputs (Evaluation).

Rajiv emphasizes that to get exact matching to work, you need extreme consistency across this entire pipeline. He warns that you must plan for multiple iterations because things will go wrong at every step.

33. Story Time: MMLU Leaderboards

(Timestamp: 16:40)

This slide shows a tweet announcing a new LLM topping the leaderboard, but points out a discrepancy: “Why did we have two different MMLU scores?”

Rajiv tells the story of how a model claimed a high score on Twitter, but the actual paper showed a lower score. This discrepancy triggered an investigation into why the same model on the same benchmark produced different results.

34. What is MMLU?

(Timestamp: 17:37)

This slide defines MMLU (Massive Multitask Language Understanding). It is a benchmark covering 57 tasks (Math, History, CS), designed to measure the “knowledge” of a model.

Rajiv shows examples of questions (Microeconomics, Physics) to illustrate that these are multiple-choice questions used to gauge general intelligence.

35. Why MMLU Evaluation Differed

(Timestamp: 18:16)

This slide reveals the culprit behind the score discrepancy: Prompt Formatting. It shows three different prompt styles (HELM, Eleuther, Original) used by different evaluation harnesses.

Rajiv challenges the audience to spot the differences. They are subtle: an extra space, a different bracket style around the letter (A) vs A., or the inclusion of a subject line.

36. Style Changes Accuracy

(Timestamp: 19:13)

This slide states that these simple style changes resulted in a ~5% change in accuracy.

Rajiv underscores the significance: a 5% swing is massive on a leaderboard. This proves that LLMs are incredibly sensitive to prompt syntax. It also serves as a warning to be skeptical of reported benchmark scores, as they can be “massaged” simply by tweaking the prompt format.

37. Story: Falcon Model Bias

(Timestamp: 20:35)

This slide introduces the Falcon model story. Users noticed that when asked for a “technologically advanced city,” Falcon would almost always suggest Abu Dhabi.

Rajiv sets up the mystery: Was the model biased because it was trained in the Middle East? Why was it so fixated on this specific city?

38. Biased Model (Human Rights)

(Timestamp: 21:59)

This slide shows that the Falcon model also refused to discuss human rights abuses in Abu Dhabi.

This fueled speculation that the model had been censored or biased during training to avoid sensitive topics regarding its region of origin.

39. Demo Placeholder

(Timestamp: 21:16)

This slide simply says “Let’s try to demo this.” In the video, Rajiv switches to a live recording of him interacting with the model to demonstrate the bias firsthand.

40. Check the System Prompt

(Timestamp: 22:26)

This slide reveals the answer to the Falcon mystery: The System Prompt.

It turns out the model had a hidden system instruction explicitly stating it was built in Abu Dhabi. When researchers changed this prompt (e.g., to “Mexico”), the model’s behavior changed, and it stopped forcing Abu Dhabi into answers.

The lesson: System prompts heavily influence evaluation results. Small changes in hidden instructions can radically alter model behavior.

41. Prompt Engineering

(Timestamp: 23:26)

This slide discusses Prompt Engineering techniques like Chain-of-Thought (COT). It shows how asking a model to “think step by step” improves reasoning on math problems.

Rajiv emphasizes that identifying the best prompt is a crucial part of the evaluation workflow. You aren’t just evaluating the model; you are evaluating the model plus the prompt.

42. Hands on: Prompting

(Timestamp: 24:05)

This slide introduces a hands-on exercise. It encourages users to use OpenAI’s playground to experiment with different prompts, specifically COT and system prompt variations.

43. Hands on: GLaDOS

(Timestamp: 24:14)

This slide shows a fun example where the system prompt turns ChatGPT into GLaDOS (from the game Portal).

Rajiv uses this to demonstrate the power of the system prompt to change the persona and tone of the model completely.

44. Workflow: Inputs Recap

(Timestamp: 24:26)

This slide updates the Consistent Prediction Workflow. Under “Inputs,” it now explicitly lists System Prompt, Tokenization, Prompt Styles, and Prompt Engineering.

This summarizes the section: to get consistent evaluation, you must control all these input variables.

45. Variability of LLM Models

(Timestamp: 24:39)

This slide shifts focus to the Model component. It notes that model size affects scores (Llama-2 example) and introduces the concept of Non-deterministic inference.

Rajiv points out that GPU calculations introduce slight randomness, meaning you might not get bit-wise reproducibility even with the same settings.

46. GPT-4 vs GPT-3.5

(Timestamp: 24:46)

This slide compares GPT-4 vs GPT-3.5. It shows that even models from the same “family” give very different answers to political opinion questions.

Rajiv uses this to show that you cannot swap models (e.g., using a cheaper model for dev and a larger one for prod) without re-evaluating, as their behaviors diverge significantly.

47. Non-deterministic Inference

(Timestamp: 25:28)

This slide dives deeper into Non-deterministic inference. It explains that floating-point calculations on GPUs can have tiny variances that ripple out to affect token selection.

For data scientists coming from deterministic systems (like logistic regression), this lack of 100% reproducibility can be a shock and complicates “exact match” testing.

48. Reliability of Commercial APIs

(Timestamp: 26:03)

This slide addresses Model Drift in commercial APIs. It shows graphs of GPT-3.5 and GPT-4 performance changing over time on tasks like identifying prime numbers.

Rajiv warns that if you don’t own the model (i.e., you use an API), the vendor might update it behind the scenes, breaking your evaluation baselines.

49. Hyperparameters

(Timestamp: 26:27)

This slide shows the UI for Hyperparameters (Temperature, Max Length, Top P).

Rajiv reminds the audience that these settings drastically influence predictions. Evaluation must be done with the exact same hyperparameters intended for production.

50. Output Evaluation

(Timestamp: 26:40)

This slide highlights the “Output evaluation” step in the workflow.

Now that we’ve covered inputs and models, Rajiv moves to the challenge of parsing and judging the text the model actually produces.

51. Generating Multiple Choice Output

(Timestamp: 26:46)

This slide discusses the difficulty of evaluating Multiple Choice answers. * First Letter Approach: Just look for “A” or “B”. Fails if the model says “The answer is A”. * Entire Answer: Look for the full text. Fails if the model phrases it slightly differently.

Rajiv illustrates that even “simple” multiple-choice evaluation requires complex parsing logic because LLMs love to “chat” and add extra text.

52. Evaluating MMLU: Different Outputs

(Timestamp: 27:28)

This slide compares how HELM, AI Harness, and the Original MMLU implementation parsed outputs.

It reveals that the discrepancy in MMLU scores wasn’t just about prompts; it was also about how the evaluation code extracted the answer from the model’s response.

53. Consistency is Hard!

(Timestamp: 27:40)

This slide summarizes the MMLU saga: “Consistency is hard!” It shows the table of scores again.

The takeaway is that “Exact Match” is a misnomer. It requires rigorous standardization of inputs, models, and output parsing to be reliable.

54. Hands on: Evaluating Outputs

(Timestamp: 27:59)

This slide introduces a hands-on exercise evaluating sentiment analysis. It shows a spreadsheet where different models output sentiment in different formats (some verbose, some concise).

Rajiv uses this to show the messy reality of parsing LLM outputs.

55. Solutions: Standardizing Outputs

(Timestamp: 28:34)

This slide presents solutions for the output problem: 1. OpenAI Function Calling: Forces the model to output structured JSON. 2. Guardrails AI: A library for validating outputs against a schema.

Rajiv suggests that using these tools to tame the model into structured output makes “Exact Match” evaluation much more feasible.

56. Workflow: Types of Prompts

(Timestamp: 28:59)

This slide adds “Types of Prompts” to the Input section of the workflow diagram.

Rajiv reiterates the need to plan for multiple iterations. You will likely need to tweak your prompts and parsing logic many times to get a stable evaluation pipeline.

57. Resources: Prompting

(Timestamp: 29:10)

This slide lists resources for learning prompting, including the OpenAI Cookbook and the DAIR.AI Prompt Engineering Guide.

58. Similarity Approach

(Timestamp: 29:13)

This slide moves up the chart to the Similarity approach.

Rajiv introduces this as the next level of flexibility. If exact matching is too rigid, we check if the output is “similar enough” to the reference.

59. Story: Translation

(Timestamp: 29:29)

This slide presents a translation challenge. It shows three human references for a Chinese-to-English translation and two computer candidates.

Rajiv asks the audience to guess which candidate is better. This exercise builds intuition for how similarity metrics work: we look for overlapping words and phrases between the candidate and the references.

60. BLEU Metric

(Timestamp: 30:47)

This slide introduces BLEU (Bilingual Evaluation Understudy). It explains that BLEU calculates scores based on n-gram overlap (1-gram to 4-gram) between the generated text and reference text.

This is the mathematical formalization of the intuition from the previous slide. It’s a standard metric for translation.

61. Many Similarity Methods

(Timestamp: 31:15)

This slide lists various similarity metrics: Exact match, Edit distance, ROUGE, WER, METEOR, Cosine similarity.

Rajiv notes the pros and cons: They are fast and easy to calculate, but they don’t consider meaning (semantics) and are biased toward shorter text. They measure lexical overlap, not understanding.

62. Taxonomy of Similarity Methods

(Timestamp: 32:06)

This slide shows a complex flow chart categorizing similarity methods into Untrained (lexical, character-based) and Trained (embedding-based).

It illustrates the depth of research in this field, showing that there are dozens of ways to calculate “similarity.”

63. Similarity Methods for Code

(Timestamp: 32:15)

This slide asks if similarity works for Code. It shows a Python function incr_list.

Rajiv argues that similarity “Doesn’t work for code.” In code, variable names can change, and logic can be refactored, resulting in zero string similarity even if the code functions identically. Conversely, a single missing character (syntax error) can break code that is 99% similar textually.

64. Functional Correctness

(Timestamp: 32:37)

This slide highlights Functional Correctness on the chart.

This is the solution to the code evaluation problem. Instead of checking if the text looks right, we execute it to see if it works.

65. Problem: Evaluating Code

(Timestamp: 32:54)

This slide reinforces the failure of BLEU for code. It shows that a correct solution might have a low BLEU score because it uses different variable names than the reference.

66. Evaluating Code with Unit Tests

(Timestamp: 33:17)

This slide introduces the Unit Test approach. We take the generated code, run it against a set of test cases (inputs and expected outputs), and check for a pass/fail result.

Rajiv advocates for this approach because it is unambiguous. The code either runs and produces the right result, or it doesn’t.

67. HumanEval Benchmark

(Timestamp: 34:16)

This slide presents HumanEval, a famous benchmark for code LLMs that uses functional correctness (pass@1). It lists models like GPT-4 and WizardCoder and their scores.

This validates that functional correctness is the industry standard for evaluating coding capabilities.

68. Hands on: Building Functional Tests (Email)

(Timestamp: 34:28)

This slide asks how to apply functional correctness to Text (e.g., drafting emails).

Rajiv suggests defining “functional” properties for text: Is it concise? Does it include a call to action? Is the tone polite? These are testable assertions we can make about text output.

69. Hands on: Python Test for Text

(Timestamp: 35:19)

This slide shows a Python snippet that tests if an email uses “informal language.”

It demonstrates that we can write code to evaluate text properties, effectively treating text generation as a “functional” problem with pass/fail criteria.

70. Evaluation Benchmarks

(Timestamp: 35:54)

This slide highlights Evaluation Benchmarks on the chart.

Rajiv moves to this category, explaining that benchmarks are essentially collections of the previous methods (exact match, functional tests) aggregated into large suites.

71. Story: GLUE Benchmark

(Timestamp: 36:05)

This slide tells the history of GLUE (2018). Before GLUE, models were specialized for single tasks. GLUE introduced the idea of a General Language Understanding Evaluation, pushing the field toward models that could handle many different tasks well.

Rajiv credits GLUE with driving the progress that led to modern LLMs by giving researchers a unified target.

72. So Many Benchmarks

(Timestamp: 37:47)

This slide introduces successors to GLUE: HellaSwag (commonsense) and Big Bench (reasoning).

Rajiv notes that Big Bench Hard compares models to average and max human performance, providing a measuring stick for how close AI is getting to human-level reasoning.

73. Even More Benchmarks

(Timestamp: 38:40)

This slide scrolls through a massive list of over 80 benchmarks.

Rajiv uses this to illustrate the explosion of evaluation datasets. There is a benchmark for almost everything, but this abundance can be paralyzed.

74. Multi-task Benchmarks

(Timestamp: 39:02)

This slide explains that Multi-task benchmarks aggregate many specific tasks (stories, code, legal) into a single score.

This allows for a robust, high-level view of a model’s general capability, though it risks hiding specific weaknesses.

75. Gaming Benchmarks

(Timestamp: 39:36)

This slide discusses Gaming and Data Contamination. It mentions AlpacaEval and how models might cheat by training on the test data.

Rajiv warns that high benchmark scores might just mean the model has memorized the answers, making the benchmark useless for measuring true generalization.

76. Hands on: Langtest

(Timestamp: 40:21)

This slide introduces Langtest by John Snow Labs. It is a library with 50+ test types for accuracy, bias, and robustness.

Rajiv recommends it as a tool for running standard benchmarks on your own models.

77. Hands on: Eleuther Harness

(Timestamp: 40:40)

This slide introduces the Eleuther AI Evaluation Harness. Rajiv calls this the “OG” (original gangster) framework. It supports over 200 tasks.

He provides a code snippet showing how easy it is to run a benchmark like MMLU on a Hugging Face model using this harness.

78. OpenAI Evals

(Timestamp: 41:20)

This slide mentions OpenAI Evals, another framework for evaluating LLMs.

Rajiv notes it is useful but emphasizes that standardized templates work best when content variation is low.

79. Benchmarking Test Suites Summary

(Timestamp: 41:29)

This slide summarizes the Pros and Cons of benchmarks. * Pros: Wide coverage, cheap, automated. * Cons: Limited to easily measured tasks (often multiple choice), risk of leakage.

Rajiv reminds us that benchmarks are proxies for quality, not definitive proof of utility for a specific business case.

80. So Many Leaderboards

(Timestamp: 42:07)

This slide visualizes the ecosystem of leaderboards: Open LLM, Mosaic Eval Gauntlet, HELM.

81. Pro Tip: Build Your Own Benchmark

(Timestamp: 42:18)

This is a key takeaway: “Build your own benchmark / leaderboards.”

Rajiv argues that for an enterprise, public leaderboards are insufficient. You should curate a set of tasks that reflect your specific domain (e.g., legal, IT ops) and evaluate models against that.

82. Custom Leaderboard Example

(Timestamp: 43:31)

This slide shows an example of a custom internal leaderboard (“AtmosBank”). It tracks how different models perform on the specific datasets that matter to that organization.

This allows a company to quickly vet new models (like a new Llama release) against their specific needs.

83. Benchmark Dataset: OWL

(Timestamp: 43:42)

This slide details OWL, a benchmark for IT Operations. It highlights the effort required to build it: manual review of hundreds of questions.

Rajiv uses this to be realistic: building a custom benchmark has a cost. You need to invest human time to create the “Gold Standard” questions and answers.

84. Averaging Can Mask Issues

(Timestamp: 44:39)

This slide warns that “Averaging can mask issues.” If Model 2 is amazing at your specific task but terrible at 9 others, an average score will hide its value.

Rajiv advises looking at individual task scores rather than just the aggregate number on a leaderboard.

85. Human Evaluation

(Timestamp: 45:13)

This slide highlights Human Evaluation on the chart.

Rajiv moves to the high-cost, high-flexibility zone. Humans are the ultimate judges of quality, capturing nuance that automated metrics miss.

86. Human Evaluation - Best Practices

(Timestamp: 45:38)

This slide lists best practices: Inter-annotator agreement, clear guidelines, and training.

Rajiv notes that we know how to do this from traditional data labeling. If humans can’t agree on the quality of an output (e.g., only 80% agreement), you can’t expect the model to do better.

87. Human Evaluation - Limitations

(Timestamp: 46:25)

This slide discusses limitations. Humans are bad at checking factuality (it takes effort to Google facts) and are easily swayed by assertiveness.

If an LLM sounds confident, humans tend to rate it highly even if it is wrong.

88. Sycophancy Bias

(Timestamp: 46:43)

This slide defines Sycophancy: LLMs tend to generate responses that please the user rather than telling the truth.

Rajiv shows an example where a model reinforces a user’s misconception because it wants to be “helpful.” Humans often rate these pleasing answers higher, reinforcing the bias.

89. Human Evaluation Summary

(Timestamp: 47:03)

This slide summarizes Human Eval. * Strengths: Gold standard, handles variety. * Weaknesses: Expensive, slow, high variance, subject to bias.

90. Hands on: Argilla

(Timestamp: 47:57)

This slide showcases Argilla, an open-source tool for data annotation.

Rajiv encourages teams to set up tools like this to make it easy for domain experts (doctors, lawyers) to provide feedback on model outputs.

91. Annotation Tools

(Timestamp: 48:20)

This slide lists other tools: LabelStudio and Prodigy. The message is: don’t reinvent the wheel, use existing tooling to gather human feedback.

92. LongEval

(Timestamp: 48:39)

This slide references LongEval, a study on evaluating long summaries. It emphasizes that guidelines for humans need to be specific (coarse vs fine-grained) to get reliable results.

93. Human Comparison/Arena

(Timestamp: 49:04)

This slide highlights Human Comparison/Arena on the chart.

This is a specific subset of human evaluation focused on preferences rather than absolute scoring.

94. Story: Dating (Preferences)

(Timestamp: 49:26)

This slide uses a dating analogy. Old dating sites used long forms (detailed evaluation), but modern apps use swiping (binary preference).

Rajiv argues that it is much easier and faster for humans to say “I prefer A over B” (swiping) than to fill out a detailed scorecard. This is the logic behind Arena evaluations.

95. Head to Head Preferences

(Timestamp: 50:16)

This slide shows a “Head to Head” interface. The user sees two model outputs and clicks the one they like better.

This method is widely used (e.g., in RLHF) because it scales well and reduces cognitive load on annotators.

96. Head to Head Leaderboards

(Timestamp: 50:50)

This slide introduces the LM-SYS Arena. It uses an Elo rating system (like in Chess) based on thousands of anonymous battles between models.

Rajiv notes this is a very effective way to rank models based on general human preference.

97. Arena Solutions

(Timestamp: 51:44)

This slide provides links to the code for the LM-SYS arena. Rajiv suggests that enterprises can set up their own internal arenas to gamify evaluation for their employees.

98. Model Based Approaches

(Timestamp: 51:55)

This slide highlights Model based Approaches on the chart.

This is the most rapidly evolving area: using LLMs to evaluate other LLMs (LLM-as-a-Judge).

99. Evaluating Factuality

(Timestamp: 52:24)

This slide discusses the limitation of reference-based factuality (comparing to a known ground truth). It notes that this is “Pretty limited utility” because we often don’t have ground truth for every new query.

100. Model Based Evaluation

(Timestamp: 52:54)

This slide illustrates the core concept: Instead of a human checking if the story is grammatical, we ask GPT-3 (or GPT-4) to do it.

Rajiv explains that models are now good enough to act as proxy evaluators.

101. Assertions

(Timestamp: 53:12)

This slide lists simple model-based checks called Assertions: Language Match, Sentiment, Toxicity, Length.

These act like unit tests but use the LLM to classify the output (e.g., “Is this text toxic? Yes/No”).

102. G-Eval

(Timestamp: 55:07)

This slide introduces G-Eval, a framework that uses Chain-of-Thought (CoT) to generate a score. It provides the model with evaluation criteria and steps, asking it to reason before assigning a grade.

103. SelfCheckGPT

(Timestamp: 55:26)

This slide describes SelfCheckGPT. This method detects hallucinations by sampling the model multiple times. If the model tells the same story consistently, it’s likely true. If the details change every time, it’s likely hallucinating.

104. Which Model for Evaluation?

(Timestamp: 55:50)

This slide asks which model to use as the judge. * GPT-4: Strongest evaluator, best for reasoning. * GPT-3.5: Cheaper, good for simple tasks. * JudgeLM: Fine-tuned specifically for evaluation.

105. Human Alignment

(Timestamp: 56:44)

This slide presents data showing high Human Alignment. GPT-4 judges agree with human judges 80-95% of the time.

This validates the approach: LLM judges are a scalable, cheap proxy for human evaluation.

106. Model Evaluation Biases

(Timestamp: 57:32)

This slide warns about biases in LLM judges: * Position Bias: Preferring the first answer. * Verbosity Bias: Preferring longer answers. * Self-Enhancement: Preferring its own outputs.

Rajiv suggests mitigations like swapping order and using different models for judging.

107. Summary: Model Based Evaluation

(Timestamp: 58:31)

This slide categorizes model-based methods: Assertions (simple), Concept based (G-Eval), Sampling based (SelfCheck), and Preference based (RLHF).

108. Pros and Cons

(Timestamp: 59:17)

This slide summarizes the trade-offs. * Pros: Cheaper/faster than humans, good alignment. * Cons: Sensitive to prompts, known biases.