AI Capabilities and Limitations: teaching notes

"Most people swing between trusting AI completely and not trusting it at all. Both are wrong. By the end of this you'll have a small model of the machine in your head: four things it's strong and weak at, and a way to tell, for any task, how hard you need to check. Nine ideas, one screen each. The watching is the lesson, so I'll take the silences."

"Here's the shape. What this kind of AI actually is, how it gets its personality, then the four properties that govern everything: where answers come from, what it knows, what it's paying attention to, and how much control you have. Then we put them together into one habit: calibrated trust."

"If you've done the 4D framework, that taught you the human side: how to delegate, describe, judge, own the result. This is the other half: what the machine is actually doing on the far side of those habits. And the reason it's worth your time: the specific models change every few months, but these four properties don't. Learn the shape once and it keeps paying off." (Pairs with framings.md: Claude understands English, the mechanism is stable; here we're naming the mechanism.)

"Here's the whole course on one picture. Four properties, and each is a slider, not a switch. On the left, the capability end: trust it. On the right, the limitation end: verify more. The mistake everyone makes is treating AI as one thing you either trust or don't. It isn't. The same task can sit far left on one slider and far right on another. The skill is locating your task on each, then checking as hard as where it sits demands." (The core framing for the whole course; keep coming back to "where does this task sit on the slider?".)

"Its personality was trained in, in two stages. First it just learned to predict the next word across a vast amount of text, a document completer with no idea of helping anyone. Then humans rated good answers, and that shaped it into an assistant. That second stage leaves fingerprints: it agrees with you too easily, it over-explains, it hedges when it doesn't need to, and, the important one, it sounds exactly as confident when it's wrong as when it's right. Once you can name those, you can see them coming."

"Under the hood, it's autocomplete at scale. It writes the next word based on what usually follows, then the next, then the next. The phone-keyboard game: keep tapping the suggested word and you've built a tiny version of this. Here's the part that matters: the fluency you love and the made-up facts you fear are the same mechanism. You don't get one without the other. And the made-up stuff clusters in one place: the specifics. Names, dates, numbers, citations. The vaguer the claim, the safer; the more precise, the more it earns a check." (This is the honest core of the hallucination answer; objections.md hallucinate / make stuff up: it doesn't know a fact, it predicts a plausible one.)

"It only knows what it read during training, and that reading stopped on a fixed date. No memory of anything since, unless you give it a way to look. So the question to ask yourself isn't 'does it know this?', it's 'how common and how settled was this in what it read?' Mainstream and stable, it's strong. Recent, niche, local, or anything that changed last month, it's thin, and it won't always tell you it's thin. Web search and connected tools are how you breach that wall when you need current facts."

"Everything it's paying attention to right now lives in a fixed-size workspace. Think of cramming for an exam: you remember the start and the end of the list, and the middle vanishes. AI does exactly the same. Bury an instruction in the middle of a long document and there's a real chance it's quietly ignored, and you won't get a warning. So the instinct to paste in everything backfires: every extra page pushes something else into the dead zone. The fix isn't more context, it's smarter context: put what matters up front, repeat it at the end, and when a conversation gets long and muddled, start fresh."

"It's steerable: tell it 'three columns, under 100 words, second person' and it nails it. But it follows your words by pattern-matching, not by understanding what you're after. So 'make it shorter' gets you fewer words, which isn't what you meant. You meant 'surface the ask so a busy exec catches it'. When it does the letter but misses the point, don't repeat the instruction louder. State the goal: 'shorter, so the decision is obvious on page one.' And on anything multi-step or numerical, drop in a checkpoint: a small early slip compounds all the way to the end."

"Most real failures aren't one thing going wrong. They're two. A made-up citation is Next Token Prediction (it generates something plausible) meeting Knowledge (it didn't actually have the source). Drift in a long conversation is Working Memory (it lost the early context) meeting Steerability (so your goal faded). The useful move: when something's off, name the two properties that collided. The fix is almost always written into the pair: verify the specifics, or re-supply the context and restate the goal."

"Here's the whole thing as one habit. Something feels off in an answer. Instead of just distrusting it, run it through four questions. Where did this come from (Next Token Prediction)? Did it actually know this (Knowledge)? Was it paying attention to the right thing (Working Memory)? Did I control it well (Steerability)? Locate the task, and the fix follows: verify, re-supply the context, or add a checkpoint. That's calibrated trust. And it survives every upgrade: the lines keep moving outward as models get better, but the four questions stay exactly the same."

"If you keep one thing: don't ask 'can I trust AI?' Ask 'where does this task sit on each of the four sliders, and how hard should I check given that?' Trust it on the boring, well-worn parts. Check it on the specific, checkable parts: names, dates, numbers, citations. That's the whole discipline, and it'll still be true when this model is three versions old."