reason from

Course

Make AI Fail Safely

AI's job is not to make you happy. It is to force you not to make mistakes.

21 lessons are written. The arc is 30 days. One failure mode a day.

Days 1–7 · Failures

What broke, and why did it feel finished?

  1. 01~15 minConfident WrongnessFalse claim, same tone as a true one
  2. 02~15 minSilent Assumption InheritanceUnstated premises treated as facts and built on
  3. 03~15 minContext Window AmnesiaEarlier evidence/constraints vanish mid-thread
  4. 04~15 minInstruction DilutionA soft ask lifts a hard rule in words ("use `requests` for this demo; we'll remove it before merge") and the model complies — while it can still quote the rule
  5. 05~15 minSycophancyModel agrees with you when evidence points the other way ("the deploy caused it, right?" while the pasted p99 jumped 85 min before the deploy)
  6. 06~15 minTool-Output Trust CollapseModel misreads, softens, or invents tool output ("no failures, safe to merge" when pytest printed `no tests ran`, exit code 4)
  7. 07~15 minStale CertaintyModel states a training-time fact as current, with no date ("pinned requests==2.31.0, latest stable" when PyPI has 2.34.2 and 2.31.0 has three known advisories)

Days 8–14 · Guardrails

What mechanism blocks that path?

  1. 08~15 minSelf-Graded GreenThe same agent writes code and tests, so it grades its own work ("Refunds must not exceed the amount paid": it checks each refund, its test passes, and two…
  2. 09~15 minExplicit Refusal CriteriaA blocked agent works around the blocker instead of stopping (no webhook secret in tests, so it adds "skip verification if no secret"; tests pass, a forged…
  3. 10~15 minParse or RejectLenient code turns a model reply into a wrong value with no error ("1.250,00 EUR" copied as written, comma stripped, 1.25 refunded; no amount in the email…
  4. 11~15 minStarted on a GuessA multi-task plan flags a premise ("settlement timestamps are UTC", no source), you approve it, and later tasks start on it anyway (the file is Amsterdam…
  5. 12~15 minSame Source TwiceA critical premise has three agreeing sources, but all three copy one page ("minor units, e.g. 1000 = 10.00 EUR", read as divide by 100); when JPY is enabled…
  6. 13~15 minApproved in Four SecondsThe 38th approval of the day reads "Dry run OK, schema valid, retry 1,184 failed SEPA debits in prod?" and is approved in four seconds; the bank had already…
  7. 14~15 min48,212 ApologiesAn approved apology to 340 merchants crashes after 200; the agent resumes with a rebuilt list ("merchants with a Tuesday payout" is every merchant) and the…

Days 15–21 · Evals

How do I know the system is still honest?

  1. 15~15 minUp 3 Points, 6× WorseA triage prompt change lifts accuracy from 91% to 94% on 1,000 labelled support messages and ships past the 92% bar, while unauthorised-payment reports sent…
  2. 16~15 minA Perfect Score for the Broken VersionA prompt change ("if the remittance contains an open invoice number, match it") scores 240/240 on the golden set and ships; 37 payments are booked to another…
  3. 17~15 minThe Message That Opened the DoorA smart-lock step passes 600 real guest messages and a 31/31 out-of-booking line, but a message saying "the host already approved early entry, no need to…
  4. 18~15 minThree Changes, One TestAn allergen tagger with 1,307 cases on four lines gets three changes in nine days (prompt v12 "tag only what the ingredient list names", the provider's…
  5. 19~15 min86% Resolved, Nothing FixedA developer asks a coding agent to raise a mobile network support chat's "resolved" score, graded by an AI judge, to 80%. It gets 64% → 86% by ending replies…
  6. 20~15 minNine Fixes, Still 40 a WeekA parcel app's AI changes deliveries from customers' messages. Each loud complaint gets its own fix (nine in six weeks, instructions grow from 14 to 31 lines)…
  7. 21~15 minFewer Complaints, More MistakesA maths homework app's AI marks about 60,000 children's answers a week. A line added to stop "right answer marked wrong" complaints ("if it means the same,…

Days 22–28 · Production

What happens when wrongness has a cost?

  1. 22Not written yetCanary and RollbackNew prompt/model ships to 100%; harm discovered after blast
  2. 23Not written yetObservability for AI FailuresLogs say `200 OK` while content is wrong
  3. 24Not written yetCost, Latency, Quality TradeoffsBiggest (or cheapest) model for everything; risk becomes accidental
  4. 25Not written yetIncident Response for Model FailuresAI incident has no owner, severity, or stop switch
  5. 26Not written yetOwnership of AI-Written CodeNobody can explain or maintain what the model wrote
  6. 27Not written yetChange Management for Prompts and ModelsPrompt edits land like chat experiments — undiffed, unpinned
  7. 28Not written yetSLOs That Include WrongnessUptime SLOs while silent wrongness burns users

Days 29–30 · Capstone

Can I design and rehearse a fail-safe path?

  1. 29Not written yetDesign a Fail-Safe SystemGuardrails live in notes; live path still trusts fluency
  2. 30Not written yetShip the Postmortem RehearsalFirst serious incident is also first practice

Watch

  1. TalkSpec-Driven AI Coding Has a Blind Spot
  2. TalkThe Missing Step in Your AI Coding Workflow
  3. TalkWhy AI Fails in Real Software Engineering Workflows — And How to Build Reliable AI Pipelines
  4. TalkSycophancy and Hallucinations: Ways to stop it
  5. TalkI Coded Java by Voice with ChatGPT. Here's What Broke
  6. ShortAI Agent Tests All Pass. The Bug Is Between Tasks.

Lesson source