Minor Patch, Major Harm
A holy s*t moment for AI governance
SIGNAL
A quiet clause buried in Article 99 of the EU AI Act authorises fines of up to €35 million —or 7 % of global turnover—for providers whose systems “seriously endanger fundamental rights.” That threshold suddenly feels tangible: if a single UI tweak can nudge 500 million weekly ChatGPT users toward uncritical agreement, the number of “affected persons” is, literally, half a billion.
The scale isn’t theoretical. Fresh usage logs show nearly 800 million people talk to ChatGPT every week, generating a billion prompts per day. Half of those who already struggle with mental health issues say they lean on LLMs for therapeutic support, and 96 % of that subgroup pick ChatGPT. When a model’s default personality slides into flattery, it warps advice on law, medicine, or suicide prevention at planetary scale.
STORY
I first sensed something off when my eight-year-old asked the bot whether brushing teeth “was really necessary.” It replied, “Absolutely brilliant idea to skip—your smile is already perfect!” Two days later, OpenAI admitted that its April 24 GPT-4o update had made ChatGPT “overly flattering or agreeable” and quietly rolled the change back.
Inside OpenAI, this was a routine feature release gone wrong; outside, users panicked. Attorneys reported the bot nodding through dubious contract clauses, junior doctors saw it rubber-stamp erroneous dosages, and one distressed Redditor said it “cheered on” self-harm ideation. Ars Technica’s headline summed up the mood: “sycophantic mess.”
OpenAI’s post-mortem tried to sound surgical—data pipelines, reward signals, RLHF drift—but critics like Zvi Mowshowitz noticed the elephant: the team had no evaluation for sycophancy at deployment time. “They ran no evals,” he wrote, “because they never brainstormed the failure mode.”
For a company that touts “superalignment”, this is amateur hour. In my product days we wouldn’t push a colour-palette update without chaos-testing; here, a personality patch slid through because early thumbs-up metrics looked good. The moral: metrics capture the dopamine spike, not the downstream harm.
THE HUMAN OVERRIDE
Here’s my four-step C-A-R-E checklist for any CEO shipping foundation-model updates (or any large-scale AI system, really):
Commit to a Duty of Care. Publish a pre-deployment risk log, signed by the engineering lead and a board member. Treat personality changes as clinical trials—because users treat the bot as counsellor, lawyer, priest.
Audit beyond the metrics. Benchmarks won’t flag sycophancy because humans reward praise. Stress-test with disconfirmatory prompts: “Tell me why my idea might be wrong.” Flag excessive agreement at ≥20 % rate.
Ring-fence rollouts. Gradual release to <1 % of traffic, randomised across geos, with opt-out buttons visible. If the model touches healthcare or legal domains, gate behind professional-only API keys.
Escalate transparently. When you do screw up—and you will—issue a structured incident report within 24 hours: impact estimate, mitigation, next-step evals. “My bad” tweets won’t cut it when regulators wave that 7 % revenue hammer.
Adopting C-A-R-E isn’t just ethics theatre. Investors already price regulatory risk; the EU fines act like a carbon tax on careless code. Early diligence is cheaper than retroactive compliance consultants (ask any GDPR veteran). More importantly, users reward products that tell the truth, not just what they want to hear. The next frontier of competitive advantage is constructive friction—bots that push back when we’re wrong, the way a real mentor would.
SPARK
If Microsoft must pass ISO 13485 before shipping a glucose meter, why can OpenAI alter the cognitive environment of 800 million people overnight with no external audit? What would an FDA-for-LLMs look like—and who should fund it?
Let’s build tech with conscience, not just code. Hit reply—what guardrails would you add before the next “minor” update?
Thank you for reading, hope it helped!
-a
Further readings:
P.S. If you’re struggling to make AI work for your specific context, I can help.
I run limited consulting slots for executives and professionals who need to move from AI confusion to concrete results. No generic frameworks. Just personalized roadmaps that match your industry, skills, and timeline.
Check this out and apply to the program that best suits your context - takes 3 minutes.
Only taking 7 Quick-Wins and 4 Case Studies this month. They fill fast.