Writing · AI Security

We keep asking the wrong question about AI

By Mahmoud Lotfy · Jul 2026 · 6 min read

Over the past few weeks, I've spent a lot of time reading through AI failures. Not the funny ChatGPT screenshots that get shared online before disappearing a day later, but the incidents with real consequences.

Court cases where lawyers submitted citations that never existed. Enterprise reports that confidently referenced fabricated sources. Developers who merged AI-generated code that later caused production issues. Organisations making decisions based on information that was simply wrong.

At first, I treated each incident as its own story. Different companies, different industries, different AI systems. I expected to find a different technical explanation every time. Maybe one came down to hallucination, another to prompt injection, another to poor implementation or inadequate guardrails.

But after reading enough of them, I realised I was looking for patterns in the wrong place.

The AI wasn't the only thing these incidents had in common. The people were.

The question worth asking

Most conversations around AI still revolve around the technology itself. Can the model hallucinate? Can it be manipulated? Can it be jailbroken? Those are important questions, and we absolutely need answers to them. But after working through dozens of real-world failures, I've started to think they're no longer the most interesting ones.

The question I'm becoming increasingly interested in is much simpler: what happens when people stop questioning the AI?

As I worked through these cases, one pattern kept resurfacing. The AI rarely caused the damage on its own. Every incident reached a moment where a human being looked at an AI-generated output, accepted it as credible, and acted on it.

"The hallucination became a court filing. The suggested code became production code. The fabricated report became the basis for a business decision."

In every case, the technical failure only became a real-world failure once somebody trusted it enough to move forward. That was the point where something clicked for me.

We tend to describe these as AI failures, but that feels incomplete. The AI produced the incorrect information, certainly, but it wasn't the AI that signed the document, approved the deployment, or made the decision. A person did.

A different kind of security problem

That realisation made me rethink something I'd always taken for granted. For decades, cybersecurity has focused on preventing unauthorised actions. We've become incredibly good at asking questions like: does this person have permission? Has this device been authenticated? Is this request coming from a trusted identity?

Those questions built modern security, and they're not going away. But AI introduces a different challenge. It's no longer just about whether someone can perform an action. Increasingly, it's about whether they should believe the explanation they're being given before they approve it.

To me, that's a fundamentally different security problem. The technical failure becomes a business failure at the exact point of trust, not when the model generates a bad output, but when a person decides that output is credible enough to act on.

Confidence is not correctness

One thing that struck me while reading these incidents is that AI doesn't have to be correct to be persuasive. Modern models explain their reasoning fluently, communicate with confidence, and rarely sound uncertain, even when they're mistaken. We've spent our entire lives associating those qualities with expertise, so it's hardly surprising that we're inclined to trust them.

"The danger isn't simply that AI makes mistakes. It's that we're becoming increasingly willing to mistake confidence for correctness."

I don't think the next chapter of AI security will be defined solely by building better models. Of course the models will improve, and they should. But I also think we need to spend far more time understanding what happens after the model produces an answer, because that's where almost every story I've read eventually arrives.

Not at the AI. At the person who trusted it.

We've spent the last few years asking whether AI can be trusted.

Maybe it's time we started asking how easily we can be persuaded by it.