AI writes insecure code, right?

We’ve certainly heard that while AI can write code, it also tends to create vulnerabilities that are preventable, to create things that shouldn’t be put out in production. It can stick secrets into code without being asked. It can generate buffer overflow vulnerabilities. It can create authentication that is easily bypassed. Honestly, a lot of that is true.

But why is it true, and more importantly, when?

The conventional wisdom says that AI writes insecure code because it doesn’t know how to do better. It doesn’t know, the claim goes, how to write against important but nuanced requirements, including, apparently, not making exploitable stuff.

But is that actually true? AI has been trained on a lot of code, including code with tight security requirements. Surely this isn’t true in general, right?

Right?

Human Baseline

Let’s back up a bit. If you work in software, you have encountered all kinds of programmers. They vary in capability, education, interests, and diligence. They vary in expertise. They vary in age. Importantly, they vary in how well they write tests and how well they understand and implement basic security practices. Those who are good at writing tests that catch things the first time are a fairly rare breed. Even more rare are those with the training and the gumption to write code that is secure against even the most common exploits like SQL injection. It doesn’t have to be this way, and many security professionals that I know bemoan this state of things, but we don’t get to live in the world we want, we get to live in the world we have.

There’s a reason, though, that security, and its more popular cousin, testing, are not universal skills among programmers, and that reason is pressure.

For a software engineer, pressure is a constant. It is part of the background of life. You feel pressure to deal with critical bugs, launch major features, document your code for those that follow, and just generally ship stuff for impact on your performance review. You feel pressure to launch. You feel pressure to land. You feel pressure to sustain your peers.

You rarely feel pressure to focus on security.

If you do feel that pressure, it largely comes from within. Sure, there might be a bug bounty program that you have to react to, or a vulnerability that is keeping ops folks up at night, and those can be external pressures. But unless you produce very high-value, highly targeted software or systems, those external events are somewhat rare. That’s why programmers who understand and make security part of their everyday lives are equally rare: if they don’t create their own security-focused pressure, they don’t feel it at all, and every other bit of noise drowns it out.

This is not a criticism of engineers, it’s a statement of reality. Businesses don’t tend to feel pressure to be secure until it becomes an existential threat, and then it’s firefighting time. Then security gets funded for a while, until things feel comfortable again and the funding starts to disappear in favor of other pressures. It’s the circle of (software) life. It’s the fate of preventionists.

AI Accelerants

So now you bring AI into the mix. This is like hiring a team of interns that are really eager to please, super fast at typing, and have massive memories but not a lot of independent wisdom or judgment. They’ll do what you ask of them, and they’ll respond positively when asked to do it again or adjust some critical thing. It’s amazing.

To put the AI situation in perspective, though, you really have to think of it this way. We have a lot of experience thinking about how humans react from their different perspectives and experience starting points, so it’s important to anthropomorphize AI just a little bit. Team of interns. Massive memory. Exposure to many ideas. Eager to please. Fast typists. Attention of goldfish.

OK, that last one mostly applies to AI (though check out my Engram project – it helps a great deal with the goldfish problem; and yes, a human wrote that, I use emdash in my writing all the time, thank you very much).

So given that context, what happens when you bring this team of interns into an environment where every pressure but security is present most of the time?

They do what you ask them to do. They get stuff ready and they ship it fast. They also don’t do what you don’t ask them to do. They don’t worry about secure code patterns.

But what happens if you do ask them to write secure code, and you explain what you mean? They also do that!

AI security is pretty much only as good as the prompts given to it. But importantly, it is every bit as good as the prompts given to it, as well!

Real Life, Real Experience

I’ve been working (again) on EntroQ, an orchestration system with exactly-once semantics, inter-queue atomic operations, and an extreme focus on fault tolerance. This system is subtle in its complexity. It has strict security requirements on queue access. It has to carefully pass contexts around and not mess up with permissions escalation. It has to maintain atomicity guarantees with all of this. After all, it’s used in government systems. It’s tiny but mighty, and there were a lot of changes on my wishlist that I just couldn’t get to while employed. Layoffs changed that. But so did AI.

I pointed my team of interns, all inexplicably named “Claude”, at the code and started issuing instructions. The rate of feature burn-down was astounding. Truly. I came away more impressed than ever; at some point in the last few months, we crossed a threshold where this became truly useful and capable in a way that actually works for me. Claude found real timing issues (minor, because this had been tested to death already). It found strange code patterns and suggested fixes. And when I asked it to, it gave me rock-solid security advice and helped me design the rest of the system in a way that is defensible and flexible for all kinds of security needs. It did make mistakes along the way, but when I prompted it to, it fixed them easily and we moved on quickly.

AI writing secure code isn’t a dream, it’s the reality right now, today. But, you do need to know what to ask it to look for, and you do need to know how to ask it to do the right thing.

The good news is that you still don’t need everyone on your team to be security-focused. I got an awful lot done on a complicated codebase in a couple of days all by myself, with Claude as my witness. Imagine what can happen if you have even one true expert in secure software engineering on, say, a team of 20. Make sure reviews go through there, give them plenty of tokens, and watch how the whole idea of AI writing only insecure code starts to look like a myth.

We Have Met the Enemy, and They Are Us

Secure code is more a matter of investment than raw capability. It always has been. Find the right security engineers, and you get secure code out of them (for the most part; AI and humans alike make mistakes). That has not changed with the introduction of AI-based productivity enhancements. If anything, it stays exactly the same: keep the right balance of software engineers to security engineers and watch AI accelerate all of them equally.

Sure, AI can write insecure code. So will most of your engineers; it’s kind of rocket science. Really. That’s why you never roll your own cryptography. That’s why you hire specialists. It’s a skill and practice that takes a different path of study than distributed systems, or mobile UI, or any number of other specialties. AI is cool because it can kind of be any specialist. It is also terrible about being every specialist at the same time, but then so are people. Give it a role, give it a skilled human, and watch it do the right thing, over and over again.

It is, after all, only as good as you are.