Everything’s breaking, and yes, IT’S AI’S FAULT

Why are banks, major websites, cloud services, and critical infrastructure suddenly experiencing so many outages? Is it a coordinated cyberattack? AI-powered hackers? Or something else entirely?

In this episode of NEXT with John Koetsier, Josh Koenig, co-founder of Pantheon, a web security operations platform managing more than 700,000 websites, joins John Koetsier to explain why AI may be making the internet less reliable, but not in the way most people expect.

Check it out:

The problem is clear … AI is dramatically accelerating the speed at which hackers can exploit software vulnerabilities. Security teams that once had weeks to deploy patches may now have just hours. The result? Organizations are rushing to update critical systems faster than they’re designed to handle, potentially causing cascading failures.

We discuss the Optimizely supply chain incident, Microsoft’s extended outages, AI-powered cyberattacks, the loss of experienced infrastructure engineers, and whether the next 18 months could bring even more disruption.

Plus, can AI eventually hack anything? And what happens when quantum computing enters the picture?

Transcript: everything’s breaking, and yes, IT’S AI’S FAULT

John Koetsier: Why is everything breaking?

Josh Koenig (Pantheon): In short, it’s because of AI, but not the way you’d expect.

John Koetsier: This is Josh Koenig. He’s the co-founder of Pantheon. They’re a security platform, hosting platform. They manage 700,000 websites, and we’re talking about what the heck is going on in the world right now.

We see banks down. We see the NSA has something down. We see European supermarkets where their fridges are at 100 degrees Celsius all of a sudden, and all these things. Apple’s down. All this stuff is going on.

Josh Koenig (Pantheon): Yeah.

John Koetsier: What is happening? Why are so many critical systems breaking right now, Josh?

Josh Koenig (Pantheon): Well, first of all, there are definitely instances that are coming up, and they’re clear when they are, where people are seeing their systems compromised.

John Koetsier: Mm-hmm.

Josh Koenig (Pantheon): I don’t want to downplay the potential for that, as the number of exploits being reported is 10X what it was 12 months ago.

But from what I can see, my vantage point in the industry, and just observing the way a lot of this downtime and outages and spotty availability are taking shape, it reads a lot more like the side effect of the pace of people patching their systems at a rate that they simply have not set themselves up to be successful at doing.

AI is requiring a lot of organizations to move fast in a non-negotiable way, and they’re moving fast and breaking things, which isn’t great for all of us out here.

John Koetsier: No, it’s not great for all of us out here.

So what I’ve seen, and you tell me if that’s accurate, is that there isn’t a huge, massive coordinated hacking attack right now. It’s not a massive attack on the United States or the Western world or something like that. But there are serious AI threats out there right now.

People are patching their systems as quickly as they can. They’re taking stuff down and fixing it. I mean, Fable came out, what was it, a couple of months ago, and they gave preferential access to some of the big infrastructure players, some of the big tech companies. I guess people didn’t take it too seriously back then?

Josh Koenig (Pantheon): Well, I think part of the challenge is it’s not like these are things that you are able to preemptively defend against, right?

The way cybersecurity works for most companies is a vulnerability is disclosed, then it’s patched. At that point, you can patch your system, but you have to wait for that patch to be available.

And the game has changed with AI because the time it takes for a bad actor to reverse engineer the disclosed vulnerability and exploit it at scale has shrunk dramatically.

The number of sites that are getting exploited out there by currently unknown, what you would call zero-day vulnerabilities, again, it’s not zero, but it’s pretty small. And that’s mostly very professional cybercriminals, nation-state actors.

The things that we’re seeing right now have the feel of stuff that’s coming out as a result of disclosed vulnerabilities, and organizations that used to take a week or a month to patch those, now they’re trying to move on it in hours, if not days.

And that’s making them move their systems and introduce change into their systems at a much faster rate than they’re used to doing, and it’s creating instability across the board.

John Koetsier: And the Optimizely thing, that brought down banks, that brought down CNN, that brought down a bunch of other massive companies, the BBC, a bunch of massive companies. That was supply chain, right? That was supply chain, just like SolarWinds was a few years ago.

Josh Koenig (Pantheon): Yeah, and that is an example of it’s not just the change escalation. That’s somebody found an edge. They found an angle. They found a way in this web of interconnected systems that we all operate to sort of sneak in and put something in there that let them potentially exploit customers.

They got on top of it pretty quick, and it doesn’t seem like the impact was too terrible, but it’s a great example of there’s a lot of surface area to cover, and a lot of things are more connected, or connected in ways that you might not expect.

So you can see failures that cascade across different providers or things that seem to affect multiple things at once that you wouldn’t necessarily think of as being connected.

John Koetsier: Explain supply chain real briefly for people who might not have heard it.

Josh Koenig (Pantheon): Yeah, absolutely.

Pretty much all software right now is built on preexisting components, right? We’re all standing on the shoulders of giants. We all install libraries, we all leverage open-source components and tools on some level. Those are all maintained sort of upstream from us.

And even if you’re in a pretty closed environment, even in the modern Windows dev way, there are lots of external components. But even if you’re in a closed environment, you’re still getting updates from some other vendor. You’re not creating all your software from scratch by yourself.

And supply chain vulnerabilities are when someone doesn’t compromise you, they compromise one of the upstream vendors or providers or projects or libraries that you rely on.

And you do what you think you’re supposed to do and say, “Well, it’s time to update so-and-so package.” And unbeknownst to you, that has introduced some sort of malware or other security vulnerability or risk to your system.

John Koetsier: I will say, Optimizely, in case people don’t know it, is something that helps you optimize your website, run tests, A/B tests. Does this work? Is this better? Does this convert better? Does that convert? All that stuff.

I’m shocked that banks were running Optimizely on their websites. I’m literally shocked because, I mean, you’re running code that you don’t necessarily know what’s all there, but they did it.

I was on a panel at Web Summit in Lisbon just literally a week ago with a CTO, and he’s saying, “You know, I’m not too worried, AI,” this, that, everything.

Then this week, we see that Google came out and said, yep, AI is definitely being used on the fly for hacking, cracking attempts to break into stuff, and we see all of this catastrophe that’s been happening here.

How are the bad guys using AI these days?

Josh Koenig (Pantheon): Well, I think that there are a couple of key ways.

One is you’ve had, since well before the advent of AI, just the internet being a hostile threat space. There are botnets that are constantly trying to compromise other systems. They’re using big, long libraries of known exploits and vulnerabilities, and this is where the people who forget to patch their system or forget that they have a system to patch end up falling prey to that.

The volume of that has just gone up significantly with AI because AI makes it easier for everyone to automate everything, including bad actors.

Recently, though, the big inflection point, as you mentioned, with Fable and this most recent class of models, is they’re better able to analyze code and identify where there are potential vulnerabilities and then craft an exploit.

And this is where you get this new thing happening where a vulnerability is disclosed and it’s being exploited within hours.

John Koetsier: Mm-hmm.

Josh Koenig (Pantheon): Almost certainly happening because, I can’t say in every case, but the frequency at which that is happening is definitely related to AI because there’s just not enough qualified security researchers working for bad actors to be scanning all the patches that come out every day to find those exploits.

But if you can scale that out with no limit because you have AI to do the work for you, now the clock between someone releasing an upstream security patch and that patch being weaponized, it used to be days, if not weeks or months.

And these SLAs within organizations of, “We’ll patch it within seven business days. We’re fine.”

Now it is hours, literally single-digit hours. And so the pressure to get those releases out super fast has gone way, way up.

John Koetsier: I love the seven business days, right? The bad guys don’t work on weekends, so why should we, right?

Josh Koenig (Pantheon): Yeah.

John Koetsier: You started off this podcast by saying that the problem is AI, but not how you think.

I want to run this by you. I’ve seen a couple of things from people saying that they are senior IT people, senior leaders, some who used to work at Microsoft, and say, “Hey, you know what? This Microsoft outage, this Outlook outage that was super long, they laid off a whack load of people, including the people who really understood the tech at a deep level structurally. And now they have people who are using AI to build code and maybe even coding a little bit themselves, but didn’t understand everything.”

Is that accurate?

Josh Koenig (Pantheon): We don’t know for sure, right? But there is absolutely maybe a smell to some of these outages that feels more like a cascading systems failure.

There’s a whole discipline, a sub-discipline of software engineering, called site reliability engineering, or SRE. And that’s a pretty senior discipline. Those people don’t come cheap.

Those are the folks who dedicate their careers to understanding distributed systems and how they interconnect, and what it means when, if a queue over here backs up and it starts to fail jobs over there, what is that then downstream?

It’s complicated.

John Koetsier: Yes.

Josh Koenig (Pantheon): Compared to just writing a simple piece of software, it’s quite complicated.

And if you don’t have sufficient SRE capacity, or if the SREs that architected your system aren’t around, if you’ve had brain drain, then I think you are at significant risk of getting yourself into an outage that’s very hard to climb out of.

And that’s where something that could have been as simple as a short-lived storage issue in some Azure instance snowballed into a day-and-a-half outage.

I will say, though, in fairness, these sorts of things happened before. Famously, I think AWS, not when they were new, this was like 2015 or 2016, they had a nine-hour outage. And it all boiled back to, “Oh, we didn’t actually think about what happened if this particular storage bucket ever overflowed.”

So you do encounter novel failure modes that are new, that even the most experienced people didn’t design for or haven’t encountered yet.

And when you’re operating a system like Outlook that’s really busy all the time, there is a real thing that happens in systems where, when they get broken, you kind of have to turn them off again for a while to be able to safely turn them back on again because you’ve got to handle all that load.

John Koetsier: It’s the old…

Josh Koenig (Pantheon): For sure, but the brain drain thing, there’s a vibe of that for sure.

John Koetsier: That’s the oldest thing in the book. Turn it off, turn it on again, see if it works, right?

Josh Koenig (Pantheon): Yep, that’s right. That’s right.

John Koetsier: Wow. Okay, so where do we go from here?

Is this our new normal? Are we going to reach some state of equilibrium where the good guys sort of get up to speed? Or what do you think?

Josh Koenig (Pantheon): So I have a hypothesis, and it’s really just a hypothesis, but it’s sort of informed by an understanding of what software is and how it works, which is there’s not an unlimited number of vulnerabilities to be uncovered.

There’s this notion that is prevalent in kind of the high-level or maximalist AI thinking that sufficiently advanced AI can hack anything, and that’s…

John Koetsier: Mm-hmm.

Josh Koenig (Pantheon): It’s just not true.

John Koetsier: Mm-hmm.

Josh Koenig (Pantheon): AI is not going to figure out how to break the encryption that is used for HTTPS, because to do so would physically require billions of compute hours.

John Koetsier: Mm-hmm.

Josh Koenig (Pantheon): Even billions. There’s a whole blog post about this.

There are certain things that are just mathematically not really feasible, and AI doesn’t get to change the laws of physics.

So software is deterministic. AI is not, but software, generally speaking, is. It operates in a certain way with reliability.

And so there’s just not this notion of a never-ending well of vulnerabilities to be plumbed.

And so I do think that there will be, I would predict 18 months from now, maybe that’s not long enough, maybe it’s too long, but at some point in that kind of midterm future, you’ll be able to look back and see that the rate of disclosures that were material and significant, there will be a sort of big surge, and then it will decline back down to a more normal state.

And the real question is, can we keep up with it while it’s in that elevated state?

Because teams are getting slammed right now. People are getting burned out. It’s actually a really rough time to be in security.

Can we sustain ourselves through this surge? And then can we evolve the practices so that going forward, we’re writing more secure software from the get-go?

John Koetsier: Hmm.

Josh Koenig (Pantheon): Because there’s a little bit of maybe a counterargument against my hypothesis, which is if we make it so easy for anyone to write software, AI doesn’t inherently write secure software. It just writes software that passes whatever prompt you gave it or test you gave it, right?

If we increase the volume of bespoke software by 10X, we’ll still be seeing a large number of vulnerabilities uncovered.

John Koetsier: The good news about that is that that’s a very diverse target set, and those are all tiny little apps that I run for me and you run for you and everything, and it’s not a bank that’s running a billion customers or something like that.

So that’s at least a positive there.

Josh Koenig (Pantheon): Yeah, yeah.

I mean, security through obscurity is an anti-pattern for a reason, but it is also on some level true. If you are the only person with your app in existence, the chances that… There is no world where someone’s going to read a public disclosure and reverse engineer your app.

They’re going to have to aim at you specifically.

John Koetsier: Yeah.

Josh Koenig (Pantheon): If we create a big enough surface area like that, maybe that does work out.

And again, I also think there are real improvements that are being made with how we responsibly integrate AI into software engineering practices that will get commodified, baked into the tools.

You’re going to see more and more…

John Koetsier: Mm-hmm.

Josh Koenig (Pantheon): The Antigravities or the Codexes of the world, or someone who’s trying to create a little bit of competitive advantage against them, start to invest in features to ship safe code by default.

And we’ll see R&D efforts put towards that, and there are things that can be done to make that more standard.

John Koetsier: I believe that.

And even though there’s tons and tons of legacy code there, we have really good ways with AI assistance to rewrite that to modern code as well, and so we can get that passed by and secured. I think there’s good things there.

The 18-month timeframe that you mentioned, where we’re in for a bit of a rough ride, I guess we just have to buckle up and see how it goes.

I cover quantum too, and there’s some quantum that’s not too far off.

And as you were talking about, you know, it physically can’t break HTTPS or other things like that, I was thinking, “Oh, what about when quantum comes?”

Josh Koenig (Pantheon): Yeah. Again, I still doubt. I would put my skeptic hat on with that.

John Koetsier: Yeah.

Josh Koenig (Pantheon): If it turns out that you can, then you’re going to have quantum encryption on the other side, so then…

John Koetsier: Yes.

Josh Koenig (Pantheon): Another set of patches to…

John Koetsier: Do you?

Josh Koenig (Pantheon): To upgrade your encryption library, right?

But again, I think it’s important to be grounded in understanding that this is not… I don’t think this is going to infinitely increase forever. That doesn’t seem plausible.

I don’t think there’s a world… I don’t think the mindset of sufficiently advanced AI is unstoppable or could hack anything, I think that’s actually a very unhelpful way to talk about the technology.

John Koetsier: Mm-hmm.

Josh Koenig (Pantheon): It pulls people away from what’s real and possible today and puts them into an existential fear state about Skynet.

And you don’t make good decisions when you think you’re in T2.

So I like to promote a more realistic, kind of grounded perspective.

John Koetsier: Josh, I appreciate you jumping on the phone here, into your day, and having this conversation. Thank you so much. Really appreciate it.

Josh Koenig (Pantheon): Yeah. Thanks, John. This was fun.

Subscribe to my Substack