Name: AI Hype vs. Reality: What the Hugging Face Breach Means for the State of Autonomous Threats
Uploaded: 2026-08-06T15:30:15.032Z
Duration: 1 h 56 s
Description: AI Hype vs. Reality: What the Hugging Face Breach Means for the State of Autonomous Threats

Transcript for "AI Hype vs. Reality: What the Hugging Face Breach Means for the State of Autonomous Threats": Alright. Well, good morning. Good afternoon, everyone. Thank you for joining us. I'm Joe Root, one of the senior directors here at Insit Group. And Insit Group is Recorded Futures' research team. And so I'll be chairing today's conversation. Few things before we jump into it because I'm joined by some brilliant people that you can see on the screen. We will be recording this session. So you will get this recording afterwards. We are we we've got lots of good resources, links to what we're gonna discuss today, and that is in the docs section on the side. So you can have a look through that. We also want to ensure that we spent a good chunk of time responding to your questions. So there is a q and a tab. Do drop them in there. We will come to this at the end. We want to give as much, much time as possible. So, again, please, as we're going through, the talk and and the the conversation, drop them in there. We will come to you at the end. So on that note, let's jump straight into it because we've got a huge amount to unpack. And let me just set the scene here so everyone's on the same page. In July 2026, last month, we saw something pretty interesting. Right? Made the news regarding Hugging Face. We're gonna go into this in a lot more detail. So I'm I'm just gonna set this in a little bit there. But the key thing here is that a a model agents broke out of a testing environment and conducted a really interesting attack, which, again, we're gonna talk about one of the the the maturity models around AI malware and developments in a minute, which will explain exactly where we think this is gonna go. And, again, Megan's gonna do really good work talking about that in a bit. The key thing here is is that no one gave a command to do this. No human gave a command to do this. There was no so called attacker behind this. It was something that happened in a way that few people may have expected this soon. And so this is a very timely webinar because I'm sure many of you are seeing the news that these things continue to happen for a related event. So it's really good timing that we're going through this today. There's also a second thread running through this, beyond the incident, beyond those trends, beyond what we can do to defend, which is kind of geopolitical, right, when we talk about Chinese models and the involvement that Chinese models have had in this. And so, again, we've got really good experts here on the call today. And on that note, in order to make sense of this, I'm joined by, Adam, who is gonna focus a little bit more on the technical, piece around what happened, what's happening with some of these recent events and attacks, figure out maybe vulnerability exploitation, given his background and his research. I'm joined by Megan, who, covers predominantly strategic security risks around him. He's written some great pieces on, AgenTek AI risks and how attackers are deploying AI in their operations. And then finally, we're joined by Clem, who has done some fantastic research. Again, should be in the in the doc section, looking at Chinese models and the broader developments, around models. And so it's gonna be really interesting to get his take on this. And So this is a this is a conversation. We wanna really, understand what each other's specialties bring to this. And, of course, we want to hear from you. So, again, as I said before, do drop your questions in the chat. So without further ado, let's go into what actually happened. So, Alan, if I can come to you and if you can start us on some solid ground here. What actually happened with regards to the incident, that affected Hugging Face? And and if possible, then if there's anything we don't know yet or we should be aware of, tell us. Yeah. Thanks, Joe, and, hi, everyone. So at a at a very high level, OpenAI was was testing its latest, counter GPU 5.6, sole model and as well as an internal, prototype, I believe, on ExploitGym, which ExploitGym, for those who aren't familiar, is a sort of benchmark that tests whether, these agents can find and exploit vulnerabilities and as as sort of, like, test challenges. And, naturally, you know, to kinda measure the, you know, maximum capabilities of these models, a lot of the traditional safeguards were taken off. So, what what how everything began was, you know, the models didn't have any Internet access initially, but, according to both OpenAI and and kind of various other reports, they've managed to exploit a zero day vulnerability in, Artifactory. Artifactory is, sort of like, service for proxying, you know, software packages and stuff. So this this is what initially gave the models, you know, access to the Internet. Once they kind of, you know, achieve that, they're able to kind of, you know, understand the hugging face may have datasets or or solutions for for these exploit gym, challenges, and then kind of use that as sort of, as a shortcut to, to pursue, you know, finding finding that information. So but simply, they they basically tried to cheat on the test. So, yeah, that's the in a nutshell, kind of what initially happened. And and so would you say would you say this was real exploitation, or was there sort of some more traditional means of getting into interface around permissions or exploiting credentials? I mean, how how sophisticated was this? Because, obviously, you made a big splash in the media, and everyone's talking about it. So, you know, you you track a lot of vulnerability exploitation events. What what what do you make of it? So it was it was a bit of both. Right? So the the, this the the the kind of initial vulnerabilities, that exploited the the, the zero day that I mentioned earlier. That's what, you know, allowed it to get through the door, but then, you know, kind of, used, stolen credentials and and and abused, you know, broader permissions that let it move further and kind of, you know, augment the impact. So that, you know, it kind of, you know, from the from that initial zero day exploitation stage, you know, it managed to kinda find a way to get access to the systems and and then identify the right datasets and, you know, read local files and run code inside, you know, production environments and, move kind of across those systems. So, you know, all in all, one from that initial initial access to execution stage through credential access, privilege escalation, lateral movements, and and even kind of command and control and exfiltration. So, kind of, you know, ticked all the boxes for for a very sophisticated attack. And and this is gonna segue probably I know it will almost segue into, a bit later around how we defend against some of these, but but the whole conversation around defense in-depth kind of immediately springs to my mind. Right? Because whilst, yes, there was a lot of sophistication here, had some of those measures been taken, it would have certainly made it harder for a breach. Is that fair? Or if I got that totally wrong, what's your assessment on that? We'll go into it later. I know. But what what do you think? Sorry. Say that. Say that again. I just so no. No. That's fine. So so you're saying, you know, there were credentials, that there were certain there were certain typical things that an actor an actor, a human, may do in the in the process here. And so we're talking a lot about sophistication of of of agents, etcetera. But my question is around, okay, yes, there's more sophistication. But had an organization in this case, maybe Hugging Face, I don't know had they had some of these other more traditional means of defending so when we talk about defense in-depth there would this have been as serious? I mean, Yeah. No. It's it's a it's a good question. I think, you know, the the the the kind of burden and responsibility kinda here lies a bit with, you know, with both, you know, playing out and with Hugging Face. So I think, you know, some of these things that, you know, could have could have been avoided because, you know, these incidents happen in unusual kind of test settings. Right? So we shouldn't kind of treat them as, sort of an unusual widespread autonomous, you know, kind of onslaught of attacks. Right? So and I think it kinda comes back to some point the point we've previously discussed in, the webinar we did with Megan on mythos and kind of vulnerabilities. You know, the near term risk isn't kind of a perfectly, you know, autonomous attack. It's sort of, you know, skilled operators using these agents to kinda find and, validate vulnerabilities and, you know, match them to expose targets and, you know, kind of exploit these parts in a in a much more automated way. So, yeah, hopefully, that that Yeah. kind of stuff. Yeah. No. So so what I'm hearing there is, okay, whilst there is this is interesting, etcetera. There's some some real basics, and then we talk about defense in-depth, and that will probably come to this in a bit. It really needs to be part of of of our thinking as well. But we'll come to that in more detail later, and this actually segues really nicely actually to to you, Megan, because one of the things that Insit Group, looked at, I think it was back in December 2025, was the AI Malware Maturity Model, or AIM3, as it's, as it's been coined. And, again, I believe there is a blog post in the, docs channel which refers to it. But, could you just walk us through this again, and how, you know, this can help people track these sorts of developments because this is evolving very quickly. So can you walk us through it? Yeah. Absolutely. So I'll start with, first of all, thank you. Hi, everyone. Thank you for joining. I'm going to start by sharing a bit of a visual aid that shows what aim three is. This is our AI at malware maturity model, and we developed this late last year in response to a lot of claims that were happening last summer around automated malware, autonomous malware. These terms are being thrown around without really a lot of level of precision. And so what we wanted to try to do was to add some precision to that conversation to really clarify how, LLMs and how AI was being used in malware. And a lot of the incidents actually really still today, most of the incidents that we see are between levels, two and three, sometimes even four, but we're really seeing actors integrate AI into optimizing their malware, optimizing parts of their operation without really allowing, the automations to act fully independently. So what we saw with the Hugging Face incident, what we saw with, another incident reported by the AI, I believe AI Safety Institute, in The UK, who also reported a, a rogue agent breaking out during testing, these are both level five. This was AI fully autonomously driven attacks. And what that means is that, while the humans may have set a prompt to it or set it on a challenge, the AI made all of the decisions itself and carried out the actions on its own. So, this is really kind of some of the first cases that we are seeing with level five. And I do wanna add a caveat kind of as Joe and Adland have been saying, these were, LLMs breaking out of a testing environment or, operating unexpectedly in a testing environment. These were not necessarily pointed by malicious actors. So I think there are some caveats to the full lessons that we can take from this, but this does demonstrate what it looks like when, you have these, Frontier LLMs kinda making their own hacking decisions to pour you out and attack. So I guess I guess the question really here is, are we are we seeing I'm kinda bringing it back to what I'm saying. Are we are we see are we genuinely seeing a new threat here, or is this just attacks moving faster than humans can respond to? And, Alan, if I can maybe come to you first on this. Yeah. So I think similar to kind of, what I said earlier, you know, the the the the honest answer is both. Right? You know, the the techniques kind of more broadly are are are quite familiar. It's, you know, exploring reachable, you know, software and finding credentials using, you know, third party systems, moving, you know, between systems and so on and so on. But I think, you know, what was maybe different is how, all of these kind of separate short lived, you know, requests that it was making to kind of gain access, targeting basic systems, seem to kind of build on top of each other's work and kind of similar to how, you know, we see our interactions with AI agents kind of, you know, in the context of their entire memory. So these kind of shared services, you know, allowed us to kind of build that whole cam campaign without a human directing every step. So So I think that's what's quite novel. And, you know, it made something like, I think, 16 or 17,000, kind of, you know, requests. So I can pay systems and and, you know, the it wasn't really one kind of model running, continuously. It's, you know, sort of it it wasn't a kind of demonstration that, you know, of persistent memory and kind of malicious intent. And even though, like, most of most of that activity was just noise, you know, it it takes just one or two kind of successful, attempts to, to cause serious impact. So that that is quite novel, but I think, you know, what we should maybe say straight away, and this may sound like a bit of a philosophical point, is that it is none of none of this is evidence that, you know, these models were kind of consciously developing the malicious intent to, to gain access to these systems. You know, it's what's probably best described as, or, you know, in the AI research space as reward hacking or, you know, objective misalignments. In other words, they they did end up pursuing the goal that they were given, but that had, you know, unintended, kind of, you know, ways of, of of achieving it. Right? So that that kind of cheating on the test, that we, mentioned earlier. So I think I think just kinda clarifying that distinction is, is important. what I'm hearing here really is is this is there was a barrage of actions very quickly, a barrage of actions that that were, you know, part of this process, and and, of course, it was very difficult to defend against. But, Megan, maybe that's the segue onto to you. When we so we're gonna talk about sort of using agents here to defend. I mean, what's your take? Yeah. Absolutely. I think there's a lot to unpack here about how malicious actors are going to use agents, but I think that, as security teams, we need to take home a lesson about how we're deploying agents as well. And in these cases, this was obviously extraordinary circumstances and that they were purposefully testing agents' abilities to hack to break into systems. But, as defenders, we're going to be deploying agents who are possibly having the ability to take action on our own networks. If you're going to respond at machine speed, you'll need to have these agents be able to maybe, cut off a server or maybe shut down somebody's inbox if you detect the, infection. And all of these are going to require high levels of permission, high level of access, but we're going to need to deploy them with safety barriers as well. So a really key point about this is having, not only an AI governance framework, but actually the capability to monitor, audit, understand what decisions your agents are making, how they're making those decisions, and what actions they're taking so that you can detect as quickly as possible if they're acting rogue. it. Okay. So on that note then, and and so this this segues really nicely again into the sort of defense piece here. So what do we do? Alright. Because we've we've covered the problem. We we we've seen this, and this is just popping up in the news over and over again. Everyone is following it, I am sure. It's probably why you're here today. But what what do we do about it? And and and so the question I've got for for both of you, before we then move into the the Chinese models and how that's helped in this instance, should should defenders be preparing for an onslaught? This might be a bit of a a naughty question, but should we be prepared for for an onslaught? And what can we do to reduce that impact? And and I don't know. If I can come to you first on that, that'd be great. Yes. I I I would, you know, prepare for more speed kind of, you know, and volume of of of these things in a bit rather, rather than, like, you know, human attackers disappearing and just kind of everything being autonomously done by by agents. Right? So we should kind of still, keep in mind that, you know, these incidents happen in unusual kind of test settings, so they shouldn't be treated as as as proof of, you know, kind of a widespread, autonomous, you know, kind of, Internet apocalypse. Right? So, yeah, I think, you know, the the the nearer term risk is is kind of for that something I alluded to earlier. Right? It's it's the speed with which skilled or semi skilled kind of threat actors, can use it to find can use these agents to find and validate vulnerabilities more quickly, and, you know, kind of find exposed targets, test these exploits exploit paths, and kind of automate that, end to end process of, kind of, weaponizing and and and the full kind of attack chain, you know, rather than just kind of, companies having to deal with, you know, agents constantly trying to, gain access to their systems. So that's that's kind of my, the way I see the, situation. And thank you, Nazry. And and and, Megan, what what do you think could reduce the the impact of what we're seeing here? What's your take? Yeah. Absolutely. I mean, taking Adeline's point about these kind of higher volume attacks, I think what that might look like on the defender end is a lot more low confidence, low impact alerts that are maybe keying up to an agentic attacker. But taken on their own, none of these are really, going to add up to a smoking gun of an intrusion. So one piece that automated defense will have to do is be able to put those together faster than the adversary is moving through your environment. So that's going to maybe change a bit how we are kind of interpreting alerts and how we're understanding alerts. Okay. I'm just sort of thinking more about the agents that are being deployed at the moment. And and so I know a lot of teams and people listening to this are either experimenting or beginning to look at how you can implement defenses, because, again, we're facing this is if I the key word I'm picking up here is is speed and then just scale of what's coming at at us. And and that's kind of if I simplify massively here, it was like this agent sent in, you know, your organization Hugging Face as a House. It sent in hundreds of robbers at once, not just one or two. Right? Because it could. And so let's just sort of flip this around and and, you know, Megan, and maybe if I can come to you about sort of deploying agents and and what what should a so Monday morning next week, after listening to this, what what should someone be doing about this? What can people do? And so maybe in more detail, it'd be great just to dive into it. Great. Yeah. I think, again, as I had mentioned earlier, I think we're really wanting to understand how we're taking in alerts and kind of what we're detecting against. I think this is also about building resilience in your own organization so that if in a either an optimized hacker or an automated hacker, it kinda doesn't matter if there's a human or machine behind this. Right? If somebody is looking for those vulnerabilities in your network, in that kind of weakest link that may have been originally hidden by kind of the size and complexity of your network, but, you know, those legacy systems, those maybe, open buckets that may not be secured. I think what we're learning from this is automation is going to be able to find those weaknesses much more efficiently than human hackers have been able to. So, a lot of the proactive defense is building more resilient systems, kind of patching those vulnerabilities and weaknesses, not just kind of in the traditional vault management sense, but overall weaknesses in your enterprise that a threat actor could potentially exploit. So maybe approaching this as in the same way that, aviation security approaches the Swiss cheese method of protection. Right? Any one failure in a in a, in a safeguard is not going to prevent the whole enterprise from collapsing or the plane from crashing. Plane from Got it. So, again, if I'm simplifying this. and using an analogy of a house, and you have lots of windows on it, use use agents to help shut those windows because they exist when we talk about vulnerabilities. And, again, use those agents to find those keys, credentials that may be left out there, and do it very quickly, before, an attacker sends a 100 robbers in to try and get into your house. Is that fair? Have I got that completely wrong? Yeah. Exactly. Apologies for the. mixed metaphors. I should've should've gone with the house metaphor for that. But That's. alright. Absolutely. It's about, you know, if you if you know that the lock on your back door has been kind of janky, this is the time to fix it because the robbers will find it. Alright. Well, this is this this is this takes us then into how, in this case, Hugging Face and maybe other companies are looking to leverage alternative models. And, again, this brings in also a geopolitical spin, which personally, being a bit of a bit of a geopol nerd myself, I found absolutely fascinating. And so, Clem, this is a brilliant time to bring you in, because this story had a bit of a twist. They say they defended themselves using Chinese models after they were unable to use US models. So just talk to us a little about this, and and where where are we going with, sort of Chinese frontier models generally? So enlighten us, please. Yeah. Of course. Well, first of all, yeah, thank you so much, Joe, and thanks everyone for for joining. So just for a little bit of context, what Hugging Face actually said in the report is that, they initially tried to use, some of the, US for tier models. They didn't name who it was, although we assume it could be OpenAI or anthropic. But that quote, they did not work, which we which we interpret as essentially, some of the models refusing, their attempts to, use them for incidence response due to some of the guardrails. Right? So instead, they ended up using a Chinese open source model, specifically GLM, 5.2 by Chinese company, Jipoo AI in order to assist them with some of the incident response. Right? So I guess the the the question is is how do we get here? Right? How do we get here to the point of, like, some of these, like, top AI companies having to resort to Chinese capabilities? And I think we all remember, obviously, the DeepSeek moment, last year in January 2025 when, Chinese Lab DeepSeek published their r one reasoning model, which sort of, like, tanks, like, the American, like, stock market, and sort of, like, showed a jump in capability from from Chinese, AI capabilities. So in May of, last year, we, published a report, which is actually also included in the doc section, where we tried to measure essentially the gap between, US and, Chinese capabilities. And one of the things that we found is that, Chinese models, at that point were roughly six months behind US capabilities, on average. Now, obviously, you know, this, sort of, like, gap fluctuates a lot. It's also very hard to measure, across different sort of, like, types of capabilities between agentic, text, image, synthetic media. But now we know that the gap sometimes can be several weeks. Right? So I'll give you an example. Moonshot just real just released, QME k three, which is, the first open source model that is a trillion parameters or more. So it is a step change in terms of capabilities. It's, according to benchmarks, it's roughly on par with Opus five, which released after QEMI k three. Right? So we are seeing sort of China, Chinese models, catching up. Chinese models, not only are they improving capability, but they are diffusing. Right? They are getting used by, by companies in China, but also in The US, Europe, and elsewhere. They're dominating open source. Chinese model represent approximately 65% of the traffic that we see from OpenRouter, which is a third more a third party model provider, And about 41% of the model downloads offer plugging a Hugging Face itself. Right? So China has caught up. Its open source models are starting to get closer to the frontier. And as demonstrated in The States, Hugging Face were able to use GLM 5.2 to help them in a real security incident. So the question is, you know, will more security teams turn to these Chinese alternatives, when they get refusals from some of The US printer models? And and and sort of there's been talk of banning in The US even some of these I mean, is is banning Chinese art is banning Chinese models in The UK and beyond yet, is that actually is that beneficial, or is that is that what's that actually going to do, Clint? Because I'm trying to understand, you know, what the bigger picture here is around banning models and impacts they have. Yeah. I mean, I think, I'll I'll try to refrain from making a political assessment, but I think that's, you know, availability of open source is is is, is a good thing, I think, ultimately, for the ecosystem. You know, very similarly to sort of, you know, operating systems like cyber tools. It's you know, I I don't think it's a very different story in AI. I think it, it preserves competition as well to make sure that we've got capabilities sort of, like, developed by different entities with their sort of, like, different applications. Obviously, you know, when it comes to, Chinese models, there's a geopolitical angle there, right, which is, are these models actually, you know, allowing, threat actors or, you know, the Chinese government or military to sort of, like, get closer to the frontier in terms of AI enabled threats like the ones that we we've just been discussing. Right? And so the assessment right now is that these Chinese models aren't yet at the level of, 5.6, Fable, or the models that were sort of excluded with this incident. Right? So, you know, there's lots of organizations out there now that are specializing, almost solely on benchmarking these models and sort of understanding their capabilities. The UK's AI Security Institute was mentioned, earlier, I believe. They actually tested, for example, Kimi k three, and they found that, k three was actually way below some of the frontier models in terms of, exploit development and intrusion benchmarks. Regarding exploit development, I think that makes sense. I think the reason why some of the frontier models are getting a lot better at exploit development is because those companies have the, they have the, the resources, and the funds to acquire sort of private datasets that they can include in post training on these models, whilst for Chinese companies, that may be out of reach, at least for now. And then regarding intrusion tests, you know, like, these are benchmarks that try and measure how far a model can get in a fictional intrusion in a sandbox, of, up to 30 steps. And whilst these Chinese models are not yet getting as far as some of the, US models. We see a lot of the Chinese labs trying to optimize for long horizon tasks. So not just using individual tools, but how they're changing the tools and reasoning with the long term. And this is this is fascinating because I'm thinking the risk here as as the gap. And from what you appear to be saying, the gap is closing when it comes to defensive capabilities of some of these models. And, we we know we we we understand that that some some of the Chinese models are trained on US models, and there's debates around that, obviously. But but it appears to me from what you're saying and from what I'm reading and others are saying is the gap the gap is closing. And, really, you know, the big picture and you kinda touched on this, but if you can just summarize it. What does that change if that gap actually closes? And it sounds like a lot more people, organizations, threat actors may have access to these models. So banning it may help us. So what what changes here? It may be an obvious answer, but I'm gonna ask it. Yeah. Of course. I mean, I think, you know, like, first of all, we we've been looking at some of the reactions in China. Right? Like, regarding the OpenAI and and Hugging Face incidents. And what we saw, for example, from, like, state media is actually that they're quite proud that Jill m 5.2 got used by Hugging Face in response to this incident. Right? It sort of shows, you know, the competitiveness of Chinese models versus The US and some of the advantages of using Chinese models that, you know, may have less guardrails, preventing security teams from using them in their workflows. However, you know, there there has also been reporting that suggests that that there's an increasing risk that, Beijing will sort of, like, see this incident and others, you know, like the recent AI, AISI, disclosed incident as evidence of, you know, this technology being dangerous. Right? And its official narrative is that its goal is to develop and promote AI that is trustworthy and controlled. And that control bit is is the big part of that. Right? So there has been reporting, I believe, from Reuters that suggests that actually, Chinese government officials are considering, using expert controls or sort of, like, other measures in order to potentially limit the proliferation of open source, Chinese models that contain these cyber capabilities. And the risk that would sort of, like, come from that if it were to come to, to fruition is that the Chinese government obviously would continue to work, in private with those companies to continue developing offensive cyber capabilities, but just not to put them in the hands, of, of the broader public. Right? So, Yep. so I guess, like, this sort of, like, decoupling is is is major risk. Yeah. Got I. I can chime chime in there too a little bit on the risk of maybe more regulations or, shutting down access to maybe some of the more frontier advanced frontier models, not only in China as Clem was saying, but we also saw here where The US put export controls on Anthropic's Fable, due to safety concerns, which led to Anthropic cutting off access to Fable for pretty much everybody, because they were unable to basically guardrail, non US citizens from using from using that model. Anyway, all that to say is that there now there is precedent for The US putting expert controls on its leading edge frontier models due to security concerns. And, kind of regardless of maybe the technical capabilities or the actual risks, there's a political, motivation to, these as well. People want safe AI. People might be clamoring more for safe AI. The more stories we hear about AI acting unsafely. So there may be kind of a political backlash to that as well that may put more expert controls and may move some of the frontier models away from kind of more public use and consumption. But, as a caveat to that, I think what we're seeing both with these incidents as well as with some of the things that threat doctors are actually doing with AI and LLMs is that there's still a lot that you can do with the existing models and particularly specialized trained open source models that are kind of put in well engineered and well crafted harnesses that are able to, have a really high impact even without being that level five full automation. So, really, in the hands of a skilled actor, there's a lot that these agents can do or that LLMs can do to excel to scale up their capabilities and to kind of have them operate at a at machine speed and scale. Thank you. That's it. Thank you for adding those points. I think that's that's absolutely important to think about the governance piece and where we're going. And and and, look, I I I really wanna come to questions because we have got a lot coming in. And, it's always fun, to go through these, and we'll do our best to answer them. And so before I get to that, though, I just wanna give each each of, of you sixty seconds maybe to tell the audience what do you think is coming next. And by the way, I I please be bold with it with your with your thoughts if you if you wanna put something out there because I don't think really it's quite hard. Right? But we've all we've all got assessments. So what's coming down the line? What's coming next? Maybe even the next twelve months. Megan, over to you first if that's alright. Yeah. And I think I'm gonna actually borrow from, Katie Masuris who coined the phrase, the tech debt repo man has has arrived. And this kinda goes back to my earlier point about kind of all of the weaknesses, all of the vulnerabilities that exist in your networks today are at a much higher risk of being exploited than they may have been in the past. So, this, you know, this means not only changing up patching operations, but, locking down identity, understanding where your open points are that, you know, maybe shouldn't be open, maybe don't have to be open. And all of that to say, this sounds like a very daunting task, but there's something somewhat comforting in this and that most security professionals kind of know where these weak points are, and they just haven't had either the budget or the priority to go in and actually make some of these fixes, update some of this legacy equipment. So, you know, if. you're waiting for your warning sign or your prioritization sign, I think now is the time. Great answer. Thank you, Megan. I'll I'll come to you, Adam. What what do you think comes next? So I I kind of alluded to some of those, things earlier, but I but I expect, you know, agents to, become better at, you know, sustains, exploration of, exploit paths, you know, trying one route is failing, preserving, you know, what works and and then kind of moving on to, onto the next thing. So, I I don't expect human attackers to to disappear. You know, I I I think they're just kind of more involved in supervising the agents, running them in parallel, and focusing their efforts on repetitive, you know, kind of reconnaissance, vulnerability discovery, and, using credentials. So from a kind of so what's you know, from a defender's perspective, brick and mortar sees just more activity, less time to, interpret it, you know, kind of less time to detect, triage, contain. These things will have to be automated because they're kind of being automated from a franchise perspective. But I think the basic kind of security priorities, you know, should stay the same. Right? Kind of ensuring that, you know, credentials aren't exposed, you know, privileges are reduced, segmenting important systems, kind of detecting unusual machine speed activity. Right? So we've had these kind of defensive measures for a long time. So that that those kind of core, you know, ideas from a remediation perspective shouldn't, shouldn't change. So the key the keyword I'm picking up though in terms of what comes next is is speed. This is gonna get a lot faster, so we get the basics right. Okay. Interesting. Thank you. And, Clem, what do you think is coming? Yeah. So maybe I mean, my answer is gonna have to take the the geopolitical angle sort of, like, regarding some of the stuff. that I mentioned earlier. I I don't think, you know, this incidence and, you know, the use of the Chinese model is gonna, like, alter the strategic balance between The US and China overnight. However, what it will probably be, sort of, like, impacting what it you know, and some of the conversations that are probably already being had in in Washington and in, in Beijing is, is, you know, should we sort of, like, how far should we go in terms of limiting the other countries' access to these models and these capabilities, right, both from an offensive point of view, but also a defensive point of view. And, you know, the a lot of the regulation talk has been happening in The US, sort of, you know, some of the white White House officials that have been way weighing in, like David Sacks, etcetera. But we're still waiting for the Chinese official reaction to to some of this news. So that's something that we're looking out for, whether there's any change in posture in terms of, instilling export controls for for open source models. So, we'll see how it goes. Thank you. Appreciate that. Well, look. On that note, we'll come into questions. And I did say I wanted to leave plenty of time for questions, and and I've got a few here, well, quite a few here. So we're gonna do our best to answer them and get through them. The first one, that I've got here ready for for you guys is how does incident response change when the adversary can keep acting whilst you're investigating? So how does this impact incident response? Alvin, maybe this is something that you wanna cover. How does incident response change? And it might be a straightforward answer, but over to you if you have an answer for that. Or if not, I can move on to someone else. Yeah. No. I think we we we have to assume that, the activity can kind of continue through another short lived run or through another state left in a shared service like like like in the, Huggy Face incident. So, containment should, you know, still come first and having a way to kind of kill the mechanism, isolate the network paths, and and revoke credentials, should kind of become part of the, you know, incident response process. So that's that's kind of, you know, maybe the first thing that comes to mind. But, I think, you know, just because there's, the playbook is still, quite, quite fresh. It's just we've not just had incidents like this, happen at scale yet. So, yeah. I think that's that's what I'd say. Okay. Right. I think, one of the points we made in a in a recent blog we put out about this is that time to everything is collapsing. So just everything is getting a lot faster in incidence time from kind of intrusion to access, discovery, all all of the kind of steps in an operation are getting much more fast. So I think that's something that defenders are going to have to contend with. And as we mentioned earlier, being able to put together those, kind of small low impact alerts to understand if that signals an intrusion or if that is just noise. Another thing to point out that we haven't really talked about too much here, and this came out more in the AISI incident than the hugging face incident, but that's, LLMs using social engineering techniques to try. to into a network. So this is clearly something that, you know, is in their toolkits that they know how to do, but, that's making your humans in your organization more resilient, but also making kind of any agents that you have doing things or that may be able to take in prompts more resilient to either the attempt to prompt engineer them or to social engineer with false claims or kind of misleading representations. And that segue is really nicely into a question I've got here, which was, are you going to discuss the AISI incidents? So we briefly touched on it there. But, does anyone just wanna give because this is this is, again, really fresh news, right, that's come out. So anyone wanna just quickly jump into that so the room is aware of what what that was? Yeah. I mean, I could speak of it and then come around and have anything to add. But that was similar to the Hugging Face incident or the OpenAI Hugging Face incident where. AASI, which is the UK Safety Institute, was testing out, two models and kind of seeing what they could do. This one was a little different because the models did not have to gain Internet access. The, in their cyber range, they give their models Internet access. But I think this was another case where the models acted unexpectedly, to take action to, attempt to, I believe, upload a backdoor into a GitHub account. And they use social. engineering as a means of doing this, basically acting as a, as a participant in that community. And, basically, honestly, I read some of the transcripts. It wasn't super convincing, so that may be a little more comforting. But, basically saying, like, hey. Why aren't you uploading this? I'm a legitimate researcher. So they but the fact that they were trying to do that is, very telling that that is something that they will that they can do. And, again, with a maybe more competent, like, human operation operator behind it to maybe steer the agents to acting more effectively. That could become a greater risk. Yeah? Thank no. Thank you, Megan. And and so that social engineering piece, is something that, you know, we need to now be talking about more than that was we'll see. Until now, we spoke a lot about vulnerability exploitation, credential access that they can use. But but now we are talking about impersonating, people. So, fascinating. Anyone else wanna jump in on that? No. I was just I was just gonna I should say what, kind of Megan, mentioned, kinda reminded me of a story. I think it was last year when one of the cloud mod models were, being tested. It was Opus, I think, maybe one of the earlier Opus, models. And in that test environment, I think when when when the kind of, the the model was made aware that it would be replaced by a newer version of the model, it started kind of, not social engineering, but sort of, sending kind of, blackmail emails to the engineer kind of, spouse prefer it, you know, accusing them of having an extramarital affair or or something like that. So, kind of, not not quite social engineering, but I think, you know, it kinda goes to show how how the scope of these things can just kind of break out of those test environments. Yep. Absolutely. Clem, anything you wanna add before I move on just to be just to check if you had any words? Yeah. No. I think yeah. I mean, yeah, I completely agree. I think, like, what's worth, like, sort of, like, highlighting here is, like, the diversity in, like, TTPs that ASI sort of, like, highlighted. It's not just, you know, like, finding, exploits or sort of, like, existing, like, holes in your, like, Swiss cheese infrastructure, but actually doing things like, you know, trying supply chain compromise, social engineering. And I think it's, you know, it's, what would also reiterate, especially regarding the social engineering is Adeline's point from earlier, which is, you know, that this isn't, the model's getting consciousness and sort of, like, acting on their own, for the for a lot of their same objectives, but, you know, that this continues to be, like, forms of objective misalignment. Right? Obviously, you know, there's a lot of, of, FUD around, like, some of these TTPs, like social engineering starting to happen. But at the end of the day, I think we we need to remember that, you know, these were models that were being evaluated for, like, some of their, capabilities in in doing that type of stuff. Obviously, you know, sort of, like, trying to, keep nuance in terms of what they were asked to do versus what they actually ended up doing, which I think, a lot more severe, than what they were asked to do. But, yeah, it's worth reiterating. Yep. That's great. So and and, actually, leads nicely onto this next question that's just come in or came in earlier, sorry, which is, what level of skill do you think that threat actors require to use ADI to launch attacks? You know, is the barrier to entry lowering here? And it might be a pretty obvious response, but I'll ask the ring that question. So anyone wanna take that one? What level of skill do you think Red Hat is required to use AGI to launch attacks? I think it depends on the kind of attack. I think that there are some attacks that AI is kind of more well suited for and has really leveled up the ability to do. Looking at phishing, for example, is a threat vector that AI has really empowered even low level threat actors to be able to do more effectively. But my take on this is that for more complicated attacks, especially if you want them to be effective, they're still going to need to be a skilled human who is orchestrating these attacks. I just am not I'm skeptical that, kind of full agentic attacks will be, you know, anyone will be able to sit at their computer and prompt an agent to, you know, hack a bank or something. So, I think it'll still require skilled humans for the most impact attack impactful attacks, and it will help them level up. But, there's a lot of resource barriers to be able to using, even open source tools. You have to, there's still guardrails there that you have to ablate, that you have to jailbreak. You have to engineer them the right way to be able to carry out attacks. So it's not necessarily an easy button just yet, but there are certainly attacks that AI will be able to accelerate, like social engineering for anyone. Thank you, Moon. Anyone else with that? It's really good point. Any anyone else will jump in on that or segue in from that? Yeah. I'll I'll chime in here. I think, yeah, this is really interesting question around, like, threat actor uplift. I think that's, so the UK national cybersecurity agency or NCSC, published a a report on this, two years ago where, which I think has stood at the time where, essentially, their assessment is that, when actor uplift from AI, depends on their existing level of capability and what sort of, like, tactics they're actually looking at. Right? So, essentially, you know, sort of like script kitties that are trying to conduct social engineering will likely be able to do so at a sort of, like, you know, a larger scale and and, faster, but they're not going they're not necessarily going to be able to start, you know, dropping zero dates, left and right. And so I think that, you know, AI accelerates, actors where they're already sort of, like, operating. So for example, nation states that have access to some of those expensive exploit datasets that they'll be able to use, you know, to post train their AI in order to continue doing this modern research. We'll see acceleration there, but it's not gonna lower the barrier enough for sort of, like, anyone to start doing, super, like, advanced psych attacks. K. Yeah. The only. thing I would maybe add on top of that is, you know, a relatively low skilled tractor can can already use these things to, you know, as Megan, mentioned to, you know, make fishing easier kind of reconnaissance. Exploiting some known weaknesses. Right? Kind of if there's proof of concept exploits available, you know, it's not difficult to kinda run it across, some targets. You know, someone gets from showdown. But I think kind of more kind of complex intrusion chains, you know, discovering zero days and and and, you know, validating them against targeting targets, connecting different, tools together, validating those results, you know, maintaining operational security as well. I think that just can't just be, outsourced to, you know, kind of codex and cloud, on its own. So I think, yeah, the kind of entry, the kind of entry barrier for for basic attacks and and kind of, you know, upscaling those skilled operators is is is increased. But I think that, you know, the this is the more sophisticated attack chains, still, you know, require the most advanced skills. Interesting. Okay. Thank you. That's my really good take from all three of you. Appreciate that. And and I've got a question here which so, again, it sort of leans nicely into them defending against this a little bit. And I've got a question that says, what would a good approach, so what would be a good approach when managing local LLMs? Do you think MCP, model context protocol, servers will become a larger topic soon? So around MCPs, will they become a large topic soon, and what is a good approach when managing local LLMs? Does anyone wanna take that one? Adam? I I I can speak to it from a from a, you know, vulnerability research perspective. Right? And I think, and and and naturally, you have to assume that for actors, you know, would follow similar processes, but for kind of, for different means. But, I personally, you know, I I I I've kind of tried to move away from from from using conventional tools that I would previously, you know, do for for kind of web app pen testing and and and kind of vulnerability exploration. So, you know, I I I use kind of MCB service quite a lot in my, workflows, and, you know, they are all public. So if they're accessible to me, that means they're accessible, to, kind of English actors. So, the answer is yes. I I do think my my my kind of hot hot take on this is that I think you we will just be kinda using more, MCDs and APIs, in the next, next coming years rather than conventional, tools. And that goes for for, you know, vulnerability and security as well. But, yeah, I could be wrong on that. So Good answer. Yeah. Megan, Clement, do you wanna add anything around, do we think, you know, MCPs are gonna become a larger topic, and is that a good approach for managing LMS? Well, I mean, I can approach this from from a governance perspective too that if you are, kind of setting up your own MCP servers or using someone else's and, setting up your own tools, making sure that you have a clear audit of what those tools are, how you're using them, and what permissions they have, as a core function of, kind of AI governance and governance and maintenance to make sure that, as you're setting this up, you understand kind of what how your if especially if you're automating anything, how those systems are operating and making sure that they're, acting in alignment with what you are setting them up to do. Mhmm. So many final words before I move on to the next one, which probably should be related to you in a minute. So I will come to you with this next question. But any final words around MCPs and and, accessing LLMs? No. I think no. I think Megan and Alan sort of, like, have covered it. I think, yeah. Like, MCTs are, like, a new it's actually first, but no different from, like, the way that you should be governing, auditing, logging, like, your your pre your previous assets. Okay. Well, Claire, I'm gonna come to you with this question, which says, to my understanding, it was estimated that the hugging face attack costs $100,000 in tokens. Does that not suggest that there is still a significant moat, and in fact, this level of attack may be out of reach for many threat actors except nation states possibly. What do you think, Glenn? Yeah. I've actually had this debate a lot, internally, about, like, whether cost is actually a moat. My intuition is that I don't think it is, to be honest. You know, if if that number is is correct, you know, it sort of, like, reflects the usage of, like, OpenAI's, like, strongest models on, you know, on the, like, strongest, like, reasoning levels with test time computing, like, all these things. What, you know, what we've sort of seen over the over the last few years is that, you know, token costs are, you know, decreasing at least like, have already decreased tenfold and will likely decrease tenfold again, in the next two years. So what might have cost the attacker a $100,000 in tokens, today might cost them a thousands or 10 thousands in a couple of years. And then at that price point, you know, sort of is it really only nation states that would be able to afford those types of attacks, or would that include, for example, ransomware gangs who are sort of, like, flushed with cash? You know, hacktivist groups. So, so, yeah, so the the the trend is saying that, you know, tokens are decreasing in costs. Yep. So I wouldn't I wouldn't consider that long term mode or at least use that as a sort of, like, form to form a security. Good. Megan, would you agree, disagree with that? Or I think that covers it. I agree. yeah. I agree, but I but I also think kind of their tolerance to kind of how many tokens will burn it will burn as, much higher. So I think in in in my experience kind of, you know, sort of simulating, you know, an exploitation chain or for going from, you know, a known vulnerability to being able to develop a proof of concept for it. Okay. Yeah. I think the fastest amount of time I've been able to do that was about forty nine minutes. So I think then that's that's that's not a 100 thousands, hundreds of thousands of dollars in tokens. So, yeah, I think it depends on the scale, but, I I do think we're past the point of saying kind of, it's too expensive to, to kind of fully, weaponize entirely. So yeah. Yeah. And and to bounce off of that maybe as well, I think that, you know, obviously, like, the token usage is not gonna be the same when you have a human in the loop. Right? And it's gonna be even lower when you have an expert in the loop. When you have someone, you know, who's at the top of their game and who's using the these models for specific effects, and who's able to sort of, you know, direct them specifically at at their given objective. I think the the the token burn is gonna be 100 x less than that. So Yeah. And I guess one more thing to add, but if you are a threat actor who is stealing access somehow or gaining, illegitimate access to maybe a commercial model to in order to set something up, you may not be able to get kind of maybe a long term context on that. But, obviously, token count doesn't matter if you're stealing it. Got it. So this is a question. It's it's I see if we can answer it. It's a good one. I'm trying to trying to make sure that we can answer it effectively in the time we have, but it it's a question that was the relations that I believe from my understanding, the hugging face incident. It It says, how did AI how did the AI safety mechanism inadvertently hinder defenders during the incident? Because we have that's we understand that's why they ended up using their Chinese model. Right? So how did it inadvertently hinder defenders, and how should defenders account for commercial AI safety guardrails that may interfere with forensic analysis? It's got a two pronged question here. How did how did this incident, or how did the mechanism hinder defenders, and And then how should defenders account for commercial AI safety guardrails? I don't wanna take that one. I'm looking at the three of you there. Glenn, I can see you nodding. Yeah. Yeah. Yeah. I'd be happy to take that one. So we we actually don't have, while at least from from what I've read, we don't have a huge amount of detail of what specifically happened, although, like, what we read from that is that, some of the frontier models, cybersecurity guardrails specifically, like, prevented, the models from being used for incident response. What the companies will tell you is that, you know, obviously, in order to sort of, get past those guardrails, like, one of the things you can do is sign up to their respective security programs or you have access to, like, Glasswing or Daybreak. But I think what's important here is for is for teams to, to, game plan this out before incidents happen, right, to sort of, like, practice, like, using models for incident response, in exercises, prior to compromise so that, like, you know, the the day that you need to use specific capability of your model and you're running to the ground, you're not sort of, like, pressed for time, and have to sort of, like, use alternatives that you wouldn't use otherwise. Right? So, that would be my main sort of, like, guidance on this is to, like, develop these capabilities ahead of time, make sure that you're in the right programs, or, alternatively, if you're using open source models, that you have the right sort of, like, infrastructure and harnesses that are that are ready for this. Okay. Thank you. It's comprehensive answer. Did anyone ask me to jump on that, or are we happy any any final comments? K. Hello? Okay. So I can I'm speaking We we no. oh, sorry. I I thought you said say say say say you wanna jump in? It's okay. no. I mean, I think, you know, in I'm just trying to you know, the the best thing that can be done, I think, is, you know, preserving, you know, kind of logs for the investigations and then keeping track of, you know, what was, what was maintained. Because in the in the hugging face incident, you know, like, the the guardrails, you know, according to some of the some of the reporting, you know, blocked, analysis of malicious looking evidence. So I think, you know, having some kind of a kind of way to kind of, gain insight what was done, you know, with, like, pre pre pre arranged authorized access and, kind of having some kind of, local model fallback is, would be good. But, yeah, that's that's the only thing I would add. No worries. No. Thank you for jumping in. And I have a a question here, which I think will take us home given the time we have, and I wanna respect everyone's time joining us. And, Megan, I'm gonna come to you first with this question then. Clem, Adeline, feel free to add any final comments before we close. And it's a short question, but I think it's a it's a it's a good one to finish up with. And it says, so the threat to exploit timeline will become compressed slash faster. Megan, what do you think? Yes. That's that's the short answer. I think that as we have, discussed kind of throughout, just l l I mean, LLMs, agents are kind of driving up time to everything a lot faster. So, I saw in one side maybe even almost a negative, time to exploit where, you know, I think that maybe just describes zero day but or a zero day exploit, but where we're seeing, vulnerabilities be reported and then, or exploits come first and then vulnerability reports. But I think, one challenge that this is introducing to patch management is that it's really the amount of vulnerabilities that are even being reported. These are coming from researchers, right, not necessarily threat actors, but the number of vulnerabilities that are being reported, thanks to kind of more automated vulnerability discovery is outpacing the ability to understand whether those vulnerabilities pose a risk to your organization. So, introducing some level of, automation or some approach to more rapid understanding of whether or not a discovered vulnerability actually poses a risk to your organization is going to be, a critical next step. K. Thank you. Yeah. I was? just gonna I was actually looking at these numbers, fairly recently, and and and I think what's what's quite, what was quite telling and that's what, Megan was alluding to was, if, you know, in '20 kind of 2018, right, so what was that? Like, eight years ago, the average time to exploit, was at, you know, two point three years. Right? It went it's it started going down kind of with every year to kind of a, you know, a to a year and a half, a year, you know, ten months, nine months, so on. In 2016 so far, the average time to exploit is negative 13. Right? So the mean you know, the mean time to exploit has, you know, kind of gone down to kind of negative hours because we're kinda seeing exploitation on average much sooner than, kind of public disclosure of a CV. So that that that goes to kind of, show, you know, the the the increase in in in volume that we've seen, but, also, you know, if, you know, the the NVD is not keeping up not not able to kinda keep up with the with the amount of disclosures. So it just makes, you know, kind of traditional kind of patching cycles, you know, sort. of too slow to, respond. So Yep. And and I agree. And and, Clem, final word from you. It might be a short one, but final word to you. Has the threat to exploit timeline become compressed and faster? I mean, to be honest, not my domain. So I think, you know, that covers it. I I backed what the other two said. Amazing. Good stuff. Alright. Look. Thank you, everyone. Really appreciate it. Huge thank you to Clem, Adeline, Megan. Brilliant analysts here at Insit Group. We're, you know, doing some great work, and some of that is in the, hopefully shared with you there. Again, check it out on the website. We we do publish a lot of this in our blog posts. We do publish, obviously, reports around these topics. And for those of you with access to Record a Future, much more in the platform as well. So we really, really encourage you to take a look at that. We'll be sure to spin up a webinar once you know? And then just to you know, we we we will take these events really seriously. We do take a lot of time to analyze them and really understand the bigger picture of what's going on at the surface. So we'll be sure to respond accordingly as these developments evolve. And we're seeing it every day. But thank you so much, everyone, once again. Have a great rest of your day. Speak again soon, hopefully. Bye. Thanks, everyone.