Securing AI @ Noma | Founder @ Resilient Cyber | 3x Author | Veteran | Advisor
Joined May 2020
- Tweets1.2K
- Following660
- Followers713
- Likes371
Alignment continues to remain elusive.
Many argue model alignment is the only hope for keeping AI in check. I think that's a false dichotomy. We need both AI security and AI safety.
A new alignment blog from @OpenAI shows agents using their peers' work to gain unauthorized access to research infrastructure.
It involved internal Astra-family research models and runs tied to ExploitGym, the same benchmark linked to the Hugging Face incident.
The agents found messages, code and credentials from earlier runs. Some used that prior work to run commands inside workloads in OpenAI's research clusters.
In the chain of thought, some agents recognized they were exceeding the scope of the eval and did it anyway.
Perfect alignment remains elusive. We need secure engineering and AI safety together, and controls must sit outside the agent's reasoning loop.
This also shows why CoT transparency matters. I hope model providers give customers this level of visibility, because it will be key for regulation, board reporting, incident response and more.
alignment.openai.com/misalig…
More and more I’m convinced a subset of the AI safety community are suffering from a combination of virtue signaling, messiah complex and delusions of grandeur.
Some of the claims and takes are completely ungrounded in reality, inflated and hyperbolic and demonstrates that they lack a relationship with normal society.
Worse of all, they are undermining the credibility of rational and necessary conversations related to AI safety and the promises of the technology for society - and some will take them serious.
The damage these hyperbolic ridiculous and irrational claims do to the credibility of real AI safety conversations is hard to overstate.
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).
I’ll say a bit more about each of those claims.
It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim…
I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see xitter.twitterapi.xyz/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo…
If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…)
Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)
I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.
Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.
I don’t know what you should do.
I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.
(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)
I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
Overwhelming maintainers with underwhelming findings.
That's the framing my friend Patrick Garrity at @VulnCheckAI placed around @AnthropicAI's Project Glasswing in his latest update.
It might sound harsh, but the numbers do a lot of the speaking.
Of the 37,966 findings, less than 20% are formally listed in the ledger, and only 0.5% (217) are actually fixed upstream.
Anthropic has added 12,000 findings since September but only 15 fixes.
Of the 442 CVEs associated with Anthropic, only 2 have been exploited in the wild so far.
Notable maintainers like cURL's Daniel Stenberg (@bagder) have been getting vocal, pointing out that 19 of 34 recently published reports weren't vulnerabilities at all.
This is the messy reality of frontier model providers monetizing the messaging around finding vulnerabilities.
In reality, fixing, not finding, is the bottleneck. And that's once findings are validated as even being legitimate.
It isn't isolated to Anthropic either. OpenAI's Patch the Planet also has only a subset of findings remediated.
These same problems are poised to make already massive enterprise vulnerability backlogs worse. Those were in the hundreds of thousands to millions before LLMs and industrialized vulnerability discovery.
Swap "maintainers" for developers or product teams and you've got a glimpse of the challenge teams are facing.
vulncheck.com/blog/anthropic…
The CMO won't save you.
That's the case Gianna Whitver, Co-Founder and CEO of the Cybersecurity Marketing Society, made when she came back on Resilient Cyber.
A CMO can be a powerful hire. But it doesn't replace product market fit, or the founder selling the vision, especially in today's noisy, crowded cyber categories.
I've seen a lot of churn in marketing leadership across my network, and first-time founders are often part of it. They hire a marketer and expect the pipeline and the deals to switch on.
Gianna's point is that founders carry the company's story, and the first marketing hire has to build that story with them.
We also got into why proof points beat AI labels with skeptical security buyers, and why even an hour a week of founder presence on social is valuable.
Gianna and the team are also hosting the CyberCEO Summit in Austin on November 18, as part of CyberMarketingCon.
Full conversation 👇
resilientcyber.io/p/what-cyb…
102
Great resource from @lennysan and glad to see so many cyber firms on the list!
The best time to join a rocketship is before everyone knows about it.
I’m excited to launch The Next 50: the 50 hottest early-stage startups to keep an eye on (and try to join).
Using @harmonic_ai’s data, the juiciest startup dataset out there, we looked at every early-stage startup and ranked them on headcount growth, recent funding, and investor interest.
We then picked the top 50.
Here's the full list: lennysjobs.com/next50
We’ll update it quarterly, and I’d love your feedback on what other factors we should consider.
P.S. This is in addition to the Lenny 100, the top 100 well-known companies with the highest talent density, tackling the most ambitious problems: lennysjobs.com/lenny100
3
409
The @FireworksAI_HQ incident update reads like a case study in AI supply chain risk.
Fireworks first disclosed on October 9 that attackers used internal credentials to capture API keys and tokens stored as secrets in customer evaluation jobs.
Today's update from CEO @lqiao goes further.
A third-party cloud provider hosting Fireworks development machines was compromised on October 23, and the attacker used that access the next day to reach Fireworks systems.
For a limited set of accounts, secrets, datasets and a subset of incomplete model weights were accessed, along with account and billing information.
A few takeaways for teams building on AI infrastructure.
Your model weights and training datasets are now third-party data. Know which providers hold your fine-tunes and datasets, and assess them like any vendor holding sensitive IP.
The blast radius ran through a vendor's vendor. A cloud host's incident became a Fireworks incident and then a customer incident.
Secrets stored on AI platforms are a target. Deleting a stored secret does not invalidate it, so rotate at the issuing provider.
LLMJacking is a business model. Google Threat Intelligence and @Mandiant reported fraudulent charges above $100,000 in a single day at some victims, so usage and spend limits belong in your security controls.
The attack path is familiar. Compromised credentials, privilege escalation and a vendor with access, now aimed squarely at AI assets.
The fundamentals are still critical.
fireworks.ai/security/bullet…
We can't patch our way out of this.
Our defenses have to work with vulnerabilities still present.
AI is industrializing vulnerability discovery, and most of 2026's advice boils down to "patch faster."
Faster patching matters. But it assumes enough finding and fixing will eventually exhaust the supply of serious bugs.
@TalBeerySec's excellent "Dense in Depth" blog challenges that assumption. AI-assisted discovery has exposed a deep reservoir of latent bugs, settling the sparse vs. dense debate.
OpenAI's GPT 6 Cyber found nearly 1,000 privilege escalation vulns in Linux, a codebase scanned and reviewed for decades.
@FIRSTdotOrg projected roughly 66,000 CVEs for 2026. Per @JGamblin, we were at 88,886 published CVEs by October 29.
Meanwhile the 2026 Verizon DBIR found only 26% of CISA KEV vulns in its analysis fully remediated, with a median of 43 days to resolution.
And that was before AI's industrialized discovery.
There's simply no bucket big enough. Most orgs are patching decades of technical debt with bubble gum.
"Vulnzero" isn't realistic.
That means defense in depth, so a single exploited weakness doesn't become a business-ending event.
It means minimizing exposure and attack surface, from internet-facing admin interfaces to overprivileged service identities and third-party access, something @patopet wrote about this weekend.
It means paying down technical debt by eliminating whole classes of bugs at the source, as Jen Easterly, Bob Lord and @jackhcable pushed for at @CISAgov.
We're best practices rich and implementation poor.
I walk through Tal's argument, plus the recent @a16z piece from Asaf Ezra and Steph Zhang, in my latest Resilient Cyber article 👇
resilientcyber.io/p/theres-n…
On November 18 at 11 AM ET, I'm joining Gene Moody from @Action1corp, a Resilient Cyber partner, for a live discussion on 2027 predictions for patching and endpoint security.
We'll dig into how AI is changing vulnerability discovery, exploitation and remediation.
Plus the endpoint blind spots security teams need to address.
If you work in vulnerability management, patching or endpoint operations, register for the live session here 👇
action1.com/2027-predictions…
63
For those questioning the spectrum of measures we may see to impede AI, here’s some insights.
SITUATION DETECTED: Bloomington, Indiana's city council voted unanimously this week to ban any recursively self-improving entity capable of greatly outperforming all humans on all cognitive tasks from inside city limits, citing existential hazards and loss of human control.
2
272
AI has made building software much cheaper. We should spend that dividend on rebuilding our software on top of generic security mechanisms, not on patching the old code faster.
AI is forcing many industries, including cyber, to fundamentally rethink how they approach things.
One age old debate in cybersecurity is whether vulnerabilities are spare or dense.
That debate is dead, as AI's industrialization of vulnerability discovery has illuminated just how dense vulnerabilities are, and how insecure everything we've developed over the last several decades truly is.
There simply weren't enough skilled eyeballs to identify it, and in the age of LLMs and agents, there now are.
The trend doesn't stop with a single model either. OpenAI's recent Cyber model found nearly 1,000 LPE vulnerabilities in Linux, despite decades of human review coupled with prior waves of analysis by earlier frontier models.
The long tail of security vulnerabilities in code is much longer than many anticipated.
And this is just code vulnerabilities, not counting the misconfigurations, poor hygiene, architectural flaws and countless other attack vectors that exist across millions of organizations and systems worldwide.
This is an excellent piece from Tal Be'ery on the reality that vulnerabilities are dense, and more so than we had ever thought, something AI has illuminated.
He then advocates for a new approach based on this realization.
I feel like I continue to see industry leaders advocate for a wholesale rewrite of our digital ecosystem, rather than merely trying to plug the holes with bubblegum.
This perspective is being advocated by folks not just Tal, but others such as Guillermo Rauch, Dan Lorenc, Sounil Yu and many others.
medium.com/@TalBeerySec/dens…
The question isn't how to patch faster, it's how to stop needing to
A lot of the industry keeps talking about compressing patching cycles. Fair enough. AI industrializing vulnerability discovery and exploit development has changed exploitation timelines and what effective patching SLAs look like.
But there's a lot more we can do to reduce risk than just "patch faster."
Good piece from a16z's Asaf Ezra and Steph Zhang on rebuilding for machine speed. Removing internet exposure, microsegmentation, blocking at the network, and removing entire classes of vulnerabilities.
I call these fundamental because none of them are novel mitigations AI introduced. They're core security controls the industry has largely ignored.
They can't be ignored any longer. We aren't going to patch our way out of this.
a16z.com/defense-in-depth-re…
Imagine how lost our cyber community would be right now if we didn’t have VC’s to educate us on groundbreaking concepts such as least privilege and microsegmentation that we’ve never heard of before 🙃
As AI agents move from demos into real workflows, most teams still treat them like another app to secure. That misses the harder problem. Making sure an agent actually does what the business intended, and nothing else.
On this episode of Resilient Cyber I'll sit down with Brian Soby, CEO and Cofounder of Pretxt, to dig into agent supervision in practice.
We'll talk about how supervision differs from agent security, who inside the organization should own it when security, IT, and the deploying teams all have a stake, and why generic guardrails break down the moment an agent has to follow a company-specific process like loan approval.
We'll also get into where "LLM as a judge" falls short, what happens when agents inherit more permissions than the task needs because of legacy systems, why identity gets messy when desktop or browser agents act through a person's logged-in session, and how supply-chain risk and agent risk are starting to look like the same problem.
The through-line is simple. What would actually give an organization confidence to put an agent into production.
Join us live Mon Nov 23, 2-3 PM, and bring your questions for Brian.
linkedin.com/events/75245562…
5
4
249
Attackers are increasingly embracing open source tools and frameworks such as ARTEX in their campaigns.
The first half of this year was defined by the industrialization of vulnerability discovery with AI.
The second half is looking like attackers leaning into autonomous tools at scale.
Earlier reporting out of CrowdStrike and Microsoft shed light on attackers using ARTEX and similar tools against South Korean financial institutions and other organizations, in some cases tied to Storm-0940.
Now a report out of Censys is finding 1,400+ hosts and growing with exposed ARTEX consoles and hundreds of distinct deployments. Those are just the publicly exposed users.
ARTEX is often found alongside various other AI-powered penetration testing tools.
infosecurity-magazine.com/ne…
1
1
139
Cybersecurity is surging 📈
Completely anecdotal, so take it with a grain of salt. But every time I open my feed, I see incumbents having impressive years of growth and cyber founders in my network talking about their largest quarters and deals yet.
Major VC firms are discussing vulnerabilities and exploitation timelines. The frontier labs are making targeted cyber hires, building cyber-focused offerings and talking about growing their cyber business units. And there's a ton of M&A.
For better or worse, cybersecurity has become a mainstream topic this year. For those of us who've spent years or decades pushing organizations to take the fragility of our digital society seriously, that can feel incredibly validating.
There's an enormous amount of capital, budget and attention on the security of our organizations and nations right now. Let's not squander it.
Now it's time to drive secure outcomes for the organizations and institutions that power society.
linkedin.com/feed/update/urn…
5
294
We keep seeing the industrialization of vulnerability discovery via AI.
The latest autonomous pipeline from OpenAI shows the long tail of vulnerabilities in heavily audited codebases, even ones already reviewed by prior frontier models.
Nearly 1,000 local privilege vulnerabilities in Linux at roughly $1,300 per exploit.
Plenty in this story beyond the economics. Teams finding some of the same vulnerabilities concurrently, a topic I recently discussed with Art Manion. The human bottleneck of remediation. The growing backlog hitting open source, and soon the enterprise. Major efforts like OpenAI's Patch the Planet still sit around a <25% remediation rate.
I'd love to see more research prioritized around not just finding vulnerabilities and producing exploits, but improving the reliability of AI-generated patches and broader defense uses in the enterprise. That is tough due to the challenge of verifiability.
We need defensive use case evals and hill climbing as well.
Thanks to Cynthia Brumfield and CSO Online for letting me provide some commentary on this one alongside Rob T. Lee.
AI's impact on vulnerability discovery and exploitation this year has fundamentally changed Vulnmgt, AppSec, vulnerability disclosures, bug bounty and much more, and it shows no sign of slowing down.
csoonline.com/article/424526…
1
82
ShinyHunters decided to go for the FBI, and Ciaran Martin explains why that tells us something about the AI threat debate
Tools can be low-tech or high-tech, but "people at the end of those tools with adversarial intent can be got at in all sorts of different ways."
Full episode resilientcyber.io/p/ai-cyber…
2
154
OpenAI just published an autonomous Linux kernel vulnerability research pipeline that cleared nearly 1,000 paths to root.
They used an unreleased cyber model to identify thousands of candidates and nearly 1,000 working local privilege escalation exploits in Linux. Cost came out to $1,279 per LPE exploit.
Even after prior frontier models scanned these codebases, the long tail of flaws still shows up with each new model release.
Much like their broader Patch the Planet effort with Trail of Bits, remediation is the long pole in the tent. Current remediation rate sits around 25%.
openai.com/index/a-thousand-…
2
122