Founder @Railway, the peaceful way to ship software. People, Product, Production.
SF via 🇨🇦
Joined June 2012
- Tweets21.5K
- Following1.3K
- Followers35K
- Likes36.1K
Bottlenecks towards 10,000x more software:
- Writing code
- Compiling code
- CI/testing code (you are here)
- Reviewing code
- CD/deploying code
- Observing code
Expect the b̶a̶s̶i̶l̶i̶s̶k̶ bottleneck to move accordingly
4
27
2,236
This is indeed a very concerning trend
Our investigations into the recent wave of AI inference infra attacks point to a concerning pattern.
We believe Anthropic’s crackdown on distillation may be accelerating a shift toward stolen credentials and compromised inference infrastructure.
Our findings suggest certain Chinese operators are involved in networks reselling frontier model access through heavily discounted AI gateways.
The economics are simple: steal API keys, hijack inference capacity, and resell tokens at a fraction of legitimate prices.
We suspect these are not isolated incidents, but part of a growing underground economy built around stolen AI compute. Inference infrastructure is becoming a prime target for organized cybercrime.
We’ve published our full post mortem from October 21st
We encourage people to read it in depth. Attackers now have access to alarmingly powerful tooling, and are hitting companies all across our industry
We hope this can help other companies, big and small, protect their users
A detailed report of the October 21st security attack
blog.railway.com/p/incident-…
Last night, I became aware of the concerns raised in Josh's post. I reached out publicly and over DM to offer a potential explanation based on the limited information available at the time, while working to understand what happened.
As background, we’ve internally been pushing 100% on security of the Railway platform. At Railway, we believe that cyber threats are real and mounting, and we’ve built out tooling for automatic scanning + CVE remediation (for both our infrastructure and customers). Over the last few weeks, oncall engineers were instructed to red team our infrastructure. We maintain specific policies and short lived grants for when these actions occur.
Subsequently, I became aware that an employee at Railway had constructed, on their own time, a tool for proactive identification + reporting of bug bounties within the broader community.
The employee, using their own personal accounts, personal credit card, and time outside of work hours, signed up for a personal subscription to a hosted service running Deepseek. I can confirm they ran this tool once and only once, on a singular external provider, and halted it proactively within 15 minutes of starting.
Railway does not condone these actions. We expect all our employees to uphold our standards of conduct, whether acting on behalf of Railway or in a personal capacity. We create an environment where internal red teaming has strict guidelines, and when bug bounty disclosures must occur, we follow every letter of responsible disclosure.
We have addressed this incident with the employee directly and have taken appropriate action. We are also reinforcing our expectations for employee conduct to help prevent similar situations.
Readers added context they thought people might want to know
Primcloud does not have a bug bounty program, per its founder. Its Acceptable Use Policy forbids unauthorized probing, scanning or vulnerability testing without prior written consent.
x.com/joshmanders/st…
primcloud.com/aup
7
8
10,495
Last night, I became aware of the concerns raised in Josh's post. I reached out publicly and over DM to offer a potential explanation based on the limited information available at the time, while working to understand what happened.
As background, we’ve internally been pushing 100% on security of the Railway platform. At Railway, we believe that cyber threats are real and mounting, and we’ve built out tooling for automatic scanning + CVE remediation (for both our infrastructure and customers). Over the last few weeks, oncall engineers were instructed to red team our infrastructure. We maintain specific policies and short lived grants for when these actions occur.
Subsequently, I became aware that an employee at Railway had constructed, on their own time, a tool for proactive identification + reporting of bug bounties within the broader community.
The employee, using their own personal accounts, personal credit card, and time outside of work hours, signed up for a personal subscription to a hosted service running Deepseek. I can confirm they ran this tool once and only once, on a singular external provider, and halted it proactively within 15 minutes of starting.
Railway does not condone these actions. We expect all our employees to uphold our standards of conduct, whether acting on behalf of Railway or in a personal capacity. We create an environment where internal red teaming has strict guidelines, and when bug bounty disclosures must occur, we follow every letter of responsible disclosure.
We have addressed this incident with the employee directly and have taken appropriate action. We are also reinforcing our expectations for employee conduct to help prevent similar situations.
Hey @Railway, is there a reason employee's of yours are trying to breach my infrastructure?
Your employee Jake Runzer, just attempted to tonight.
Readers added context they thought people might want to know
Primcloud does not have a bug bounty program, per its founder. Its Acceptable Use Policy forbids unauthorized probing, scanning or vulnerability testing without prior written consent.
x.com/joshmanders/st…
primcloud.com/aup
We built some pretty neat technology to allow you to do private connections to databases from anywhere (including the browser)
Excited to bring this to templates soon :)
You want a spacetime view of all actions across every surface area
Anchored to “What you, personally” know
This is the shared representation of how both you and your agents think about these things
8
12,675
This bug has been in the kernel for 10 years BTW
It’s hard to communicate to folks the rate at which these models move now
What a disaster atlasum.net/blog/cve-2026-98…
Jake retweeted
The ultimate form of preparation is not planning for a specific scenario, but a mindset that can handle uncertainty.
You have to say close to the customer
Even when the customer is punching you in the face
Probably especially then IMO
With CVEs having gone parabolic, the only way to fight is automatically
We will now mitigate known vulnerable versions automatically for users
Much more to come here!
The Preliminary Report about our Security Incident on October 21st is now live
There are a variety of things in here that we've done, so it is worth reading thoroughly
A full report is to follow
The preliminary report related to the October 21st Security Incident is now available.
blog.railway.com/p/prelimina…
Some notes:
- We believe the attacker was using abliterated models with large compute
- We believe they were targeting AI API keys for distillation
- We notified providers, up to "Every key we had for the provider", shortly after unauthorized decrypts occurred, before our initial public disclosure
- We confirmed that the attacker was removed
- We have kept details high level for the security of our users
- We will be posting subsequent follow ups, with more details, as our investigation concludes
2
34
3,588
First round of known affected customer disclosures went out last night
Post mortem work continues and should be out at some point today, likely closer to mid-day PST
Preliminary Post Mortem is now Available:
xitter.twitterapi.xyz/Railway/status/2113671…
The preliminary report related to the October 21st Security Incident is now available.
blog.railway.com/p/prelimina…
13
1,682
All deploys have now been restored
Affected customers will finishing receiving emails in the next hour or so
Finally, out of an abundance of caution, we will be posting the post mortem around 9am PST tomorrow
Updated on the 9am PST target:
We will likely be trending closer to midday PST
xitter.twitterapi.xyz/JustJake/status/211366…
First round of known affected customer disclosures went out last night
Post mortem work continues and should be out at some point today, likely closer to mid-day PST
2
1,051