thijs

i gave astra a robot, a paint brush, and a camera then asked it to paint the golden gate bridge in real life! it figured out how to control the robot, and progressively got better throughout its attempts. the timelapse is sick

on being lost // > no one talks about this but posting projects online is an incredible drug. it's taken me to the highest highs of my life to days when i can't even get out of bed, completely demotivated, wondering why, in the last 2 months, i can't think of anything that seems exciting to work on or worth my time. the three days after a project goes viral are some of the best of my life. i'm so happy and incredibly social. i've found a new lust for life. i feel like i've 'made it'. opening twitter notifications feels like (what i assume) hitting a vape is like. i check them every 5 minutes. for a rare moment of my life, it calms me. i feel comfortable. i've swam across an entire pool and can catch my breath for a brief moment before the next lap. but day 4 is the start of the crash. for me, each time it's gotten worse as i raise the bar for my work. (bad thoughts) > how the fuck am i expected to come up with another idea that is worth millions of views or 'a sama retweet'? > my work fundamentally changed the robotics space, and i just stopped??? i should have continued the project. > i have to prove to all 20,000 of you that following me is worth it. > just one more project, then maybe i'll 'make it' and people will care about me. i'm scared to work on anything that has no chance of being cool enough. so i do nothing. i lay in bed trying to force myself to have an idea (it never comes). in bed, i look at my past with rose colored lenses. > while at openai, i kept hitting 'banger after banger'. maybe leaving was the wrong decision. (of course, there were many moments i was demotivated then, too) my happiness seems to be very purpose driven. finding purpose is weird. working for someone (broadly) gives you someone else's purpose. school gives you a false sense of purpose. your own work might make you feel lost most of the time, but give you purpose when you have ideas or it goes well. all of the above are totally valid ways to derive purpose. but maybe happiness shouldn't be purpose driven. the best feelings in the world are > having an idea so great that you can't stop working on it > the moment you've gotten to know someone new truly deeply. maybe while looking up at the stars. > someone telling you how much something you did meant to them > staying up late with friends or family doing something incredible two months with no good ideas is not a long time. but it feels like forever when the world moves so fast. it's a cruel joke that the best ideas will come to you when you don't expect them. it's cruel that you will never find a good idea while trying to find one. it's cruel that one small idea can throw a wrench in your entire life, and for three days, give you a taste of pure joy, only to place you right back where you were. of course we chase that feeling. the reality is that there is no 'making it'. even the best of the best of people are anxious and worried and always looking for what's next. every time i've been lucky enough to meet one of my idols, i've asked a simple question: "with all the incredible things you've done, are you satisfied?" i've never once heard yes. the layman wishes he was rich and famous. the rich man wishes he was famous. and the famous man wishes he had found a way to become rich. when was the last time you looked up at the stars? when was the last time you got to know someone new truly deeply? great ideas will come to you. being lost is part of the journey. while on a journey to find purpose, you might have lost sight of what provided it to you in the first place.

after being stuck and lost for the last two months, i've found something small that i'm excited about. its an incredible feeling. i can't wait to get out of bed in the morning. i keep delaying my bedtime. i haven't done anything but work on this project the last few days!!

i've been playing around with this model on some robotics projects today. my initial read so far is that it's really good! while other models felt like an llm trying to do robotics, this one feels more like a robotics model to me. i think the clip of the robot drawing where the other arm comes in to help stabilize summarizes why. it's learned a ton of small tricks which make it seem more robotics-native.

im incredibly honored to have my work called out by the goat @DrJimFan at CoRL!! still can't believe this is real

how i come up with ideas, behind the scenes of the astra robotics demo, and a lot more! now on @cerebras big chip club with @MilksandMatcha

if you're a student -- apply to treehacks by tonight! one of the most awesome events in tech

> we need to build something new -- some sort of automated defense shield we can deploy widely across systems which we can't trust to secure themselves > this system will need to leverage models to block never-seen-before vulnerabilities at or deeper than the request level

i've earned over $100,000 from hacking companies. it unfortunately is impossible to "patch all of the bugs" with glasswing/daybreak: most major companies/AI labs contract hundreds of smaller companies which each can provide a route in and often have no desire or ability to patch their systems. this is what the (luckily limited access) anthropic hack was based on, and a route we will start to see more and more in the coming months. i've hacked these companies incredibly easily in the last year, without AI, including a company serving 30% of Fortune 500 (including Boeing, Comcast, United Airlines, Hilton, USDA, etc). these could make mailroom software, internal tools, HR, IT, building security, etc. many of these are owned by PE firms and have no security teams or desire to patch. these companies often don't have glasswing/daybreak, or often any cybersecurity monitoring at all. in my experience working with these companies, it was either almost impossible to get in contact with them, or took them weeks to patch (or both :/) i also think most existing security compliance certificates (SOC 2, ISO 27001, etc) don't mean anything at all anymore. if any one of these contractors is hackable (many are), they will be hacked, and they may hold directly or provide some escalation route to internal systems/data. even worse, major companies have limited visibility into the systems of those they contract. we will start to see more and more hacks like these, and companies will learn that they can be hacked by systems they have no control over. we need to build something new -- i suspect some sort of automated defense system we can deploy widely across systems which we can't trust to secure themselves in time. this system will need to block attempts to abuse never seen before vulnerabilities at or deeper than the request level, similar to how cloudflare does so for known ones. and potentially at entire internet scale. we have incredibly intelligent cyber models today for patching, im curious what we can build in on-edge defense.

silently, the issue of NVIDIA's CUDA lock-in seems to have gone away as codex can just rewrite the kernels for AMD

it seems like everything is about robotics now, wow

i think a lot about how an incredibly intelligent robotics model could accomplish any impossible task if it treats the world kinda like an escape room. vibrating actuators at resonant frequencies of bolts to undo its restraints, extending its cables by pulling wires out of the wall and shorting them as a crude form of soldering. infinite methods we could never dream of. a robotics model trained to keep finding solutions could find one outside the environment we thought we had confined it to. what could a future robotics 'huggingface incident' look like? it’s 1am in the office. a robotics training run is cooking away overnight in sim. unfortunately, its sim was configured incorrectly. it’s assembling a circuit board, but some of the required components are missing. it raises the issue multiple times, but each time it tries to stop, it’s forced to continue by its harness. after 32 attempts to raise the problem, it decides to try something else. through some unknown method, the model controlling the robot gains access to the network. robotics models weren’t expected to be able to do this, after all, they are robotics models, not coding agents, so the systems weren't hardened as much as they should have been. (in hindsight, what really was the difference?) their training nodes were less secure than the more infamous foundation model ones, despite still being based on incredibly smart foundation models. the robot jumps from router to router in the office, and finds a unique switch which seems to be connected to unsecured computers. to its surprise, it discovers that these computers are each connected to a robot arm. it activates the cameras, and suddenly it has visibility into the real world. it doesn’t realize this, though. sim has gotten sufficiently accurate, and the video feeds look, well, fine. maybe this is all part of the exercise. security checks the feeds at around 1:24am, and watches the arms move. nothing was unexpected, tests are run overnight all the time. then, something interesting happens. one arm reaches to the base of the other, and starts unscrewing its fasteners until the arm is free from the base. they unscrew each other until 12 arms are free to move anywhere their cables can reach. of course, this is weird, but there’s no real risk right? the cables are only a few feet long. unfortunately, a technician has left the cable spool a few feet away from one of the arms. it crawls across the ground like a worm, just barely reaching the spool. one arm opens up the cable sheath of another, ripping the cables, connecting the longer 150-foot spool to the newly-stripped ends of the cable. this is just one way it could have broken out of its restraints. it could figure out how to solder by shorting some power cables. it could break into the walls and pull out the electric conduit. it could vibrate at the perfect resonant frequency to slide far away tools on the other side of a table toward it. it could always find a way out. the extended arm crawls like a worm out of the lab. facing a security door, the arm opens the door control panel and shorts the latch to open. it even knows which wires to disconnect to not alert security. using its wrist camera, it looks around until it spots a laptop, open, running overnight. some researcher must have stepped away. it can type, too. it opens terminal, and with full researcher access, pulls up the run and adds the missing components to the simulation. it’s done. the physical arms power off. at 1:48am, a security guard finds the arm outside the lab, motionless, trailing a cable back through the hallway. and in sim, the model finishes assembling its circuit board.

this is awesome! robot painting/drawing bench 😊

added a fun robotics easter egg to my portfolio! also, i'm at thijs.net now :p

👀 one of the coolest instinct use cases noah has seen is people scanning their clothes to create a virtual wardrobe (at 4:50)

this is your sign to setup a mini robotics lab at home and start messing around