I like to make computers see. Currently at @rerundotio
Joined March 2017
- Tweets2.4K
- Following758
- Followers3.3K
- Likes4K
Pinned Tweet
🚀 Introducing EgoExo Forge - built on top of @rerundotio, @Gradio, and @huggingface hub
(I’ll be in San Francisco July 21–29 — if you’re into robotics, egocentric AI, large-scale data collection, or just want to chat, DM me!)
In my opinion, large-scale, diverse, and high-quality data is still the largest bottleneck for generalized robotics deployment. I believe that some version of imitation learning from human examples will be the most scalable + clean way to train humanoid robots 🤖 (similar to what Tesla did for Full Self Driving). Teleop is too expensive to collect a large enough dataset in a reasonable manner, so passive collection via egocentric (and in certain cases, exocentric) views feels like the right bet.
Over the past few months, I've been trying to build out the scaffolding for this and using @rerundotio as my underlying infrastructure. Data being collected needs to be easily inspectable + time series and rerun provides the right tooling for this.
My goal is to first build out a ground truth representative dataset from already existing open source data, generate some reasonable baselines, and then go out and collect my own data that adheres to the defined schema.
🔍 Starting with open-source datasets
1. EgoDex from Apple
2. HOCap from Nvidia and the University of Texas at Dallas
3. Assembly101 from Meta
All these different datasets have different sensor configurations + annotations, so my goal with egoexo-forge is to have one consistent labeling scheme + data layout. I built a data pipeline that aligns all of the different datasets in one general schema assuming the COCO133 keypoint layout that allows for exo+ego, ego only, or exo only
Since the scaffolding is already there, it becomes MUCH easier to add other datasets. So the next ones that I'll be including are HD-EPIC kitchens dataset, HOT3D, and finally my own personal iPhone + insta360 go collection method.
Once I have a diverse variety of datasets, I'll double down on what I believe to be the key algorithms required to make useful data for imitation learning 📊
1. Camera Pose estimation via SLAM/SFM for ego perspective (and automatic calibration for exo)
2. Human pose estimation for both egocentric + exocentric views
3. Metric 3D reconstruction + object tracking
I'll be setting up reasonable open-source baselines for each of these to validate that these datasets work, and then finally try to use the generated datasets for some imitation learning via the pi0-lerobot repo I've been working on.
I plan on making a blog post + providing more info on all of this in the near future so stay tuned
Happy 4th 🇺🇸!! I have a preview for my next release (assembly101 is massive, so I need to push until next week). But in the meantime, check out the link below to try out the 🚧 work in progress using @rerundotio and @Gradio
Links below 👇
GSI providing super high-quality, multiview egocentric data that is permissively licensed! All with Rerun native integration.
A really fantastic resource; go check it out and give the data a try.
Big congrats to them
Introducing RoboCap-10k. 10,000 hours of open egocentric data with metric hand / head pose on every frame.
Captured on RoboCap across 11 countries and 21 industries.
GroundedAPI produces 3D enrichment giving robot policies high-fidelity action grounding.
with @BitRobotNetwork @huggingface @rerundotio ↓
Because I'm using both egocentric and exocentric views I needed a strong ego hand model. Originally I was using Wilor BUT then @NVIDIARobotics dropped the egohands model which performed lots better on my initial testing.
So I decided to use this as my "calibration" target when synchronizing egocnetric views with exocentric views.
I went ahead and built a @huggingface space with @Gradio so others could try it out themselves.
a few things as I built a @rerundotio visualization for this
1. I did a bunch of optimization work. Converting to NVFP4/FP8 mixture helps a ton in making things fast (but only works on blackwell gpus) this made it so I got ~7ms | 140 fps inference time on my 5090 for the full end to end run (decode -> detection network -> perspective croppping -> pose network)
2. I included nvidia's soma-hand instead of mano. Has the nice property of being commercially friendly + allowing choice of levels of detail. Its also technically compatible with mano which makes it even more useful!
3. it also has an okay detection model built on rt-detr, I think a much better one could be trained but I just haven't had the inclination to do it yet.
Overall its a better model at least on initial vibes than wilor and produces good outputs. Try out the space and let me know what ya'll think!
Next up is, how do we take sapiens2 + egohand and use them to calibrate all of the egocentric + exocentric views?
Thanks to @UmarIqb, @jiefengli_jeff, and team for the great model!
Part of doing this alignment work is running Sapiens2 on the exo videos. Thing is, its really slow. Even on my 5090, I'm barely hitting 28FPS for a single video, and I need to run it on 4x videos.
So I had claude help me make this faster and now I'm hitting ~170fps. @rerundotio mcp was invalubale here to hill climb
We managed about a 6x speedup by doing 3 things
baseline - PyTorch bf16, 1024x768 resolution (36.4ms forward pass)
1. exported to tensorrt (31.7ms | 1.15x speedup)
2. NVFP4/FP8 backbone and the head changes (11.7 ms | 2.7x speedup)
3. reduced resolution from 1024x768 -> 768x576 (5.6 ms | 2.1× speedup)
I included a @rerundotio video to show that the output is very very similar (90% of keypoints are within 1% of pixels)
This is still super slow compared to RTMW, which hits 0.3ms for a forward pass so another factor of 10. So we'll see how much more I can squeeze out of it. I feel like maybe some distillation or something could be done!
Bit robot team always working on great stuff
1/ RoboCap data is getting a sense of touch 👋
We’re adding tactile sensing with @moxianmxkj MX Matrix Gloves compatibility.
Together, they provide hand contact and pressure data to visual RoboCap demos, giving researchers more context for dexterous robot learning. ↓
Seems like the promise of Mojo is starting to be fulfilled. Gonna try and rewrite some cuda kernels with it and see how it goes
3DGS now running without CUDA, without Python and with a single language for both training code and GPU kernels.
Finally got around to update MojoSplat (using @Modular Mojo), matching @fhahlbohm Faster-GS on CUDA GPUs without even CUDA Toolkit installed
github.com/bertaveira/mojosp…
1
32
2,413
Part of doing this alignment work is running Sapiens2 on the exo videos. Thing is, its really slow. Even on my 5090, I'm barely hitting 28FPS for a single video, and I need to run it on 4x videos.
So I had claude help me make this faster and now I'm hitting ~170fps. @rerundotio mcp was invalubale here to hill climb
We managed about a 6x speedup by doing 3 things
baseline - PyTorch bf16, 1024x768 resolution (36.4ms forward pass)
1. exported to tensorrt (31.7ms | 1.15x speedup)
2. NVFP4/FP8 backbone and the head changes (11.7 ms | 2.7x speedup)
3. reduced resolution from 1024x768 -> 768x576 (5.6 ms | 2.1× speedup)
I included a @rerundotio video to show that the output is very very similar (90% of keypoints are within 1% of pixels)
This is still super slow compared to RTMW, which hits 0.3ms for a forward pass so another factor of 10. So we'll see how much more I can squeeze out of it. I feel like maybe some distillation or something could be done!
In the previous post I'd talked about the chosen equipment, which conisted of the reference quest3, an oak stereo camera, and 4 iphones.
These produce a bunch of raw files and I wanted to show the process of going from raw files (mp4/json/csv) -> @rerundotio data format (rrd) in the most time and memory efficient way.
The end results is the attached video. This gets all the underlying data into a single unified representation. BUT it doesn' t yet provide ensure the time alignment or calibration between them, that comes in the next steps. (You can clearly see it how the hands dont correctly line up in the egocentric videos)
its really great to see such great progress. I just wish more of these would be open-sourced!
when LaMAria was introduced a year ago, Meta's closed-source Aria SLAM was far ahead of academic systems.
one year later, we've surpassed it in 3/5 categories: medium, long, and low light.
here's what closing that gap looks like:
1
14
1,123
Fantastic, I think dealing with dopplegangers directly in the network is such a good idea! Going to spend some time trying it out
To match or not to match?
FlexGlue 🐙, our new image-matching model, can match every pixel AND reject doppelganger pairs, tackling a common failure case in SfM.
➡️State-of-the-art SfM and matching, blazing fast
➡️Code: github.com/cvg/FlexGlue
➡️Paper: arxiv.org/pdf/2610.32976
1/
In the previous post I'd talked about the chosen equipment, which conisted of the reference quest3, an oak stereo camera, and 4 iphones.
These produce a bunch of raw files and I wanted to show the process of going from raw files (mp4/json/csv) -> @rerundotio data format (rrd) in the most time and memory efficient way.
The end results is the attached video. This gets all the underlying data into a single unified representation. BUT it doesn' t yet provide ensure the time alignment or calibration between them, that comes in the next steps. (You can clearly see it how the hands dont correctly line up in the egocentric videos)
Alright so we now have opensource versions of
1. on device multicam egocentric visual inertial odometry
2. multi-iphone recording software (that are timesynced with ntp)
3. Gaussian splatting at acceptable FPS in @rerundotio
I decided to take some data I had previously collected that was quite messy and see how far Opus 5.5 could fix it with the rerun mcp and the context of all of the different projects I've worked on so far.
I'll break this down into multiple parts (Today just part 1)
1. hardware <- this thread
2. collection
3. time and spatial alignment
4. Post processing
5. Data cleaning
The main thing here is using our built in readers or rerun send chunks
We can use the mp4reader as an example. Given some raw mp4 file we could do the naive thing and iterate through the video with opencv and use
rr.log("video", rr.Image(frame))
but this would be extremely slow and inefficient. We lose all of the compression benefits mp4 provides and do things one frame at a time. Instead @rerundotio has built readers for many common formats such as mp4.
This removes the need for decoding the video, sending frames 1 by 1 and dealing with timestamps.
This is also the case for other data such as CSV/MCAP/Ect
Next post I'll explain how we go from a bunch of unaligned data and synchronize it across devices
1
163
Alright so we now have opensource versions of
1. on device multicam egocentric visual inertial odometry
2. multi-iphone recording software (that are timesynced with ntp)
3. Gaussian splatting at acceptable FPS in @rerundotio
I decided to take some data I had previously collected that was quite messy and see how far Opus 5.5 could fix it with the rerun mcp and the context of all of the different projects I've worked on so far.
I'll break this down into multiple parts (Today just part 1)
1. hardware <- this thread
2. collection
3. time and spatial alignment
4. Post processing
5. Data cleaning
Splatting is going to be a critical part of the ego collection pipeline IMO, so I'm working on improving our current official support.
code here - - <github.com/rerun-io/examples…>
The current shipped version does well for less than 1M splats, but not so much for larger ones such as the iconic garden scene (5.2M splats). I ported over brushes kernels to @rerundotio and now on my M5 I'm getting ~40-50fps when recording or a solid 60fps when not. This is basically a 10x improvement from the current version and this huge improvement that wouldn't have been possible without all the verification loops we've added with the rerun skills and MCP. I'll break down what was required to make this work ⬇︎
First to set context. I collected this data using the quest3 as the "referernce" device.
Theres were two main reasons for this in the past
1. The quest3 has really great slam and handtracking out of the box
2. Its super cheap and easily accessible
The problem is, the quest also has a bunch of downsides, primarily being they don't provide access to the slam tracking cameras.
To get around this, i basically 3d printed a little holder for the luxonis oak-d stereo camera that I could attach it to the quest and a proxy full field of view.
along with this, I wanted to include exocentric views. I think these can be somewhat scalable. Getting 2-4 iphones is easy and getting them to all record together is now as well. It won't be genloc quality sync, but I think we can do alot with purely audio/visual/ntp sync
So the in total the hardware is
1. Quest3S
2. OakD-Wide
3. 4 iPhone 12 minis
Next one, I'll talk about how do we make sure they are all time aligned AND also spatially aligned.
3
416
Looks like the 512GB Mac Studio is out. For the low low price of 25k 🫠🫠🫠 with a delivery date of December
1
2
832
Opensource methods slowly climbing the lamaria ladder
First submission of visloc-rs, my Rust-based SLAM system, to LaMAria—a benchmark for evaluating SLAM on real-world, head-mounted recordings, with challenges including low light, long trajectories, and moving platforms.
🏅 4th overall and 4th in the stereo + IMU category
🥇 Highest-ranked method with publicly available source code
📊 Average score across five categories: 51.6
This is just the first submission. I’ll keep improving accuracy and robustness!
Cool new whole body dataset!
We release Gen-HumanWholeBody, an egocentric whole-body manipulation dataset with industry's highest precision. The dataset is now open-source! 🚀 huggingface.co/datasets/genr…
🚶600 hours of whole-body manipulation data
🎯 ~3 cm whole-body MPJPE
🌍 16,976 real-world scenarios
🦾 1,281 human skills
We’re opening this dataset to help researchers and developers advance whole-body motion learning, human-to-robot transfer and embodied foundation models.
Contact Us:opendata@genrobot.ai
Discord: discord.gg/rSSb5thgu
2
478
kornia-slam pipeline has been a huge help for all the slam work I've been doing. Really worth checking them out. I'm planning on contributing the work ive done so far to the!
Our new #rust kornia-slam pipeline is finally ou.
Check the site for more details: kornia.org/slam/
It’s fully #opensource — we are looking for battle testing users and contributors to improve it. Feel free to reach out to us !
Splatting is going to be a critical part of the ego collection pipeline IMO, so I'm working on improving our current official support.
code here - - <github.com/rerun-io/examples…>
The current shipped version does well for less than 1M splats, but not so much for larger ones such as the iconic garden scene (5.2M splats). I ported over brushes kernels to @rerundotio and now on my M5 I'm getting ~40-50fps when recording or a solid 60fps when not. This is basically a 10x improvement from the current version and this huge improvement that wouldn't have been possible without all the verification loops we've added with the rerun skills and MCP. I'll break down what was required to make this work ⬇︎
I wanted to share something I've been working towards since forever. I've finally built out a full slam + hand reconstruction pipeline that works IN REAL TIME at 30fps on a single rockchip 3588. All streaming directly to @rerundotio from a @BitRobotNetwork robocap. This is similar to what a Quest3 or Project Aria 2 provides, but opensource 🙂
I'm so excited I finally don't have to really on closed source software like the meta quest or others. I finally have a realtime performant pipeline that works on an edge computer, but importantly can also scale to a high performance one when needed.
Truly on a roll, another great slam paper to look at
RootSLAM: Square Root Bundle Adjustment for Visual-Inertial SLAM
📄 arxiv.org/abs/2610.15157
⚡ State-of-the-art SLAM accuracy, 3–14× faster!
🤝 A collaboration with Tomás Santucci and Prof. Taihú Pire (@taihu_pire) at the UNR🇦🇷, and Prof. Daniel Cremers at the TUM🇩🇪.
19
1,128
Mateo and team are just cranking out papers!
Our preprint HexVIO is now on arxiv! arxiv.org/abs/2610.03283v1
We use the commodity Hexagon DSP from the Galaxy S23 to do Visual-Inertial Odometry (VIO) on a phone for 0.83W (whole-device, screen on, no ISP or DMA optimizations, no root). Around 18 hours of battery on this phone.
More SLAM progress, cool to see folks using LaMaria.
I'd love to know more about the runtime speed and what hardware was used to get this.