Driving a 6-DoF robot arm in real time from a Meta Quest 3, with an in-headset visual guidance layer, using no leader arm and no dedicated teleop hardware.
SO-101 Robot
VR Teleoperation
Project OVerview
Role: Sole developer. Hardware setup, IK integration, VR client, debugging.
Goal: Collect real-world manipulation demonstrations for imitation learning without buying a leader arm or a commercial teleop rig.
Stack: LeRobot, Pyroki, MuJoCo, Isaac Sim, Vuer, WebXR, Python, Meta Quest 3, SO-101 with Feetech STS3215 servos.
Outcome: End-to-end teleop holding a stable 60 Hz control loop on a consumer laptop, with an operator-assist visual layer that, as far as I can find, has not been documented for Quest-to-SO-101 control.
why this problem is worth solving
Robot manipulation policies learn from demonstrations, and demonstrations are the bottleneck. Simulation scales cheaply but cannot originate contact dynamics, friction, or sensor noise it has never observed. Real demonstrations have to come from a human physically driving a robot.
The standard way to do that is a leader-follower rig: a second arm the operator holds, mirrored by the robot. It works, and it means every additional operator needs another piece of hardware. At industrial scale those rigs run into six figures. Even at hobby scale, the leader arm doubles the bill of materials for a setup that already costs [$X].
I wanted to find out how much of that could be replaced by a headset most people already own.
The constraint
Removing the leader arm
The SO-101 ships as a leader-follower pair. You hold the leader, the follower copies you. It's the reliable approach and it has a structural problem: every operator needs their own physical leader arm. The hardware cost scales linearly with the number of people collecting data, which is exactly the wrong shape for a bottleneck you're trying to widen.
I bought only the follower. The follower is the arm that does the work, and it can execute any task the pair can. The open question was what replaces the leader as an input device.
A VR headset is a 6-DoF pose tracker with sub-millimeter precision that millions of people already own. If the controller pose can be retargeted onto the arm's end effector, the leader arm becomes software.
That was the bet. The rest of the project is what it took to make it hold up.
Simulation first, and where it stopped
Removing the leader arm
Before touching the hardware I trained and tested the SO-101 in MuJoCo and NVIDIA Isaac Sim. Simulation was the right first move: it let me get the URDF, joint limits, and IK behavior correct without risking servo damage, and it forced me to understand the kinematic model rather than treat the arm as a black box.
It also showed me the ceiling. A policy that performs in sim degrades on contact-rich tasks in the real world, because simulation cannot originate the physical effects it was never given: friction on a specific surface, servo backlash, cable drag, sensor noise, the latency of the real control loop.
Simulation amplifies real data. It doesn't produce the seed. To get the seed I needed a human driving the physical arm, which brought me back to the input device problem.
What I built
Building the teleop stack
I tested several public VR-control repos for LeRobot arms. None worked end to end on my setup, so I took a few Github Repos as a backbone and rewrote what didn't hold.
Quest 3 controller pose streams over WebXR to a Python host. Pyroki solves IK for the end effector target. LeRobot's SO101Follower drives the Feetech STS3215 servos with relative target clamping so a fast hand movement can't command a violent joint jump. Pinch maps to the gripper, and a USB camera feed streams back into the headset. The loop holds 60 Hz on an M3 Pro MacBook, CPU only. No workstation, no motion capture, no leader arm.
Making it controllable, not just controlled
Following my hand was the halfway point. The operator can't see where the arm's reachable space ends, so you command a target it can't reach, IK returns the nearest feasible solution, and the arm lurches.
I fixed that in two iterations. First, a bounding sphere in VR representing the arm's real range of motion, with controller position inside it mapping one to one onto end effector position. Most of the erratic movement disappeared, because most of it was me asking for the impossible.
Second, a rigged 3D mesh of the SO-101 in the headset, driven by the same joint solution going to the hardware. The operator sees full joint configuration instead of just end effector position, so a folding elbow or a wrist hitting its rotation limit is visible before the real arm gets there.
I could not find a documented implementation of this for Quest-to-SO-101 control. The pieces exist separately. Combining them into a working operator-assist layer on consumer hardware appears to be new.
VR Headset controller tele-operating SO-101 Robot Arm
Left - Follower Arm Right - Leader Arm
