Based on my very limited knowledge... if you would have dexterous hands with sensitivity on them you could theoretically train a VLA model to do that task, planned by a high-level model like Argon 4, that orchestrates everything. That would as well require in first place a VLA that has similar tasks in its training set and fine-tune it by means of recording hundreds of examples performing teleoperation of the robot hand and arms movements.
Not easy, but definitely getting more and more doable these days.
PS: Didn't spend 20k on hands for my 6k robot, but can confirm it has the cable harness to support them :D
What's the latency like? Is the llm used to create an instruction set for the robot? How is it interrupted for reflex actions like controlling stumbling?
In any case latency is acceptable for navigation I guess, but avoiding moving obstacles is definitely a no-go right now.
It's 1.23 and weights ~30kg. Even though only the R1 Edu can be in theory programmed, it can be easily modded to enable SSH and root access to deploy custom movement policies and interact with the different on-board daemons to control the movements, grab the video data, etc.
I bought a second-hand Intel DeepSense camera and plugged it on one of the spare USB ports available in the main board to have 3D sensing (even though I really need to play a bit more with it). Right now I am playing with MuJoCo for RL training of new policies, integration with VLAs like NVIDIA Isaac GR00T and integration with LLMs.
I have to say its being a wild ride, and despite the small size of the robot, I have already injured myself with it when playing with the walking policies (If you activate a policy while its hanging, it starts literally kicking and punching you...) I cannot image those owning with the bigger brother of it, the G1...