Robotic Policies may not need much data - Vibe Robot Part 2

💡 tldr; I got a robot to put duct tape in a box after giving it one recorded example (previously took over 100 examples).

I've been messing around with robotic arms for a few years now. A Theragun massager, some attempts at arm and shoulder physical therapy, a coffee robot, box folding. Most recently I trained a small pick-and-place policy that actually worked, even on my little 10GB GPU. I recorded 122 demonstrations and got an xArm to pick up a syringe and put it in a box.

I was pretty happy with that. I was less excited about recording another 122 demonstrations every time I wanted it to learn a new task. You get into robotics imagining all the repetitive work the robot will do for you, and then spend your weekend doing repetitive examples only for the robot to do them worse and slower lol.

So I wanted to try this Robocurve experiment. They gave GPT-6 Astra camera images and control of robot arms, and reported 19 out of 20 successes putting a block in a bowl. The model looks, asks for a movement, waits for the robot, then looks again.

I already had an arm and a box. Seemed worth a shot.

Show the task once

We used their published agent framework, with some adaptation for my SO-101. They used a YAM arm; mine has fewer positioning joints and two cameras, one overhead and one on the wrist. I wanted to see whether the approach worked on my desk, with my hardware.

I had a few recordings of myself moving the tape into the box. We (my bot and I) picked one clean example and gave the model images and arm positions from different points along the way. Basically: here is me doing the thing, here is what the table looks like now, please do the thing.

After all the work of collecting demos and training the last policy, I found this suspiciously little to ask of me. It turned out there was plenty of work left, just elsewhere.

Several hours of elsewhere

Getting their code to move my arm properly took a bunch of debugging. I needed a controller to translate positions into joint movements, and had problems with wrist orientation, checking whether the arm had actually reached a position, and keeping the gripper closed.

That last one was particularly dumb. When the fingers close around tape, they stop before they're fully shut because there is tape in the way. Our code confused that measured position with how hard I should keep commanding the gripper to close. So I had to fix the part where successfully holding something caused us to stop holding it properly.

I also spent time arguing with the agent about whether the arm was in an acceptable starting position. It was the exact position I use to start teleoperation. I was sitting there looking at it. Eventually I fixed the startup check and had it move itself into position.

One attempt picked up the tape and started carrying it toward the box, then stopped because of wrist tracking near the end of its travel. Another kept closing on air without grabbing anything. The model thought the fingers were lined up, but in the wrist images you could see that the finger hadn't gone far enough into the hole. In theory the overhead cam should have helped with this but i maybe should have had that camera positioned more horizontally.

I thought that run had hit its limit of 20 model calls. It had actually given up at 18. I had apparently found the robot's patience limit before finding my own, which surprised me given the conversation we were having.

Eventually it worked: 13 calls, tape picked up and transported into the box. The roll appears to have slipped free inside the box just before the open command, so the finish was a little scrappy. At this point I was very willing to accept scrappy.

Can I just tell it what it's doing wrong?

Watching it miss, I kept coming back to the example. The tape was only an inch or two from where it had been in the recording. Why all the deliberation? Follow the next part of the example, make a small adjustment, and if you miss, back up a little along the path you just took and try again.

I also wanted to tell it a few things at particular points:

• Get a finger farther inside the tape's core before closing.
• Lift a little and check whether you actually caught it.
• Lift high enough that the bottom of the tape clears the box wall.
• Get the whole roll over the opening before letting go.

The box height was especially annoying. You can get the gripper over the wall and still drive the tape hanging underneath it straight into the cardboard. I could see what was about to happen and explain it in a sentence. Previously my instinct would have been to record more examples of clearing the wall and go train again.

I put those instructions in the prompt before starting the next run. So the model had one demonstration plus my notes about the bits it had been getting wrong; I wasn't typing directions at it during the attempt.

The next morning, the first grasp worked (yay!). It lifted the tape, carried it over, lowered it, opened the fingers and moved clear. Fourteen model calls. I have watched the video more times than the complexity of the task would seem to warrant.

A very expensive way to put duct tape in a box

It pauses to think between moves, so watching it is a very different experience from watching a trained policy. I used my AI subscription, but wanted to know roughly what the two successful runs would have cost through the API:

That's nearly a million logged tokens to move tape twice, before counting any of the failed attempts or the hours spent debugging. The tape is getting excellent service.

Accounting footnote: the saved counter adds uncached input and output tokens, but omits cached input. At GPT-6 Astra's September 14, 2026 API rates ($10/million input, $1 cached, $50 output), pricing that entire counter as input gives the floors above. Output costs more and cached input adds something too; we didn't save the split needed for an exact bill. The model repeatedly reads the example, images and conversation, which helps explain the count. These figures cover only the two successful runs, and the second run's higher usage doesn't isolate the cost of my notes.

What I want to try next

The part I'm stuck on is how little I had to show it. Last time I needed 122 demos. This time it worked with only one. Different arms and tasks, and obviously a giant pretrained model has already learned a lot before meeting my duct tape. Still, I'd been assuming that my next robotics project would start with another long session of collecting data. Maybe it doesn't have to.

And I hadn't really considered that I could supplement an example by explaining a mistake in English. I spend all day giving coding agents instructions like this. Somehow, with a physical robot, I was still thinking I needed to demonstrate everything with the arm.

We have one successful run of each version, so I can't tell you that adding my notes made it more reliable. I'd like to repeat it enough to find out, and then try another task with one or two examples and a few sentences. We saved the prompt, recordings, code and configuration so I don't have to reconstruct whatever the hell we did from a chat history next weekend.

I also want to try this with an open-source model. It's slow and expensive now, but I don't see why the same basic approach couldn't eventually run locally and much faster.

So what about all the robotics companies collecting first party data with thousands of hours of people cleaning homes? Their approach will work just fine AND at some point, as open source catches up, you should be able to run a gpt-6 astra or claude fable level model on a reasonably priced machine with inference times in milliseconds and be able to orchestrate complex robotics tasks with only a few examples. And thats pretty cool!