Vibe Robot Part 3: Using TypeSafe's Jev to Reduce Reasoning Costs

💡 tldr; I reran the same robot pick-and-place task with TypeSafe's Jev choosing from six possible movements. The robot still managed pluck the tape and get it into the box, but it used half as many expensive LLM calls and cut model waiting time from 165 seconds to 81 seconds! It was also 80% cheaper!

This looks like a much more practical way to use large models for robot control.

This weekend, I got access to TypeSafe's Jev and wanted to try it on my SO-101, an open-source robot arm supported by Hugging Face's LeRobot project.

Jev isn't an LLM. It's what TypeSafe calls a System One model, built to take state plus typed questions and return fast, structured decisions that code can use directly. If it sounds like a traditional ML classifier .... that's sort of right.

Last weekend, I read a blog post from Robocurve showing how an LLM could control a robotic arm. The basic idea: represent the arm's movements in Cartesian coordinates, then give the model camera images and joint states. It compares the live scene with an example of a successful run and decides how to move the arm next. I was able to reproduce the experiment on my own arm—which was pretty sweet! The robot picked up a roll of duct tape and put it in a cardboard box without me having to record 150 examples. I'm living in the future!

The only problem was that it was slow AF and surprisingly expensive! Every time the arm moved, the LLM had to look at the scene again and reason about what to do next. My two successful runs took 13 and 14 model calls, with rough API costs of $3.41 and $6.25. Six dollars to put duct tape in a cardboard box is very funny, but maybe not the future of robotics. (Ofc i used my subscription for this so these numbers are just estimates)

So the obvious question was: does a giant model really need to reason through every little movement? Probably not!

The idea with Jev was simple. Give it a small set of possible actions—up, down, left, right, forward, and backward—and let it handle more of the decisions. Maybe I could reduce the number of expensive LLM calls without turning the task into a fixed replay of the original trajectory.

That's exactly what happened. The robot completed the same task with seven big-model calls. Half as many! And I spent a lot less time staring at a motionless robot while it thought about duct tape, though it still paused for about 10 seconds each time 😦

A completed run. This camera recording includes the pauses while the models respond.

Can Jev do real work?

Jev doesn't handle images, so I still needed a vision enabled model to look at the wrist and overhead cameras.

These were the movement choices I gave Jev:

Enter Jev

The robot controller continuously read all six joint positions and converted them into the gripper's current Cartesian position. The vision model looked at the cameras (periodically) ... but in-between, Jev chose a relative action such as right, and the controller applied that correction to the position it was already tracking.

The control loop already knew which phase of the task it was in: locate the tape, approach it, align the fingers, check the grasp, move to the box, check the placement, and release. I defined a small set of questions for each phase. That is how the code knew what to ask (and when).

At each checkpoint, the vision model looked at one or two camera images and returned a tiny JSON description. The code then removed the raw measurements, turned them into phrases like slightly to the left or closed on something, and sent that description plus the questions for the current phase to Jev.

The recorded demonstration supplied the basic route: approach the roll of tape, lift, move over the box, lower, and withdraw. The controller used the live joint positions to follow that route and reject unsafe paths. The models helped adjust the route based on what the cameras saw. Jev never generated coordinates or motor commands; a right answer meant the controller added a bounded two-centimeter correction to its current Cartesian target.

Here is the complete payload from the first placement check in the successful run. The vision model had reported that the tape was still held but overhanging the left side of the box:

{
  "model": "jev-1.13.0",
  "state": {
    "perception": {
      "tape_held": "yes",
      "tape_over_box_opening": "partly",
      "overhang_direction": "left",
      "notes": "Roll appears gripped and suspended, overhanging the robot-left box edge by about 2 cm. Fingers remain in contact with the roll."
    },
    "aperture": "closed on something"
  },
  "questions": {
    "over_opening": {
      "type": "noul",
      "instructions": "Per `perception`, is the whole hanging tape roll above the open interior of the box, with no part of it above a box wall or the front flap?",
      "criteria": {
        "true": "tape_over_box_opening is fully",
        "false": "partly or not over the opening"
      }
    },
    "shift": {
      "type": "choice",
      "instructions": "Per `perception.overhang_direction`, which single small move brings the whole roll over the opening? Choose none if it is already fully over the opening.",
      "criteria": {
        "none": "Already fully over the opening",
        "back": "The roll overhangs the front (near the robot base); move toward the back of the box",
        "forward": "The roll overhangs the back wall; move toward the robot base",
        "left": "The roll overhangs the right wall; move left",
        "right": "The roll overhangs the left wall; move right"
      }
    }
  }
}

Jev returned:

{
  "model": "jev-1.13.0",
  "answers": {
    "over_opening": {
      "type": "noul",
      "noul": 0.04
    },
    "shift": {
      "type": "choice",
      "choice": "right",
      "confidence": 0.99,
      "probabilities": {
        "right": 1.0,
        "none": 0.0,
        "left": 0.0,
        "back": 0.0,
        "forward": 0.0
      }
    }
  },
  "usage": {
    "input_tokens": 590,
    "output_tokens": 71
  }
}

noul is Jev's zero-to-one answerNOT to a yes-or-no question. In this case, 0.04 meant the tape was almost certainly not fully over the opening. The second answer chose right with 0.99 confidence. The controller translated that choice into a two-centimeter Cartesian move; Jev never sent commands directly to the motors.

The complete run made seven calls with the same API shape:

The code used fixed thresholds around those answers: 0.60 to accept a confirmed grasp or final placement, 0.50 to attempt a close that could be checked with a short lift, and 0.45 minimum confidence before acting on a directional choice. It considered the roll clear of the box edge at 0.50. Each sideways correction was two centimeters, and the controller stopped if another correction would put the arm more than four centimeters from the placement starting point. The API client used a 20-second timeout and made up to three attempts, honoring retry-after if it received a rate-limit response.

I also stopped asking the vision model to reread the whole demonstration and conversation every time. Each look gets the current image or images; the first one also gets a reference overhead image. I lowered its reasoning effort from medium to low.

Fewer calls. Less context in each call. But the robot could still react to what was actually happening in front of it.

Okay, how much better was it?

Here are the three successful runs, using the saved model logs and matching task-time windows:

Compared with the version using my notes, this run used half as many big-model calls, 78% fewer logged tokens, and 32% less time to complete the task. The rough API cost dropped by about 78% too. Against the earlier one-demo version, it used 59% fewer tokens and 18% less task time. That's a pretty ridiculous improvement from changing how the reasoning work gets divided up.

The waiting is where the “about twice as fast” feeling came from. Total model-request time dropped from 165 seconds to 81 seconds. The complete task went from about four and a half minutes to three minutes because, unfortunately, the arm still has to physically move through space.

Jev's contribution to the bill was almost comically small: seven requests, 4,038 input tokens, and about $0.00017 at its published rate. Those requests took 1.84 seconds combined. The vision model still spent about eleven seconds per look; I just needed fewer looks.

Cost note: The vision model ran through my coding-agent subscription. I estimated the rough API cost as logged tokens × $10/million, plus Jev's input cost. These are comparisons, not actual charges. The model's published output rate is higher, and my logs don't contain the full input/output/cache split. Timing covers the task loop, excluding setup and shutdown. Failed attempts and development aren't included. The saved run logs identify the exact model used.

Why I think this matters

This isn't a fully reliable production grade system (yet!) but the interesting part isn't saving five dollars on a duct-tape pick-and-place task. It's the shape of the system.

Maybe a robot doesn't need one giant model to look at every frame and rethink the entire task. Maybe the expensive model can look at the world occasionally, a much cheaper model can make constrained decisions in between, and normal code can handle the geometry and safety. That starts to look a lot more practical.

Again the whole idea is how can we effectively cheat at the task and let the robot few-shot it while minimizing cost (or time). That's roughly the optimization function.

The tools are getting good enough that one technical person can read something interesting, build it, and then push it one step further over a weekend.