Teaching an SO-101 to Pick and Place: From Simulator Debugging to 1,000 Demonstrations

The task sounded simple: pick up a black perfume cap, move it into a cardboard tray, open the gripper, and withdraw.

Getting a robot to do that in simulation was not just a matter of starting PPO and watching the reward curve rise. We had to establish that the contacts were physically plausible, that the grasp target was actually between the fingers, that “success” meant the sequence we wanted, and that a successful rollout was good enough to become a training demonstration.

This is the story of that process, including the approaches that did not work.

The result so far: a learned, state-based PPO policy and 1,000 filtered simulation episodes in LeRobot format, with top and wrist images at 30 Hz. This is a data-generation milestone, not evidence that we have solved real-world transfer or already improved a vision-language-action model.

Why we started in simulation

Our original motivation came from real SO-101 demonstrations and SmolVLA training for tape pick-and-place. The vision policy appeared to overfit to the tape’s location. We wanted a way to generate more varied robot experience without manually recording every variation.

We started from NVIDIA’s Sim-to-Real SO-101 Workshop, reusing its robot and task infrastructure. Its vial-to-rack task became a simpler cap-to-tray task: a rigid black cylinder, 36 mm in diameter and 40 mm tall, and a shallow, open cardboard tray.

This was an intermediate task, not a claim that perfume-cap data automatically solves tape manipulation. The longer-term plan remains to use simulation experience to help a vision policy generalize, then measure that improvement on the real robot.

A Collision Problem in Isaac Sim

The cap could penetrate the gripper and remain stuck even after the jaw opened. Tests pointed to a collision issue in our Isaac Sim/PhysX setup, not a reward or PPO problem. The cap was overlapping the robot, not deforming.

Replacing its collider with a convex mesh enabled natural release in our tests without changing the visible object or disabling collisions. We did not establish whether the underlying cause was a simulator bug, the asset configuration, or their interaction.

What the policy actually learns

There are three different systems in this project, and their results should not be mixed together.

System	Inputs and role	What it establishes
Controlled motion expert	Simulator state and a hand-designed sequence	Whether the current geometry and actuators can complete the task
Learned PPO policy	89 privileged state features; six joint-position targets as actions	Whether a neural policy can learn the simulated manipulation sequence
Intended vision policy	Top/wrist RGB, robot state, and a task instruction	Future camera-based control and real-world transfer; not yet demonstrated here

The PPO observations include robot and object state, contact information, and task-history features. These are not all available to a real camera-based policy. Its six outputs control the five arm joints and jaw through absolute joint-position targets, using the environment’s action mapping.

Training used 256 parallel environments, 120 Hz physics, and native 60 Hz policy control. Checkpoints preserved the actor, critic, observation normalization, optimizer, and learning-rate state. Restoring only the actor would not have been equivalent to resuming training.

In practical terms, learning alternates between collecting experience and updating the network. The actor samples actions, the simulator returns new observations and rewards, and a critic estimates expected future return. PPO uses those rollouts to adjust which actions the actor favors, with a clipped objective intended to discourage overly large policy changes. We supply the task and reward signals, not a correct joint command for every state. The original PPO paper describes this interaction-and-update framework.

That is why reward design matters so much: the algorithm optimizes the signal it receives, not our verbal intention. It also explains why evaluation must freeze the weights and run new episodes, rather than treating the training reward as the result.

A short imitation-learning branch also used the motion expert as a demonstration source. It did not establish a reliable improvement, and it was not the source of the final PPO result reported below. An expert artifact is not an RL checkpoint, even if both happen to use a .pt extension.

Reward going up was not the same as learning the task

Early dashboards were encouraging. One reward revision produced roughly 97% logged success. Later checks showed why that number was not a stable-placement result: the success rule was permissive, and the collider still had penetration problems.

Other runs increased total reward while grasp, lift, or placement stayed near zero. That suggested intermediate rewards might be more attractive than completing the task, but a dashboard alone could not establish the cause. Contact detection, reset state, geometry, termination, and normalization also changed across experiments.

We needed to watch trajectories and inspect the accounting together.

Measure the grasp where the grasp happens

The original reach target was roughly 90 mm away from the grasp opening. A policy could reduce its reach error by bringing the wrong part of the hand toward the cap.

We added a dedicated grasp-center measurement using a verified local offset transformed by the gripper’s full pose, while preserving the existing IK link frame. This is a useful grasp reference, not a claim that a fixed offset is the exact midpoint of a moving jaw at every opening.

Diagnostics also needed the velocity of that point, including motion caused by wrist rotation. Link-origin velocity alone could underestimate how fast the fingers were approaching.

In one ten-episode evaluation, every first approach had insufficient opening for the 36 mm cap, with approach speeds around 2.6 to 3.4 m/s. Four episodes registered moving-jaw contact; none achieved sustained grasp or lift. Contact was real, but “touched the cap” was the wrong success metric.

Opening the jaw at reset helped the opening metric but did not solve grasping. A later heavily gated, multiplicative reward candidate stopped approaching effectively. Useful feedback had become too difficult to obtain before the policy was already doing most of the right things.

Reward progress through a sequence, not indefinite possession

The sustained reward family moved toward smooth, additive pregrasp feedback and finite progress credit. Distance, opening, alignment, and approach speed could contribute before perfect contact. Secure grasp still required stronger evidence than a brief impact.

The positive shaping budgets in the later configuration were:

Component	Maximum credit per episode
Open, aligned pregrasp progress	4
Closure progress after readiness	2
Secure grasp	4
Actual lift above resting height	4
Transport toward the tray	4
Controlled lowering	4
Release and withdrawal progress	3
Settling progress	3
Total shaping budget	28
Separate strict terminal success bonus	100

For a component score s between zero and one, a per-episode high-water mark h, and budget B, the basic accounting is:

new_h = max(h, s)
progress_credit = B * (new_h - h)
h = new_h

Holding the same pose earns no new progress. Moving away and returning to a previously achieved score cannot repay it. Scores and history reset per environment; initial pregrasp credit is seeded to avoid a free reward just for starting open.

This prevents unlimited repeated shaping credit, but it is not a proof of an optimal reward design. A policy can still prefer an easy partial strategy. Discounting, time costs, action-change costs, palm-contact costs, and episode termination all affect the return. We checked full discounted trajectories, not just the size of the terminal bonus. With gamma = 0.999 at the original 60 Hz reward steps, completing earlier and completing later are meaningfully different.

Timestep accounting was another practical trap. The reward manager multiplies contributions by the control timestep. One-off progress credits and the terminal bonus compensate for that once, so terminal success pays 100, not 100 × dt. Rate-based costs retain their time dependence.

Lift is measured above the cap’s orientation-aware resting height. A cap still sitting on the table must not earn lift progress. Lowering credit combines position progress with smooth feedback for being slow near the tray floor. Previously earned lift credit is not subtracted when the cap moves down. Release feedback then rewards the supported opening, separation, and withdrawal sequence.

This supported lowering without a general-purpose hovering penalty. The remaining tradeoff is that a finite progress reward encourages reaching a state, not necessarily staying there. Stable completion still needs its own checks.

Define the sequence, not just the final location

“The cap is inside the tray” can be true after a push, a knock, or an accidental slip. That is not the demonstration we wanted a vision policy to imitate.

The stricter sequence checks required sustained two-finger grasp, controlled lift, transport, a qualifying opening near the tray floor, separation, withdrawal, and a stable final interval. Secure grasp included at least 0.25 seconds of appropriate two-finger contact, low relative motion, and no strong palm contact. Controlled lift required at least 20 mm above the resting height while securely held.

At the end, containment uses the full cylinder bounds in the tray’s frame, not just its center point. The cap must be near the floor, have linear speed below 0.03 m/s and angular speed below 0.5 rad/s, and remain clear of the monitored gripper contacts through the required 0.5-second stable interval after release and withdrawal.

Contact diagnostics were recorded at the 120 Hz physics rate. Watching only video frames or sampling once per slower policy step could miss short strikes and confuse them with a grasp.

We also found a learning-system issue: empirical normalization could amplify rare stage flags and timers. A separate variant used fixed, bounded scaling for those channels in both actor and critic while retaining normal preprocessing for continuous measurements. Applying that change alone did not immediately improve success; the policy needed to adapt. It was a measured preprocessing change, not a universal PPO recipe.

A successful placement can still be a poor demonstration

When we began reviewing data, another failure became obvious: after leaving the cap in the tray, the hand sometimes closed and opened repeatedly, occasionally making contact again.

These episodes could legitimately satisfy the strict rule later. Release history was latched, and the robot eventually withdrew and completed a clean final stability interval. The logging was not necessarily wrong. The predicate simply did not express every property we wanted in a demonstration.

We kept strict success intact and introduced a separate clean-release quality check: no post-release contact of at least 1 N, and no renewed jaw-target closing greater than 0.10 rad from its post-release running maximum.

For learning, a separate fine-tune added a targeted cost after certified release: 1 point per radian of renewed commanded closing, plus up to 5 points per second of renewed contact. The contact cost rises from zero at 0.1 N to saturation at 1 N. Appropriate opening and holding open incur no such cost; pre-release grasping is unaffected. The tradeoff is that it also discourages potentially useful recovery attempts that involve re-grasping after release.

There was no scripted jaw lock or action override. We fine-tuned the learned policy for 1,000 additional updates from the preserved stage-scaled checkpoint, then evaluated it independently.

Matched evaluation: before and after

Each row below contains 20 completed episodes using the same original-task randomized starts: base seed 12346 plus the episode index. Actions came from deterministic learned inference, with no expert intervention. Physics remained at 120 Hz.

Policy and control rate	Secure grasp	Controlled lift	Strict placement	Clean-release placement	Timeouts
Before clean-release fine-tuning, 60 Hz	20/20	20/20	9/20	0/20	11/20
After fine-tuning, 60 Hz	20/20	20/20	20/20	19/20	0/20
Before fine-tuning, 30 Hz	9/20	9/20	2/20	0/20	18/20
After fine-tuning, 30 Hz	13/20	13/20	12/20	9/20	8/20

No drop terminations were recorded in these four batches. The final 30 Hz batch still had three episodes with palm strikes. A small, single-seed-set comparison is evidence of improvement, not a reliability guarantee or a multi-training-seed result.

These two clips show the same initial seed at 60 Hz. Each combines the top and wrist views. Both episodes eventually pass strict success; only the second passes the extra clean-release check.

Thirty frames per second is not the same as 30 Hz control

Our target dataset needed 30 Hz observations and actions. Simply exporting every second frame from a 60 Hz policy would hide the intervening commands.

Instead, collection queried the policy at 30 Hz and held each target across two native environment steps, retaining the 120 Hz contact diagnostics. This changed the closed-loop behavior. The matched results above show the cost: the final policy’s strict success fell from 20/20 at 60 Hz to 12/20 at 30 Hz.

Here is a failure worth keeping in the report: the final policy at 30 Hz on the same initial seed as the two clips above. It reaches the 12-second timeout rather than completing strict placement.

The practical response was to evaluate at the intended rate and filter collection, not to relabel the failure or claim that the native-rate result carried over unchanged.

From rollouts to a usable LeRobot dataset

The completed collection contains 1,000 accepted episodes from 3,008 completed attempts, totaling 86,832 synchronized frames, or about 48.2 minutes at 30 Hz. Every accepted episode passed both strict success and the additional clean-release trace check.

The other 2,008 attempts were not accepted. That includes unsuccessful attempts and episodes that did not meet the extra quality criteria. These are success-selected demonstrations; 1,000 out of 1,000 accepted episodes passing the filter is not a 100% policy success rate. The collection total also does not count unfinished attempts interrupted by renderer crashes.

We exported LeRobotDataset v3 with:

• observation.images.top: 640 × 480 RGB at 30 Hz.
• observation.images.wrist: 640 × 480 RGB at 30 Hz.
• observation.state: six robot joint-position values.
• action: six applied joint-position targets.
• Task, episode, frame, and timestamp metadata.

Images and state correspond to the pre-action observation; the action records the target actually applied for that interval. Privileged cap poses, contact histories, and success-stage flags remain diagnostic information, not inputs we pretend the future vision policy can observe. The final shorter interval, when an episode ends between recording ticks, is handled explicitly rather than silently dropping the terminal evidence.

These are the two camera streams from the first accepted episode. They have the same episode timing, although the browser players below have independent controls.

Export validation checked episode counts, frame/action alignment, held targets, physics samples, decoded images, and the success traces. The clips include the stability interval needed for acceptance; they do not imply a separately recorded extra second after every success.

There is an important remaining calibration boundary: matching LeRobot field names does not make simulation joint values interchangeable with a real SO-101’s calibrated units. The export uses the workshop’s joint mapping. Joint order, signs, ranges, gripper calibration, camera conventions, and timing need explicit checks before mixing it with real demonstrations or deploying a learned vision policy.

What We Learned

Higher reward did not always mean better behavior. The clearest evidence came from watching the robot complete the whole sequence: grasp, lift, place, release, and withdraw. Even successful episodes needed a closer look before becoming demonstrations worth learning from.

What Comes Next

We now have a learned PPO policy and 1,000 filtered simulation demonstrations. Next, we will combine simulated and real data to fine-tune SmolVLA, then evaluate it on new object locations with the real robot. That transfer is still to be demonstrated.

The question now is simple: will this experience help the real robot succeed when the object moves somewhere new?

Evidence and credits

The numerical comparison comes from four saved 20-episode evaluations and the completed dataset/export validation records. A compact results record includes the protocol, counts, and checkpoint fingerprints; it does not publish the full dataset or model weights. The videos above are actual recorded rollouts, not generated illustrations.

This work builds on NVIDIA’s SO-101 workshop, Isaac Lab, and the LeRobot project. The task changes, reward iterations, evaluations, and data-quality findings described here are specific to our setup. Results are reported as of September 18, 2026.