Camera-Based Collision-Free Pick-and-Place on a Low-Cost Manipulator
A geometric safe-action projection layer with vision–language semantic monitoring.
Task success does not guarantee that every intermediate command is collision-free, in-workspace, and physically admissible. SafeBench-LCR wraps any policy — a classical planner or a learned transformer — in a layer that returns, at every step, the closest command satisfying eight forward-kinematics-coupled geometric constraints, backed by a reactive obstacle detour and a vision–language semantic monitor, all driven by one fixed RGB-D camera. The demo above is exactly this layer, running live in WebAssembly.
Contributions
Five first-class elements — geometric admissibility, sensing, and semantics.
CMDP safe-action layer
Eight FK-coupled constraints (obstacle, workspace, min-Z, joint & velocity limits, smoothness, semantic veto, e-stop). Returns the closest admissible command to the policy's raw one — no retraining.
CLASP
Confidence- and latency-aware projection: the hold radius is an explicit function of the semantic belief and its age. Provably safe ∧ live — the only policy that is both.
Shortest-path detour
Tangent–arc taut-string routing around a hazard disc with a route-or-stop guard — the behaviour you drive in the demo above.
VLM monitor, on-device
A vision–language model reads the raw RGB frames to judge whether an intruder is a person. FastVLM-0.5B gives the verdict on-device in 0.24 s.
RGB-D fusion
A calibrated image-to-robot homography (0.92 mm) fused with metric depth from the same Intel RealSense D435i localises objects and intruders.
Key results
Simulation for the policy comparison; closed-loop on the physical NED3 Pro for the sensing & safety chain.
| Policy | Static obstacle: delivered | Hand reach-in: collision-free | Verdict |
|---|---|---|---|
| Worst-case (always human) | 0% (over-stops) | 100% | safe, not live |
| Trust-latest (stale verdict) | 100% | 88% (routes into reach-in) | live, not safe |
| CLASP | 99.6% | 100% | safe ∧ live |
| Intruder | Verdict | Behaviour | Outcome |
|---|---|---|---|
| Inert object on the line | object | route around | 3.6 cm detour · 4.0 cm clearance · delivered |
| Human hand on the line | human | freeze | 0.0 mm for ≈35 s, then resume & deliver |
How the demo works
- The real MJCF model (
scene.xml+ned3pro.xml+ STL meshes) is written into the MuJoCo WASM virtual filesystem and compiled withMjModel.from_xml_path. - A Three.js scene is built from the model's geoms; every frame each body is repositioned from
data.xpos/data.xquat. - The controller is a JavaScript port of the paper's stack: a damped-least-squares Jacobian servo (
mj_jacSite, fingers-down), a grasp latch, and the CMDP tangential deflection — so the obstacle you drop is routed around (≈15 cm clearance), or the tool freezes and holds the object if the drop point itself is blocked. - No server, no backend: MuJoCo physics, the safety layer, and rendering all run in your browser.
Code & links
Built with the official Google DeepMind MuJoCo WebAssembly bindings and Three.js. The policy comparison (ACT vs sampling-based replanner) and the semantic-monitor evaluation are reported in the paper; this page demonstrates the geometric safe-action layer interactively.