Objectives as Policies:
Vision-Language Model Cost Programs
for Contact-Rich Manipulation

University of Illinois Chicago

*Equal contribution

Place the eraser on top of the box and release it.

OaP connects a vision-language model to a shared physics-based controller through an editable objective program.

Abstract

Vision-language models (VLMs) have shown strong semantic and spatial reasoning capabilities in 3D environments. Recently, connecting VLMs with model-predictive controllers (MPC) has emerged as a paradigm for combining strong semantic reasoning with physics-based planning. However, connecting these two for contact-rich robotic manipulation requires a task representation that bridges the two and captures how physical interactions can be reasoned about by VLMs.

In this work, we introduce Objectives as Policies (OaP), a framework that connects a pretrained VLM to a shared sampling-based MPC through an editable objective program. Each program specifies a sequence of stages for completing a task, combining running and terminal costs from a cost library with stage completion conditions. Given a natural-language task instruction and scene observation, we build a simulation-based digital twin through real-to-sim, and the VLM generates an objective program that the MPC executes in simulation. The VLM iteratively updates the program based on MPC rollouts before deployment in the real world. We evaluate OaP in simulation and on a physical robot across four tabletop manipulation tasks.

Video

Download the video (MP4, 15.9 MB)

Method

OaP builds a digital twin from a scene observation, generates a staged objective program using a VLM and a cost library, refines the program from simulated execution feedback, and deploys it with MPC and a stage verifier on the real robot.

Generate an objective program

The VLM uses the task instruction, scene observation, and cost library to specify stages, running costs, terminal costs, and completion conditions.

Refine in simulation

A shared MPC executes the program in the digital twin. The VLM uses execution feedback to revise the program before deployment.

Execute on the robot

The MPC replans from the measured state using the fixed program. A stage verifier advances the task when its completion conditions are met.

Real-world execution and its digital twin

Each pair shows a real-camera frame (left) and a MuJoCo reconstruction (right)
from the same run’s recorded robot and object states.

Results

The residual library and self-review both improve success in simulation. Without execution feedback, the four variants achieve:

Simulation success across four tasks, ten trials per task.
Each valid program executes once; generation failures count as failures.
Objective representationWithout self-reviewWith self-review
VLM-generated expressions8/40 (20%)14/40 (35%)
Residual library20/40 (50%)26/40 (65%)

With the library, self-review, and at most two feedback rounds, cumulative simulation success reaches 36/40 (90%), allowing up to three executions per trial.

On the physical robot, selected simulation-successful programs succeed in 32/40 executions (80%) across two perturbed scene configurations. See the paper for the full protocol, baselines, and limitations.

BibTeX

@misc{lu2026objectivesaspolicies,
  title  = {Objectives as Policies: Vision-Language Model Cost Programs
            for Contact-Rich Manipulation},
  author = {Zihao Lu and Trong Gia Hung Nguyen and Fuquan Wang and Yifan Zhu},
  year   = {2026},
  url    = {https://zihaolu001.github.io/oap-project/}
}

The arXiv identifier will be added after publication.