ActOR

01

Operating rooms are short on staff, not on repetitive work.

Operating rooms increasingly contain ambient sensing infrastructure to optimize workflows and utilization. ActOR uses it to guide a humanoid: a wall-mounted camera and the robot’s head-mounted camera form a shared object-level digital twin, and a language-model agent plans over that twin with pre-programmed locomotion and manipulation skills, without task-specific training. Scene updates and skill outcomes then trigger retries, another approach, or a new plan.

ActOR overview. A humanoid carries bottles to a biohazard bin; robot and ambient views form a shared scene representation; an LLM plans, executes, verifies, and receives feedback from robot skills.

1

See the room together

Ambient sensing supplies room-scale context; the head camera supplies near-field geometry. Object labels, 3D positions, extents, and in-hand states are fused in one world frame. Spatial relations ground the planner, while an occupancy map supports obstacle-aware navigation.

2

Plan with reusable skills

Given an instruction and the digital twin, the agent composes Observe, Explore, Navigate, Pick, and Place. Upper-body manipulation and lower-body locomotion are decoupled, with AMO providing whole-body control.

3

Use feedback to adapt

Skill outcomes and scene updates return to the planner. A missed grasp, an incomplete approach, or a moved object can trigger another attempt, a different approach, or a revised plan.

ActOR architecture. A shared scene representation feeds scene state to a closed-loop agent, which sends skill commands to navigation and manipulation and receives feedback.

02

Real-world demonstrations.

Multi-Object Disposal

3×

Failure Recovery

3×

Adaptation to Scene Changes

3×

03

What makes the system work?

Controlled MuJoCo studies isolate ambient perception and closed-loop feedback across retrieval, multi-object collection, and room reset. Each method and condition is evaluated over 60 trials across two ORs reconstructed from real rooms, using the same perception and skill stack as the physical system.

Pulmonology operating room

JHOC operating room

Three simulation plots compare robot-only with ActOR for task success and Explore calls, and open-loop with ActOR for success under nominal and perturbed execution.

Ambient perception reduces exploration

With targets initially outside the robot’s view, mean Explore calls fell from 1.05 to 0.12, while success increased from 66.7% to 80.0%. Perception-related failures fell from 13 to 1. Per-task confidence intervals overlap; the clearest effect is the reduction in exploration.

Feedback enables recovery and replanning

Compared with stopping at the first skill failure, ActOR increased success from 46.7% to 83.3% under nominal execution, and from 8.3% to 51.7% with injected navigation errors, object relocations, or grasp misalignments. This comparison measures the combined contribution of retry and replanning.

04

Citation & acknowledgments.

BibTeX citation

@misc{zhang2026actorambientaibasedagentic,
title         = {ActOR: Ambient AI-Based Agentic Orchestration for Humanoid Autonomy in Operating Rooms},
author        = {Han Zhang and Sicheng Lu and Aleks Santari and Jeffrey K. Jopling and Masaru Ishii and Mathias Unberath},
year          = {2026},
eprint        = {2611.04255},
archivePrefix = {arXiv},
primaryClass  = {cs.RO},
url           = {https://arxiv.org/abs/2611.04255}
}

Acknowledgments

Supported by NSF CAREER Award No. 2239077.