TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning

ZhiYuan Feng1,†, Yu Deng2, Ruichuan An3,†, Zhenhua Liu2, Qixiu Li1,†, Keming Wu1, Zhiying Du4,†, Weijie Wang5, Haoxiao Wang5, Shuang Chen5, Sicheng Xu2, Yaobo Liang2, Jiaolong Yang2,✉, Baining Guo2
1Tsinghua University 2Microsoft Research Asia 3Peking University
4Fudan University 5Zhejiang University
Paper Code
Video 中文讲解 · Mandarin

Author Walkthrough

A short walkthrough by the first author (narrated in Mandarin) — the motivation, the Ground → Infer → Execute pipeline, and the key results in under three minutes. Pick whichever platform loads best for you.

Or open it directly on YouTube Bilibili

The Problem

Real Requests Aren't Clean Task Specs

In real homes, people don't hand a robot a tidy goal list — they speak in situations. A request is grounded in the current scene and leaves the relevant objects, intended conditions, and ordering implicit. The agent must first figure out what to plan for.

What the user says

“The storm is getting worse, and rain is coming into the office. My grandson is calling soon. Can you get my cup and the small table ready in there?”

Grandmother Grandmother, to her household robot
What it actually means
  • Find hidden entities: the office window, the dirty cup, the small table, the target room.
  • Recover intended goals: the window must be closed, the table and cup clean, the cup on the table.
  • Respect ordering: close the window first; clean the table before placing the cup on it.

None of this is stated in the words alone. Before any action, the agent must infer an intermediate executable task structure from the complete scene and the situated request. We call this setting full-scene household reasoning — and it is what TaskGround is built to solve.

TaskGround overview
TaskGround turns a complete household scene and a situated, conversational request into a grounded skill-level action sequence through a training-free Ground → Infer → Execute pipeline — grounding only task-relevant entities, inferring and completing executable task structure, and compiling it into actions, with up to 18× lower input-token cost.
18×
lower total input-token cost on large complete scenes
9B ≈ GPT-5
Qwen3.5-9B becomes competitive with GPT-5 direct prompting
400
human-validated tasks in the FullHome benchmark
Training-free
model-agnostic; no task-specific tuning required
Abstract

Full-Scene Household Reasoning

In real home deployments, household agents must often act from a complete household scene and a situated household request, rather than from a clean task specification. Such requests require an agent to identify task-relevant entities, recover intended task conditions, and resolve ordering constraints from surrounding scene context. We formalize this as full-scene household reasoning: given a complete scene and a situated request, the agent must infer executable task structure before producing a grounded skill-level action sequence.

This setting is hard because complete scenes contain substantial task-irrelevant information, making direct complete-scene prompting inefficient and error-prone — and privacy and local-compute constraints favor compact open-weight models with limited long-context reasoning. We propose TaskGround, a training-free and model-agnostic Ground–Infer–Execute framework that grounds complete scenes into compact task-relevant scene slices, infers and completes executable task structure, and compiles it into grounded skill-level actions. On FullHome, a human-validated suite of 400 tasks, TaskGround improves success rates by large margins across proprietary and open-weight models, makes Qwen3.5-9B competitive with GPT-5 while cutting input-token cost by up to 18×, and identifies executable task-structure inference as a central bottleneck.

Why it's hard

From a Cluttered Scene to a Casual Request

A complete home has many rooms, objects, states and relations — while the request leaves goals and ordering implicit. Three forces make naive complete-scene prompting impractical.

01

Task-irrelevant clutter

Complete scenes are dominated by irrelevant objects and relations, so feeding the whole scene to an LLM is costly and error-prone.

02

Privacy pressure

Detailed household state reveals personal spaces and routines, making it undesirable to send complete-scene inputs to frontier proprietary models.

03

Local compute limits

On-device deployment favors compact open-weight models that are harder pressed to reason over long, cluttered inputs.

Method

The Ground – Infer – Execute Framework

TaskGround framework
TaskGround decomposes full-scene reasoning into three training-free stages, keeping executability while drastically reducing the reasoning burden on the base model.
1

Ground — scene slicing

A single LLM call selects task-relevant entities from the complete scene S0; a compact scene slice Sr is then reconstructed around them, cutting downstream input cost.

2

Infer & complete

Over the slice, the model infers raw goal predicates ĝ, then completes them into a consistent task structure τ̃ using household priors and process-critical subgoals — no oracle needed.

3

Execute — skill actions

A fixed, task-agnostic executor resolves generic preconditions and compiles the completed structure into a grounded action sequence a1:T.

Results

Main Results on FullHome

Goal SR and Process SR (%) on VirtualHome and BEHAVIOR. Cells shaded pink exceed GPT-5 Naive, the frontier direct-prompting reference per column; (+Δ) is TaskGround's gain over the same model's Naive result; bold marks the best in each group.

Model Method VirtualHome BEHAVIOR
Goal SR Process SR Goal SR Process SR
Proprietary models
GPT-4o Naive 8.05.0 32.323.3
TaskGround 49.0(+41.0) 37.0(+32.0) 52.3(+20.0) 35.5(+12.2)
GPT-4.1 Naive 14.04.0 32.317.1
TaskGround 57.0(+43.0) 34.0(+30.0) 64.6(+32.3) 45.7(+28.6)
Gemini-2.5-Flash Naive 0.52.0 9.25.7
TaskGround 48.5(+48.0) 19.0(+17.0) 38.5(+29.3) 37.1(+31.4)
GPT-5 Naive 45.526.0 46.231.4
TaskGround 73.5(+28.0) 46.0(+20.0) 64.6(+18.4) 51.4(+20.0)
Open-weight models
DeepSeek-V4-Flash Naive 28.515.0 27.725.7
TaskGround 63.0(+34.5) 36.0(+21.0) 47.7(+20.0) 42.9(+17.2)
MiMo-V2-Flash Naive 5.01.0 10.811.4
TaskGround 43.5(+38.5) 23.0(+22.0) 43.1(+32.3) 40.0(+28.6)
Gemma-3-12B Naive 0.01.0 0.05.7
TaskGround 37.0(+37.0) 19.0(+18.0) 29.2(+29.2) 34.3(+28.6)
Qwen3.5-9B Naive 0.010.0 1.514.3
TaskGround 47.5(+47.5) 29.0(+19.0) 43.1(+41.6) 40.0(+25.7)
exceeds GPT-5 Naive (+Δ) gain over same-model Naive bold = best within model group
TaskGround ablation on VirtualHome
Ablation on VirtualHome. TaskGround (ours) beats the Naive full-scene baseline, Ground-Act, and Ground-Infer (wo completion)-Execute, approaching the GT Scene Slice-Infer-Execute upper bound (red dashed) while using ~18× fewer input tokens than Naive.

BibTeX

@article{feng2026taskground,
  title   = {TaskGround: Structured Executable Task Inference for Full-Scene Household Reasoning},
  author  = {Feng, ZhiYuan and Deng, Yu and An, Ruichuan and Liu, Zhenhua and Li, Qixiu
             and Wu, Keming and Du, Zhiying and Wang, Weijie and Wang, Haoxiao and Chen, Shuang
             and Xu, Sicheng and Liang, Yaobo and Yang, Jiaolong and Guo, Baining},
  journal = {arXiv preprint arXiv:2605.18109},
  year    = {2026}
}