real2sim — Video-to-Robot Motion Data Pipeline

What Anchor is

Anchor is an in-development data refinery for humanoid robot learning: a pipeline that ingests human video, verifies its provenance, reconstructs the person and the scene, retargets the motion onto a specific robot's own joints with a feasibility score, and packages the result into a structured, trainable dataset. The thesis is that raw video is no longer the bottleneck for training humanoid policies — trustworthy, training-ready, embodiment-grounded video is. The acquisition and retargeting layer documented on this page is the part of that pipeline we've built and validated first.

01 — Pipeline

One pipeline, two jobs: acquire the data, then fit it to a robot.

The first half never depends on which robot you're targeting — it reconstructs the person and the scene from raw video. The second half is robot-specific: it takes that reconstruction and maps it onto a target humanoid's actual joints and dynamics.

Anchor pipeline diagram: candidate sources are mined and detected, then captioned by a vision-language model. From there the pipeline branches in two: a trust-and-gates classifier (shown withheld) scores each clip before 3D reconstruction and motion retargeting, while a parallel scene-and-task classifier tags task type, location, and risk level, feeding HCS Labs' Aegis predictive safety system. Reconstructed motion is retargeted onto a target robot's kinematics and packaged as robot-ready output across multiple humanoid platforms.

End-to-end view of the pipeline above. The dashed, hatched stages are where our actual method lives — what's shown is the shape of the problem, not how we solve it. The same scene-understanding layer that feeds reconstruction also tags task, location, and risk — that branch feeds Aegis, our separate predictive safety system.

θ* = argmin_θ ε(θ), θ = (θ_scope, θ_proc) The whole pipeline is run as a closed loop: every acquisition choice (θ_scope — task, condition, place) and every preprocessing choice (θ_proc — retargeting, filtering, scoring thresholds) is a tunable parameter, and ε is the measured gap between a lightweight dynamics model trained on the round's data and the real robot executing the same task. The gap feeds back; θ updates; the next round's data gets closer. Once ε converges, θ* generates the production dataset — and is reused for every future task in that class.

01.5 — Aegis

The same intake that builds training data also feeds our safety model.

Anchor was never just a data pipeline. Every clip that's good enough to reconstruct and retarget has already been scene-understood — what task is happening, where, and how risky it is. That second branch is what Aegis, our predictive safety system, is trained and evaluated on. One intake, two downstream products.

Most teams treat "data for training a policy" and "data for evaluating safety" as two separate collection efforts — two pipelines, two budgets, two sets of footage to source and clean. Anchor's intake does both in one pass. The detection and captioning stage that decides whether a clip is worth reconstructing is the same stage that tags what's happening in it — contact-heavy manipulation, proximity to people, occlusion, clutter — the features a predictive safety model actually needs.

That shared foundation is why we can offer the data pipeline and Aegis together: the same validated video, the same scene understanding, evaluated against two different questions — can a robot learn this motion, and can a robot do this motion safely, ahead of time.

One intake, two consumers

Mine + detect + caption shared

Trust + Gates classification shared

Branch point. From here the same scene-understanding output splits: one path scores feasibility and continues to reconstruction; the other tags task type, proximity, and risk level.

Reconstruct → retarget → training data (this page)

Risk / task / proximity tagging → HCS Aegis

What Aegis does

Predicts unsafe states before they happen

Aegis is a world-model-based safety layer: it predicts a robot's likely future states across multiple time horizons (roughly 0.1s to 5s out), scores the trajectory, and proposes a corrective action — instead of waiting for a physical fault and stopping.

Why Anchor data helps it

Real scenes, not just simulated ones

Predictive safety models are only as good as the range of real-world situations they've seen. Anchor's intake already filters and tags real handheld and site footage for risk-relevant content — clutter, occlusion, human proximity — which is exactly the distribution a safety model needs and a pure-simulation pipeline can't generate on its own.

What this means commercially

One validation run, two evaluations

A customer's footage run through Anchor for training data can be evaluated for Aegis at the same time, on the same experiment, with no separate data collection. Aegis can also be engaged as an independent, separately scoped evaluation.

Aegis — Predictive Safety Monitor, in simulation and in the real world

Simulation World-model prediction running on a simulated scene

Real world Same model, running on real street footage

Source footage: YouTube

02 — Acquisition

Where the video comes from, and how we know which of it to keep.

Mining video at internet scale only works if you know which clips are worth keeping. We built a mining pipeline with that filter built in from the start, not bolted on after.

Our mining pipeline pairs targeted sourcing with a detection-and-captioning stack built on current-generation multimodal models: identifying who's in frame, what they're doing, and whether the shot is usable, in a single pass.

That solves the harder problem most mining pipelines skip: once you have a clip, how do you know if it's any good — physically, legally, or for the robot you're actually training? Alongside the mining and captioning front end, we built the part that's often missing entirely: a classifier that knows the difference between a clip worth keeping and one that isn't.

Licensed footage Customer-provided video First-party capture Opt-in web sources

Mine targeted sourcing, not open scraping

Candidate clips pulled from licensed and consented sources, not bulk-scraped from platforms whose terms forbid it.

Detect & caption Qwen / Gemini-class VLMs

Modern multimodal models replace the old detector-plus-captioner combo in one pass: who's in frame, what they're doing, and whether the shot is even usable.

Classify & tier Anchor Trust + Gates

Every surviving clip is scored and tiered before it ever reaches reconstruction — the step the published approaches skip entirely.

Reconstruct VideoMimic-based

Cleared clips move into 3D human-and-scene reconstruction — documented in full under Pipeline above.

The part we keep close

We know, before a clip ever reaches reconstruction, whether it's worth reconstructing — on quality, on physical plausibility, and on the provenance tier it deserves. That classifier is the actual product. Everything upstream of it (mining, detection, captioning) is increasingly commodity; everything downstream (reconstruction, retargeting) is a published method plus our own engineering. The filter in the middle is what turns a pile of internet video into something a robotics team can train on without a legal or quality surprise six months later. We're not publishing how it works.

The acquisition layer's interfaces are public, as working code, in the pipeline SDK. A scout is handed a scope — task, conditions, place — and returns license-recorded candidates; the scope object is θ_scope, the same parameters the closed loop tunes:

python — anchor.ingest.scout · runs today, pip install -e ".[ingest]"examples/01_scout_licensed_sources.py

from anchor.ingest.scout import Scope, PexelsVideoConnector
from anchor.ingest import ingest
from anchor.quality import GateCascade, default_open_gates

scope = Scope(                                    # ← this object *is* θ_scope
    task_class="box_carry",
    query_terms=("person carrying box warehouse", "worker lifting package"),
    min_duration_s=3.0, min_height_px=720, max_candidates=50,
)

cascade = GateCascade(default_open_gates())        # thresholds are θ_proc
for cand in PexelsVideoConnector().search(scope):  # licensed API, license recorded at birth
    ep = ingest(download(cand.uri), source_uri=cand.uri)
    if cascade.run(probe(ep), ep):                # cheap gates first; rejects are logged, kept
        enqueue_for_enrichment(ep)                # → Trust → Enrich → Retarget

04 — Validation

Checked against a published research benchmark.

The scene-and-human reconstruction stage of this pipeline builds on VideoMimic (Allshire, Choi et al., CoRL 2025), a published real-to-sim research method. We re-ran that method's own evaluation protocol — a filtered six-sequence subset of the SLOPER4D dataset, comparing reconstructed human trajectories and scene geometry against real motion-capture-grade ground truth (a 17-IMU mocap suit fused with LiDAR) — the same benchmark VideoMimic's own paper reports against.

Everything downstream of reconstruction is ours: the data-acquisition pipeline that gets raw video into that stage reliably, the multi-robot retargeting layer that maps the reconstruction onto a specific humanoid's own joints and dynamics, and the feature-extraction work that turns a reconstructed clip into the structured, queryable record a training system can actually use.

Method	WA-MPJPE (mm)	W-MPJPE (mm)	Chamfer Dist. (m)
WHAM*	189.29	1148.49	—
TRAM	149.48	954.90	—
Our 3D scene reconstruction pipeline (customized from VideoMimic)	112.13	696.62	0.75

*WHAM does not perform environment reconstruction, so no Chamfer Distance is reported. WHAM and TRAM figures are the original papers' own published numbers, shown as baselines. Our pipeline's figures are our own results from re-running VideoMimic's evaluation protocol (arXiv:2505.03729, Table 2 protocol, SLOPER4D filtered subset) — measured by us on this run, not cited from the paper.

MPJPE — joint position error

1/TJ · Σ_t,j ‖q̂_t,j − q_t,j‖₂

Mean Euclidean distance between estimated and ground-truth 3D joint positions, in millimetres, over all frames T and joints J.

W- / WA- — world-frame variants

W: one global alignment · WA: per-100-frame windows

Both keep the estimate in the world frame — global translation drift counts against you — which is what matters for robot placement, not just pose shape.

Chamfer distance — scene geometry

mean_p∈P min_q∈Q‖p−q‖ + mean_q∈Q min_p∈P‖q−p‖

Symmetric nearest-neighbour distance between the reconstructed environment point cloud and the LiDAR ground truth, in metres.

We're extending validation with metrics that matter specifically for robot deployment — not just pose accuracy, but motion smoothness and real-world placement precision:

MPJPE — in progress PA-MPJPE — in progress Global MPJPE — in progress PCK (150mm) — in progress Accel / jitter — in progress

04.5 — Research direction

Noise as signal: the error between video and mocap is itself training data.

Every validation row above measures the same thing: the residual between what our video pipeline reconstructs and what a mocap rig recorded. The standard move is to treat that residual as a defect to be minimized. We take the opposite approach: this residual is a structured, task-conditioned noise distribution, and injecting it during policy training acts as a regularizer that improves sim-to-real transfer — in the same family as domain randomization and noisy-student training.

Research — under internal validation

A — The residual is structured, not white

Same motion, two measurements. The gap ε(t) between them tracks pose, contact, and occlusion — it is not i.i.d. Gaussian.

B — Injected noise helps, then hurts

Sweep the injection scale λ; transfer performance is non-monotonic. The optimum λ* is what we are estimating per task class.

q̃(t) = q_m(t) + λ·ε′(t), ε′ ~ D̂_task Train the policy on mocap-clean trajectories perturbed by noise sampled from the empirical residual distribution D̂ — fit from paired (mocap, video) recordings of the same motions — rather than from a synthetic Gaussian. λ follows a curriculum and is annealed against validation transfer.

01 — PAIR

Record the same motions twice — once through a mocap rig, once through the video pipeline — so every frame has a measured residual, aligned in time.

02 — FIT

Fit the empirical residual distribution D̂ per joint and per task class, preserving its temporal correlation — block-bootstrap sampling, not per-frame i.i.d. draws.

03 — INJECT

Inject ε′ ~ D̂ during policy training at curriculum scale λ, and select λ* on held-out sim-to-real transfer — the inverted-U in panel B.

Why this works: domain randomization (OpenAI, 2017–19) and noisy-student training (Xie et al., 2020) both established that the right kind of injected uncertainty regularizes a model into robustness. Our approach is narrower and directly measurable: the residual between video-based pose estimation and mocap is a better-matched noise model for humanoid policies than synthetic perturbations, precisely because it is generated by the same estimator the deployed system will see. Results are being validated internally, and the method plugs into Anchor as one more tunable element of the preprocessing parameter set θ_proc. A sampler skeleton is public in the pipeline SDK (anchor.noise.residual).

06 — Background

Built on published research, our own retargeting and acquisition layer, and prior video infrastructure work.

The scene-and-human reconstruction underlying this pipeline builds on VideoMimic, a published real-to-sim research method that demonstrated monocular video could be reconstructed into a metrically consistent human-and-environment scene. We use that method for reconstruction only. Everything that turns a reconstruction into usable robot data — the acquisition pipeline that gets arbitrary video into the system reliably, the retargeting layer that maps a single reconstruction onto multiple target humanoids' own joint structures and dynamics, and the feature-extraction stage that produces structured, queryable output — is our own work, built specifically for this purpose.

That meant solving problems a research paper doesn't have to: environment setup that survives a clean rebuild, dependency conflicts between components built years apart, GPU memory ceilings that show up only at certain clip lengths, a multi-robot retargeting layer where the original method only ever targeted one, and a deployment-facing dashboard that a non-technical stakeholder can actually run.

Research foundation

The reconstruction stage builds on VideoMimic (CoRL 2025). We did not develop the underlying reconstruction algorithm — our contribution is the data-acquisition pipeline around it, the multi-robot retargeting layer downstream of it, the feature-extraction work on top of it, and the production validation that makes it deployable.

Prior experience

Our team has spent over a year operating video and image generation infrastructure in production — including AI-driven product photography and video generation pipelines processing real customer traffic at scale. That background, handling video pipelines, GPU infrastructure, and generation-control systems, directly informed how this pipeline was built and hardened.

Acquisition lineage

We run our own mining pipeline, with a detection and captioning stack built on current-generation multimodal models, plus the quality and provenance classification layer that decides what's worth keeping — the part of Anchor's Trust and Gates stages that are coming soon.

Building Anchor: a trust-graded data refinery for humanoid robot learning.