Exercise: Humanoid training for Ultimate Bots

I recently won the Ghost Trial hackathon — teaching a Unitree G1 humanoid Scorpion's spear-throw-into-uppercut from Mortal Kombat, all the way from the 1992 arcade footage to a policy that performs the combo under physics. The six-minute video is the whole story:

The process

Six stages: start from what the game shipped, work back to how it was made, make our own version of that, and refine the target until physics will accept it.

The game ▶ 0:07

Mortal Kombat, 1992. The brief we set ourselves: make a real humanoid perform Scorpion's signature — the spear throw into the uppercut — recognisably, not approximately. A first attempt lifted poses straight off the arcade sprites; it proved the pipeline, but the frames are too few and too stylised to train from.

The OG footage ▶ 0:50

MK1's sprites were photographs of a real martial artist, and the behind-the-scenes tapes of that shoot survive. That footage is the move's ground truth — but it is low-resolution, interlaced, and copyrighted, so it became the reference to imitate rather than the data to train on.

Guided mocap ▶ 1:08

We commissioned a performer to be Scorpion: a written capture brief, a shoot, a frame-by-frame review of every take against the 1992 footage, notes back, and a reshoot at revised camera angles. The delivered video is our own — clean, 4K, and directed to match the original beat for beat.

Mapping, without physics ▶ 2:23

Video to 2D keypoints, keypoints to a 3D skeleton (1:59), and the skeleton retargeted onto the G1's 29 joints. This is pure kinematics — the robot replays the shapes with no balance, no contacts, no torque limits. It looks right, and it would fall over instantly.

Refining the target ▶ 2:49

A raw retarget is not a trainable target. Take by take (A2→A11) the motion was grounded to the floor, cleared of self-collisions, retimed to natural speed, its stance stabilised and smoothed — each edit measured against joint limits, torque and balance, so that what we ask the controller for is something a G1 can physically do.

Under physics ▶ 3:57

NVIDIA's SONIC whole-body controller was fine-tuned on the refined target in Isaac Lab — thousands of simulated G1s attempting the combo in parallel until the policy tracks it with balance held and both feet planted where the move says they should be. The payoff shot, on The Pit, is at 4:58.

Two GPUs, two targets — Nebius as the second lane

The last open question was how hard to push the target: the same move exists as multiple candidate targets with differing parameters — the smoothed baseline (A9), a 1.2× speed variant, a deeper-crouch variant, and A11 with both. Each is a separate training run, and we had exactly one GPU in house.

So the runs went in parallel: the reference target trained on the desk (RTX 5080 laptop), the pushed target trained on a Nebius H100 (gpu-h100-sxm, 4096 environments, 500 iterations), and the results were compared against the target to keep the fastest variant that still tracks. Simply sourcing the second lane of compute from Nebius halved the iteration loop — every experiment on the pushed target came back while the local run was still busy. One practical wrinkle: the 40.5 GB training image could never be uploaded over our 8.7 Mbps uplink, so the job assembles its own environment in the datacentre from public images and pulls only the two private pieces — a 469 MB checkpoint and a 119 KB motion file — from object storage.

The rig. Every take trained twice at once — reference speed locally, pushed speed on Nebius — then compared to find how fast the move can be thrown.

The stack

Stage	Tools & data
Motion sources	Our commissioned capture (two shoots, 4K); Adobe Mixamo clips for the practice move; the BONES-SEED library on Hugging Face — 142,220 motions searched for the hero moves (none existed, which settled the decision to capture our own), kept for control clips and as the 29-DoF CSV format everything speaks.
Video → skeleton	NVlabs GEM-X — monocular video to 77-joint SOMA pose, with its bundled 2D keypoint detector — plus its soma-retargeter straight onto the G1’s 29 joints. The earlier practice-move pipeline used GVHMR + GMR (FBX → BVH → G1).
Feasibility checks	MuJoCo — every take measured for foot contact, joint-limit violations, self-collisions and inverse-dynamics torque before any GPU time was spent.
Training	NVIDIA SONIC (GR00T-WholeBodyControl) fine-tuned from the release checkpoint — PPO, 4096 parallel environments in Isaac Lab 2.3.2 (the stock nvcr.io/nvidia/isaac-lab:2.3.2 image, in Docker).
Compute	One RTX 5080 laptop on the desk + Nebius H100 AI Jobs with object storage for checkpoints — the two lanes of the rig above.
Authoring & media	Blender — reviewing the solved human motion, authoring directly in G1 joint space, and the entire video edit including The Pit render. Policy shipped as ONNX.
Published	Model: DaveRc/ghosttrial-g1-scorpion · dataset: DaveRc/ghosttrial-g1-scorpion-motion (CC BY 4.0, the motion only — never the performer’s video) · code: SpiRaiL/GhostTrial-public.

Ultimate Bots · Ghost Trial hackathon 2026 · full video youtu.be/1Oo8h655VMM