4DAnyone: Turn Any Phone Video Into a Full 3D Digital Twin

By Prahlad Menon Published 2026-09-01 3 min read

Film someone on your phone. Get views from angles you never captured. Build a realistic digital twin.

4DAnyone from Ant Research does exactly this — it takes a standard monocular video and predicts what the person looks like from every possible angle, including views from above, behind, and all around them. No camera rig. No studio. Just your phone.

Repo: github.com/ant-research/4danyone
Paper: arXiv:2608.20335 (SIGGRAPH Asia 2026)

What It Does

You give it one video. It gives you 24+ synchronized videos from different angles:

Input: Single phone video (front-facing)
Output: 6-48 views at configurable angles (360° coverage)

These multi-view outputs feed directly into 4D Gaussian Splatting (4DGS) reconstruction — meaning you can build a fully explorable, photorealistic 3D model of a person in motion.

The Problem It Solves

Building a digital twin of a person traditionally requires:

Multi-camera rigs (expensive)
Motion capture suits (cumbersome)
Controlled studio environments (limiting)

4DAnyone bypasses all of this. The AI learned what humans look like from every angle, so it can hallucinate the missing views from a single perspective — and do it consistently enough that standard 3D reconstruction tools accept the output.

How It Works

Motion Recovery — GVHMR extracts 3D human pose and motion from the input video
Pose Encoding — The motion is encoded into a latent representation
Multi-View Generation — A video diffusion model generates the target views
Consistency Matching — Lighting and motion stay consistent across all angles

The tricky part is making the generated angles look consistent — same lighting, same motion timing, same clothing dynamics. 4DAnyone handles this through careful conditioning on the recovered 3D pose.

Installation

git clone https://github.com/ant-research/4DAnyone.git
cd 4DAnyone
git submodule update --init third_party/GVHMR

conda create -n 4danyone python=3.11 -y
conda activate 4danyone
pip install -r requirements.txt

# Models download automatically on first use, or manually:
python scripts/download_smplx.py
python scripts/download_model.py
python scripts/download_example.py

Usage Examples

Basic 6-View (Quick Test)

python inference.py \
  --video_path "your_video.mp4" \
  --views_per_layer 6

Dense 24-View (4DGS Ready)

python inference.py \
  --video_path "your_video.mp4" \
  --views_per_layer 24

48-View with 3 Pitch Layers

For free-viewpoint rendering with coverage from below to above:

python inference.py \
  --video_path "your_video.mp4" \
  --views_per_layer 16 \
  --layer_pitches '[-10,15,35]'

Frontal 180° Arc

For when you only need front-facing angles:

python inference.py \
  --video_path "your_video.mp4" \
  --views_per_layer 12 \
  --layer_pitches '[0,30]' \
  --start_yaw -90 \
  --yaw_span 180

Input Requirements

Your source video should be:

720p or higher (1080p recommended)
9:16 portrait aspect ratio
Full-body or upper-body framing
At least 121 frames (about 4 seconds at 30fps)
Mild camera motion (don’t wave the phone around)

Hardware Requirements

GPU: 25.4 GB VRAM minimum (after recent optimizations)
Tested on: H20-3E, H200, RTX 5880 Ada, RTX A6000
Optional: FlashAttention-3 or SageAttention for faster inference

What You Get

data/
├── gvhmr/results/<clip>/     # Reusable motion recovery
└── 4danyone/<clip>/
    ├── metadata.json          # Settings, timings, resources
    ├── cameras.json           # N-camera rig definition
    ├── skeletons/00.mp4 ...   # Skeleton visualizations
    └── videos/
        ├── sparse/            # RCP proposals
        └── dense/00.mp4 ...   # Generated target views

The cameras.json file contains the exact camera positions, so you can feed everything into nerfstudio or other 3D reconstruction pipelines.

Performance

Recent optimizations (August 2026):

1.42× speedup for the complete 24-view pipeline
Peak VRAM reduced to 25.4 GB
Multi-GPU parallelization for pose/VAE stages

Roadmap includes Sol-Engine integration (expected 2× speedup) and model distillation (expected 5× speedup for few-step inference).

Use Cases

VFX and Film:

Create digital doubles from reference footage
Generate coverage angles for shots you couldn’t capture
Previs with realistic 3D characters

Gaming and VTubing:

Turn yourself into a 3D avatar from a phone video
Create custom character models
Motion capture without the suit

E-commerce:

360° product videos with human models
Virtual try-on with realistic body models

Research:

Human pose datasets with full coverage
Training data for other 3D vision models

Limitations

Single person per video (no crowds)
Person must be reasonably visible throughout
Very fast motion may cause artifacts
25+ GB VRAM requirement limits consumer hardware

Mixamo LLM Mocap — Video to rigged character animation
HyperFrames — Code-to-video rendering
GVHMR — The motion recovery backend 4DAnyone uses

Links: