User Workflow Guide

User Workflow Guide#

The section provides the complete user workflow in UrbanVerse, covering both custom scene generation and the use of built-in simulation scenes.

User-side pipeline for UrbanVerse

(A) Generate Your Own Simulation Scenes using UrbanVerse-Gen Automatica Pipeline#

  1. Input Preparation:

  • Accepts YouTube clips or URL, phone-recorded city-walk videos, pre-existing datasets, or folders of RGB frames (see Real-to-Sim Scene Generation with UrbanVerse-Gen for more details).

  • All inputs are normalized into sequential images for scene distillation.

  1. Run UrbanVerse-Gen Pipeline:

  • Generate your own simulation scenes using UrbanVerse-Gen APIs automatically (see Real-to-Sim Scene Generation with UrbanVerse-Gen for more details).

  • Extracts camera intrinsics/poses, metric depth, 3D instance point clouds, instance masks, and 3D boxes, producing a unified distilled scene graph (objects, ground, sky).

  • Materializes the scene graph by retrieving mutiple matched assets from UrbanVerse-100K (see Use UrbanVerse-100K with APIs for more details), including 3D GLB objects PBR ground materials, and HDRI sky maps.

  • Create and export fully interactivedigital-cousin simulation scenes in Isaac Sim with metric placement, physics, and lighting.

(B) Use Built-in Simulation UrbanVerse Repositories#

You can also directly use the built-in simulation UrbanVerse scene repositories we provided, including:

  1. UrbanVerse-160: A collection of 160 real-to-sim city scenes generated by UrbanVerse-Gen from city-tour YouTube videos acorss the world (See Use Built-in UrbanVerse Scenes for more details).

  2. CraftBench: A suite of 10 artist-crafted simulation scenes, useful for benchmarking policy robustness and generalization. (see Use Built-in CraftBench Scenes for more details).

(C) Leverage UrbanVerse Scenes for Downstream Applications#

Using either custom-generated scenes or built-in scene repositories, UrbanVerse supports:

  • Reinforcement Learning: Train navigation policies in UrbanVerse scenes using reinforcement learning (PPO) (see Reinforcement Learning in UrbanVerse for more details).

  • Expert Data Collection for Imitation Learning: Collect expert demonstrations from teleoperation (keyboard, joystick, gamepad, VR), and train behavior cloning policies (see Imitation Learning in UrbanVerse for more details).

  • Multimodal dataset collection: Collect offline multimodal dataset (RGB, depth, normals, segmentation, poses, etc) from UrbanVerse scenes for training and evaluation (see Collecting Data in UrbanVerse for more details).

  • Closed-loop evaluation: Evaluate the performance of the trained policies in UrbanVerse scenes (see Test Your Robots on CraftBench for more details).

  • Zero-shot Sim2Real deployment: Deploy the trained policies on real robot platforms (see Real-world Deployment: Unitree Go2 Quadruped Example for more details).