Rex Cheng

Ho Kei (Rex) Cheng

I am a Ph.D. candidate at the University of Illinois Urbana-Champaign, advised by Alexander Schwing. Before that, I was at The Hong Kong University of Science and Technology, advised by Yu-Wing Tai and Chi Keung Tang.

I work on visual understanding, focusing on videos. My past research includes video object tracking, segmentation, and multimodal-conditioned video-to-audio synthesis. I have interned at Adobe Research, Kaiber, and Sony AI. I will intern at Meta Fundamental AI Research (FAIR) this summer.

[GitHub] | [Google Scholar] | [CV]

Research (hover over videos to play)

The Curse of Conditions: Analyzing and Improving Optimal Transport for Conditional Flow-Based Generation

Ho Kei Cheng, Alexander Schwing.

arXiv 2025

Project page / code / arXiv

Provides straighter flows through condition-aware coupling of samples from the prior and data distributions, without the test-time degradation induced by naïve optimal transport.

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji.

CVPR 2025

Project page / code / arXiv / Space demo / Replicate

Generates high-quality synchronized audio from video or text inputs, with an architecture that enables training on data from multiple sources even when some modalities are missing. Click here to watch a fun video!

Putting the Object Back into Video Object Segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, Alexander Schwing.

CVPR 2024 Highlight

Project page / code / arXiv

Uses an object transformer to combine pixel-level and object-level features for efficient and robust video object segmentation in challenging scenarios. Used by iMotions and Annolid.

Tracking Anything with Decoupled Video Segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, Joon-Young Lee.

ICCV 2023

Project page / code / arXiv

Achieves open-world video segmentation by combining universal image segmentation with temporal propagation. Easy to extend.

XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

Ho Kei Cheng, Alexander Schwing.

ECCV 2022

Project page / code / arXiv

Approaches video object segmentation from a memory perspective with a pipeline that effectively models both short-term and long-term dependencies. Used by supervisely and Track-Anything.

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Ho Kei Cheng, Yu-Wing Tai, Chi Keung Tang.

NeurIPS 2021

Project page / code / arXiv

A simple yet effective method to model pixel correspondences between frames. Used by Trioscope and BURST.

Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion

Ho Kei Cheng, Yu-Wing Tai, Chi Keung Tang.

CVPR 2021

Project page / code / arXiv

Decouples interactive video segmentation into two components: single-frame interaction and temporal propagation, demonstrating significantly improved performance. Used by Sieve.

CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement

Ho Kei Cheng*, Jihoon Chung*, Yu-Wing Tai, Chi Keung Tang.

CVPR 2020

Project page / code / arXiv / pypi

An iterative refinement network that achieves high-quality 4K+ segmentation using only low-resolution training data (less than 500 pixels per side).

Invited Talks

Object-Level Reasoning in Video Object Segmentation and Its Multimodal Applications @ Twelve Labs, September 2024
Segmenting Videos in the Open World @ IBM Zurich, Accelerated Discovery, September 2023
Large-Scale Decoupled Video Segmentation @ Apple, September 2023

Tools

nitrous-ema Low overhead post-hoc EMA for PyTorch.

vos-benchmark Fast and simple benchmarking for video object segmentation.

shared-memory-tensor-dataset A simple demo for sharing an in-memory dataset among DDP processes.

Professional Activities

Reviewed for: CVPR, ICCV, ECCV, NeurIPS, ICLR, ICML, AAAI, IEEE TIP, IEEE PR, IEEE TPAMI, IEEE TCSVT.
Outstanding reviewer in ICML 2022.

Misc

I was a proud member of the HKUST Robotics Team. A short clip.
I am generally interested in artificial intelligence. I believe in AGIs and has high hope for their potential to transform human civilization for the better.
"Man is condemned to be free. Condemned, because he did not create himself, in other respect is free; because, once thrown into the world, he is responsible for everything he does."
Look at this cat in HKUST. Another picture. Or this cat.
"Ho Kei" (with the space) is my first name and "Cheng" is my last name. "Rex" is the commonly used "english name" that is not part of my legal name. This is common in Hong Kong.