Rex Cheng

Ho Kei (Rex) Cheng

I am a Ph.D. candidate at the University of Illinois Urbana-Champaign, advised by Alexander Schwing. Before that, I was at The Hong Kong University of Science and Technology, advised by Yu-Wing Tai and Chi Keung Tang.

I work on visual understanding, focusing on videos. My past research includes video object tracking, segmentation, and multimodal-conditioned video-to-audio synthesis. I have interned at Adobe Research (open-world video segmentation), Kaiber (videos diffusion models), and Sony AI (multimodal flow matching models).

[GitHub] | [Google Scholar] | [CV]

Research (hover over videos to play)

Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji.

arXiv 2024

Project page / code / arXiv / Space demo / Replicate

Generates high-quality synchronized audio from video or text inputs, with an architecture that enables training on data from multiple sources even when some modalities are missing.

Putting the Object Back into Video Object Segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, Alexander Schwing.

CVPR 2024 Highlight

Project page / code / arXiv

Uses an object transformer to combine pixel-level and object-level features for efficient and robust video object segmentation in challenging scenarios. Used by iMotions and Annolid.

Tracking Anything with Decoupled Video Segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, Joon-Young Lee.

ICCV 2023

Project page / code / arXiv

Achieves open-world video segmentation by combining universal image segmentation with temporal propagation. Easy to extend.

XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

Ho Kei Cheng, Alexander Schwing.

ECCV 2022

Project page / code / arXiv

Approaches video object segmentation from a memory perspective with a pipeline that effectively models both short-term and long-term dependencies. Used by supervisely and Track-Anything.

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Ho Kei Cheng, Yu-Wing Tai, Chi Keung Tang.

NeurIPS 2021

Project page / code / arXiv

A simple yet effective method to model pixel correspondences between frames. Used by Trioscope and BURST.

Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion

Ho Kei Cheng, Yu-Wing Tai, Chi Keung Tang.

CVPR 2021

Project page / code / arXiv

Decouples interactive video segmentation into two components: single-frame interaction and temporal propagation, demonstrating significantly improved performance. Used by Sieve.

CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement

Ho Kei Cheng*, Jihoon Chung*, Yu-Wing Tai, Chi Keung Tang.

CVPR 2020

Project page / code / arXiv / pypi

An iterative refinement network that achieves high-quality 4K+ segmentation using only low-resolution training data (less than 500 pixels per side).

Invited Talks

Object-Level Reasoning in Video Object Segmentation and Its Multimodal Applications @ Twelve Labs, September 2024
Segmenting Videos in the Open World @ IBM Zurich, Accelerated Discovery, September 2023
Large-Scale Decoupled Video Segmentation @ Apple, September 2023

Tools

nitrous-ema Low overhead post-hoc EMA for PyTorch.

vos-benchmark Fast and simple benchmarking for video object segmentation.

shared-memory-tensor-dataset A simple demo for sharing an in-memory dataset among DDP processes.

Professional Activities

Reviewer: CVPR, ICCV, ECCV, NeurIPS, ICLR, ICML, AAAI, IEEE TIP, IEEE PR, IEEE TPAMI, IEEE TCSVT.
Outstanding reviewer in ICML 2022.

Misc

I was a proud member of the HKUST Robotics Team. A short clip.
I am generally interested in artificial intelligence. I believe in AGIs and has high hope for their potential to transform human civilization for the better.
"Man is condemned to be free. Condemned, because he did not create himself, in other respect is free; because, once thrown into the world, he is responsible for everything he does."
Look at this cat in HKUST. Another picture. Or this cat.
"Ho Kei" (with the space) is my first name and "Cheng" is my last name. "Rex" is the commonly used "english name" that is not part of my legal name. This is common in Hong Kong.