Reseach Agenda:
We target top-tier research at the intersection of computer vision, computer graphics and machine learning. Our research mission is to capture, perceive and understand the human-centric dynamic and static scenes in the complex real world. Our goal is to digitalize humans, objects and events, and eventually to enable realistic and immersive tele-presence in virtual reality and augmented reality.
(# for the corresponding author, * for equal contributors)
2023
DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance
Longwen Zhang,
Qiwei Qiu, Qixuan Zhang, Cheng Shi, Wei Yang,
Ye Shi,
Sibei Yang#,
Lan Xu# and
Jingyi Yu#.
ACM Transactions on Graphics (Proc. of SIGGRAPH), 2023.
HACK: Learning a Parametric Head and Neck Model for High-fidelity Animation
Longwen Zhang,
Zijun Zhao Xinzhou Cong, Qixuan Zhang,
Shuqi Gu, Yuchong Gao, Rui Zheng, Wei Yang,
Lan Xu# and
Jingyi Yu#.
ACM Transactions on Graphics (Proc. of SIGGRAPH), 2023.
LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse Inertial and LiDAR Sensors
Yiming Ren*, Chengfeng Zhao*,
Yannan He, Peishan Cong, Han Liang,
Jingyi Yu,
Lan Xu# and
Yuexin Ma#.
IEEE Transactions on Visualization and Computer Graphics (Proc. IEEE VR), Accepted, 2023.
Free-bloom: Zero-shot text-to-video generator with llm director and ldm animator
Hanzhuo Huang,
Yufan Feng,
Cheng Shi,
Lan Xu,
Jingyi Yu and
Sibei Yang.
Neural Information Processing Systems (NeurIPS), 2023.
NeReF: Neural Refractive Field for Fluid Surface Reconstruction and Rendering
Ziyu Wang,
Wei Yang,
Junming Cao,
Qiang Hu,
Lan Xu,
Junqing Yu and
Jingyi Yu.
International Conference on Computational Photography (ICCP), 2023.
StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset
Chaofan Huo,
Ye Shi,
Yuexin Ma,
Lan Xu,
Jingyi Yu and
Jingya Wang.
International Joint Conferences on Artificial Intelligence Organization (IJCAI), 2023.
Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos
Liao Wang,
Qiang Hu,
Qihan He,
Ziyu Wang,
Jingyi Yu,
Tinne Tuytelaars,
Lan Xu# and
Minye Wu#.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
HumanGen: Generating Human Radiance Fields with Explicit Priors
Suyi Jiang,
Haoran Jiang,
Ziyu Wang,
Haimin Luo,
Wenzheng Chen and
Lan Xu.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Instant-NVR: Instant Neural Volumetric Rendering for Human-object Interactions from RGBD Stream
Yuheng Jiang*,
Kaixin Yao*,
Zhuo Su,
Zhehao Shen,
Haimin Luo and
Lan Xu.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions
Juze Zhang*,
Haimin Luo*,
Hongdi Yang,
Xinru Xu,
Qianyang Wu,
Ye Shi,
Jingyi Yu,
Lan Xu# and
Jingya Wang#.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Relightable Neural Human Assets from Multi-view Gradient Illuminations
Taotao Zhou*,
Kai He*,
Di Wu*,
Teng Xu,
Qixuan Zhang,
Kuixiang Shao,
Wenzheng Chen,
Lan Xu# and
Jingyi Yu#.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
SLOPER4D: A Scene-Aware Dataset for Global 4D Human Pose Estimation in Urban Environme
Yudi Dai,
Yitai Lin,
Xiping Lin,
Chenglu Wen,
Lan Xu,
Hongwei Yi,
Siqi Shen,
Yuexin Ma and
Cheng Wang.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions
Ming Yan,
Xin Wang,
Yudi Dai,
Siqi Shen,
Chenglu Wen,
Lan Xu,
Yuexin Ma and
Cheng Wang.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions
Han Liang,
Yannan He,
Chengfeng Zhao,
Mutian Li,
Jingya Wang,
Jingyi Yu and
Lan Xu.
Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI), 2023.
IKOL: Inverse kinematics optimization layer for 3D human pose and shape estimation via Gauss-Newton differentiation
Juze Zhang,
Ye Shi,
Yuexin Ma,
Lan Xu,
Jingyi Yu and
Jingya Wang.
Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI), 2023.
Weakly Supervised 3D Multi-person Pose Estimation for Large-scale Scenes based on Monocular Camera and Single LiDAR
Peishan Cong,
Yiteng Xu,
Yiming Ren,
Juze Zhang,
Lan Xu,
Jingya Wang,
Jingyi Yu and
Yuexin Ma.
Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI), 2023.
Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset
Youjia Wang,
Kai He,
Taotao Zhou,
Kaixin Yao,
Nianyi Li,
Lan Xu and
Jingyi Yu.
International Journal of Computer Vision (IJCV), Accepted, 2023.
2022
RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions
Zhuo Su*,
Lan Xu*,
Dawei Zhong*,
Zhong Li,
Fan Deng,
Shuxue Quan and
Lu Fang.
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Accepted, 2022.
Human Performance Modeling and Rendering via Neural Animated Mesh
Fuqiang Zhao,
Yuheng Jiang,
Kaixin Yao,
Jiakai Zhang,
Liao Wang,
Haizhao Dai,
Yuhui Zhong,
Yingliang Zhang,
Minye Wu,
Lan Xu# and
Jingyi Yu#.
ACM Transactions on Graphics (Proc. of SIGGRAPH ASIA), 2022.
Video-driven Neural Physically-based Facial Asset for Production
Longwen Zhang,
Chuxiao Zeng,
Qixuan Zhang,
Hongyang Lin,
Ruixiang Cao,
Wei Yang,
Lan Xu# and
Jingyi Yu#.
ACM Transactions on Graphics (Proc. of SIGGRAPH ASIA), 2022.
SCULPTOR: Skeleton-Consistent Face Creation Using a Learned Parametric Generator
Zesong Qiu,
Qixuan Zhang,
Dongming He,
Yuwei Li,
Longwen Zhang,
Yinghao Zhang,
Jingya Wang,
Lan Xu,
Xudong Wang,
Yuyao Zhang and
Jingyi Yu.
ACM Transactions on Graphics (Proc. of SIGGRAPH ASIA), 2022.
Mutual Adaptive Reasoning for Monocular 3D Multi-Person Pose Estimation
Juze Zhang,
Jingya Wang,
Ye Shi,
Fei Gao,
Lan Xu and
Jingyi Yu.
Proceedings of the 30th ACM international conference on Multimedia (ACMMM), 2022.
NIMBLE: a Non-rigid Hand Model with Bones and Muscles
Yuwei Li,
Longwen Zhang,
Zesong Qiu
Yingwenqi Jiang,
Nianyi Li,
Yuexin Ma,
Yuyao Zhang,
Lan Xu and
Jingyi Yu.
ACM Transactions on Graphics (Proc. of SIGGRAPH), 2022.
Artemis: Articulated Neural Pets with Appearance and Motion Synthesis
Haimin Luo,
Teng Xu,
Yuheng Jiang,
Chenglin Zhou,
Qiwei Qiu,
Yingliang Zhang,
Wei Yang,
Lan Xu and
Jingyi Yu.
ACM Transactions on Graphics (Proc. of SIGGRAPH), 2022.
Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-Time
Liao Wang,
Jiakai Zhang,
Xinhang Liu,
Fuqiang Zhao,
Yanshun Zhang,
Yingliang Zhang,
Minye Wu,
Jingyi Yu and
Lan Xu.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. Oral
HumanNeRF: Efficiently Generated Human Radiance Field From Sparse Inputs
Fuqiang Zhao,
Wei Yang,
Jiakai Zhang,
Pei Lin,
Yingliang Zhang,
Jingyi Yu and
Lan Xu.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
NeuralHOFusion: Neural Volumetric Rendering Under Human-Object Interactions
Yuheng Jiang,
Suyi Jiang,
Guoxing Sun,
Zhuo Su,
Kaiwen Guo,
Minye Wu,
Jingyi Yu and
Lan Xu.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
HSC4D: Human-Centered 4D Scene Capture in Large-Scale Indoor-Outdoor Space Using Wearable IMUs and LiDAR
Yudi Dai,
Yitai Lin,
Chenglu Wen,
Siqi Shen,
Lan Xu,
Jingyi Yu,
Yuexin Ma and
Cheng Wang.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
LiDARCap: Long-Range Marker-Less 3D Human Motion Capture With LiDAR Point Clouds
Jialian Li,
Jingyi Zhang,
Zhiyong Wang,
Siqi Shen,
Chenglu Wen,
Yuexin Ma,
Lan Xu,
Jingyi Yu and
Cheng Wang.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenes
Peishan Cong,
Xinge Zhu,
Feng Qiao,
Yiming Ren,
Xidong Peng,
Yuenan Hou,
Lan Xu,
Ruigang Yang,
Dinesh Manocha and
Yuexin Ma.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Anisotropic Fourier Features for Neural Image-Based Rendering and Relighting
Huangjie Yu,
Anpei Chen,
Xin Chen,
Lan Xu,
Ziyu Shao and
Jingyi Yu.
Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI), 2022. Oral
SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
Xin Chen,
Anqi Pang,
Wei Yang,
Yuexin Ma,
Lan Xu# and
Jingyi Yu#.
International Journal of Computer Vision (IJCV), Accepted, 2022.
2021
TightCap: 3D Human Shape Capture with Clothing Tightness
Xin Chen*,
Anqi Pang*,
Wei Yang,
Yuexin Ma,
Lan Xu# and
Jingyi Yu#.
ACM Transactions of Graphics (TOG), Accepted, 2021.
Neural Video Portrait Relighting in Real-time via Consistency Modeling
Longwen Zhang,
Qixuan Zhang,
Minye Wu,
Jingyi Yu and
Lan Xu.
International Conference on Computer Vision (ICCV), 2021.
GNeRF: GAN-based Neural Radiance Field without Posed Camera
Quan Meng,
Anpei Chen,
Haimin Luo,
Minye Wu,
Hao Su,
Lan Xu,
Xumin He and
Jingyi Yu.
International Conference on Computer Vision (ICCV), 2021. Oral
iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering
Liao Wang,
Ziyu Wang,
Pei Lin,
Yuheng Jiang,
Xin Suo,
Minye Wu,
Lan Xu and
Jingyi Yu.
Proceedings of the 29th ACM international conference on Multimedia (ACMMM), 2021. Oral
Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions
Guoxing Sun,
Xin Chen,
Yizhang Chen,
Anqi Pang,
Pei Lin,
Yuheng Jiang,
Lan Xu,
Jingyi Yu and
Jingya Wang.
Proceedings of the 29th ACM international conference on Multimedia (ACMMM), 2021. Oral
Towards Controllable and Photorealistic Region-wise Image Manipulation
Ansheng You,
Chenglin Zhou,
Qixuan Zhang and
Lan Xu.
Proceedings of the 29th ACM international conference on Multimedia (ACMMM), 2021.
IREM: High-Resolution Magnetic Resonance (MR) Image Reconstruction via Implicit Neural Representation
Qing Wu,
Yuwei Li,
Lan Xu,
Ruimin Feng,
Hongjiang Wei,
Qing Yang,
Boliang Yu,
Xiaozhao Liu,
Qing Yang,
Jingyi Yu and
Yuyao Zhang.
International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2021.
PIANO: A Parametric Hand Bone Model from Magnetic Resonance Imaging
Yuwei Li,
Minye Wu,
Yuyao Zhang,
Lan Xu and
Jingyi Yu.
International Joint Conferences on Artificial Intelligence Organization (IJCAI), 2021.
Few-shot Neural Human Performance Rendering from Sparse RGBD Videos
Anqi Pang,
Xin Chen,
Haimin Luo,
Minye Wu,
Jingyi Yu and
Lan Xu.
International Joint Conferences on Artificial Intelligence Organization (IJCAI), 2021.
Editable free-viewpoint video using a layered neural representation
Jiakai Zhang,
Xinhang Liu,
Xinyi Ye,
Fuqiang Zhao,
Yanshun Zhang,
Minye Wu,
Yingliang Zhang,
Lan Xu# and
Jingyi Yu#.
ACM Transactions on Graphics (Proc. of SIGGRAPH), 2021.
ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References
Yannan He,
Anqi Pang,
Xin Chen,
Han Liang,
Yuexin Ma,
Minye Wu and
Lan Xu.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Oral
NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras
Xin Suo,
Yuheng Jiang,
Pei Lin,
Yingliang Zhang,
Minye Wu,
Kaiwen Guo, and
Lan Xu.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging
Ziyu Wang,
Liao Wang,
Fuqiang Zhao,
Minye Wu,
Lan Xu and
Jingyi Yu.
International Conference on Computational Photography (ICCP), 2021.
Convolutional Neural Opacity Radiance Fields
Haimin Luo,
Anpei Chen,
Qixuan Zhang,
Bai Pang,
Minye Wu,
Lan Xu and
Jingyi Yu.
International Conference on Computational Photography (ICCP), 2021.
Boosting Single Image Super-Resolution Learnt From Implicit Multi-Image Prior
Dingjian Jin,
Mengqi Ji,
Lan Xu,nbsp
Gaochang Wu,
Liejun Wang, and
Fang Lu.
IEEE Transactions on Image Processing (TIP), Accepted, 2021.
2020
BuildingFusion: Semantic-aware Structural Building-scale 3D Reconstruction
Tian Zheng,
Guoqing Zhang,
Lei Han,
Lan Xu and
Lu Fang.
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Accepted, 2020.
Neural3D: Light-weight Neural Portrait Scanning via Context-aware Correspondence Learning
Xin Suo,
Minye Wu,
Yanshun Zhang,
Yingliang Zhang,
Qiang Hu,
Lan Xu and
Jingyi Yu.
Proceedings of the 28th ACM international conference on Multimedia (ACMMM), 2020.
RobustFusion: Human Volumetric Capture with Data-driven Visual Cues using a RGBD Camera
Zhuo Su*,
Lan Xu*,
Zerong Zheng,
Tao Yu,
Yebin Liu and
Lu Fang.
European Conference on Computer Vision and Pattern Recognition (ECCV), 2020. Spotlight
EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera
Lan Xu,
Weipeng Xu,
Vladislav Golyanik,
Marc Habermann,
Lu Fang and
Christian Theobalt.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. Oral
OccuSeg: Occupancy-aware 3D Instance Segmentation
Lei Han,
Tian Zheng,
Lan Xu and
Lu Fang.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Multiscale-VR: Multiscale Gigapixel 3D Panoramic Videography for Virtual Reality
Jianing Zhang, Tianyi Zhu, Anke Zhang, Zihan Wang,
Xiaoyun Yuan, Sebastian Beetschen,
Lan Xu, Xing Lin,
Lu Fang and
Qionghai Dai.
International Conference on Computational Photography (ICCP), 2020.
Live Semantic 3D Perception for Immersive Augmented Reality
Lei Han,
Tian Zheng, Yinheng Zhu,
Lan Xu and
Lu Fang.
IEEE Transactions on Visualization and Computer Graphics (Proc. IEEE VR), Accepted, 2020.
2019
UnstructuredFusion: Realtime 4D Geometry and Texture Reconstruction using Commercial RGBD Cameras
Lan Xu, Zhuo Su,
Lei Han,
Tao Yu,
Yebin Liu and
Lu Fang.
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Accepted, 2019.
FlyFusion: Realtime Dynamic Scene Reconstruction Using a Flying Depth Camera
Lan Xu, Wei Cheng,
Kaiwen Guo,
Lei Han,
Yebin Liu and
Lu Fang.
IEEE Transactions on Visualization and Computer Graphics (TVCG), Accepted, 2019.
Real-Time Global Registration for Globally Consistent RGB-D SLAM
Lei Han,
Lan Xu,
Dmytro Bobkov,
Eckehard Steinbach and
Lu Fang.
IEEE Transactions on Robotics (TRO), Accepted, 2019.
2018
FlyCap: Markerless Motion Capture Using Multiple Autonomous Flying Cameras
Lan Xu,
Yebin Liu Wei Cheng,
Kaiwen Guo, Guyue Zhou,
Qionghai Dai and
Lu Fang.
IEEE Transactions on Visualization and Computer Graphics (TVCG), Accepted, 2018.
iHuman3D: Intelligent Human Body 3D Reconstruction using a Single Flying Camera
Wei Cheng*,
Lan Xu*,
Lei Han, Yuanfang Guo and
Lu Fang.
Proceedings of the 26th ACM international conference on Multimedia (ACMMM), 2018. Oral
Beyond SIFT using binary features in loop closure detection
Lei Han, Guyue Zhou,
Lan Xu, and
Lu Fang.