Liangyu Chen    陈亮宇

Hi! 👋 I am a research engineer at MMLab, Nanyang Technological University, advised by Prof. Ziwei Liu.

My research interest lies in generalizable and data-efficient machine learning frameworks for multimodal reasoning 🤔. I develop data collection pipelines, representations, and models for interactive or assistive AI systems.

Previously, I worked with Prof. Alan Yuille and Dr. Zongwei Zhou at Johns Hopkins University, on self-supervised learning and human-in-the-loop machine learning. I obtained my Bachelor’s degree in Information Engineering & Media from Nanyang Technological University.

Email  /  Google Scholar  /  Semantic Scholar  /  Github  /  Linkedin

profile photo
Applications, Datasets and Benchmarks
  • MIMIC-IT & Otter: Multi-Modal In-Context Learning Model with Instuction Tuning. [Dataset] [Demo]
  • CSVAL: Benchmarking Initial Queries in Vision Active Learning, MIDL 2023, NeurIPS'22 workshop.
  • OpenPSG: Dataset and Benchmark for Panoptic Scene Graph Generation (PSG), ECCV'22. [🤗 Hugging Face Demo]
  • Baconian: Model-based Reinforcement Learning Framework. [Demo]

  • Publications

    Benchmarking and Analyzing Generative Data for Visual Recognition
    Bo Li, Haotian Liu, Liangyu Chen, Yong Jae Lee, Chunyuan Li, Ziwei Liu
    arXiv preprint, 2023


    MIMIC-IT: Multi-Modal In-Context Instruction Tuning
    Bo Li*, Yuanhan Zhang*, Liangyu Chen*, Jinghao Wang*, Fanyi Pu*, Jingkang Yang, Chunyuan Li, Ziwei Liu
    arXiv preprint, 2023


    Otter: A Multi-Modal Model with In-Context Instruction Tuning
    Bo Li*, Yuanhan Zhang*, Liangyu Chen*, Jinghao Wang*, Jingkang Yang, Ziwei Liu
    arXiv preprint, 2023


    Large Language Models are Visual Reasoning Coordinators
    Liangyu Chen*, Bo Li*, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, Ziwei Liu
    Neural Information Processing Systems (NeurIPS), 2023
    ICLR Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023


    Panoptic Video Scene Graph Generation
    Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Wayne Zhang, Kaiyang Zhou, Chen Change Loy, Ziwei Liu
    Computer Vision and Pattern Recognition (CVPR), 2023


    Making Your First Choice: To Address Cold Start Problem in Vision Active Learning
    Liangyu Chen, Yutong Bai, Siyu Huang, Yongyi Lu, Bihan Wen, Alan Yuille, Zongwei Zhou
    Medical Imaging with Deep Learning (MIDL), 2023
    NeurIPS Workshop on Human in the Loop Learning, 2022


    Automatic Calcification Morphology and Distribution Classification for Breast Mammograms with Multi-task Graph Convolutional Neural Network
    Hao Du, Melissa Min-Szu Yao, Siqi Liu, Liangyu Chen, Wing P. Chan, Mengling Feng
    Journal of Biomedical and Health Informatics (JBHI), 2023


    Baconian: A Unified Open-source Framework for Model-Based Reinforcement Learning
    Linsen Dong, Guanyu Gao, Xinyi Zhang, Liangyu Chen, Yonggang Wen
    arXiv preprint, 2020


    Service
  • Reviewer for The Visual Computer, IET Computer Vision, NeurIPS 2023, ICCV CVAMD 2023, CVPR CVinW 2023, ICML IMLH 2023/22.


  • I host virtual office hours for anyone who wants to share thoughts on CV/ML research, reading (either scientific or not), or any other topics of interest. Please schedule via my calendar.

    Last update: Oct 2023      Template