Intelligent and Multimedia Science Laboratory

Sketch Generation and Applications
Image Editing and Synthesis
3D Pose Estimation and Motion Generation
Garment Modeling and Virtual Try-on
Multimedia Processing & 3D Rendering and Modeling

Sketch Generation and Applications

	Joint Stroke Tracing and Correspondence for 2D Animation Haoran Mo, Chengying Gao^* and Ruomei Wang Intro: We for the first time propose a joint stroke tracing and correspondence approach. Given consecutive raster keyframes along with a single vector image of the starting frame as a guidance, the approach generates vector drawings for the remaining keyframes while ensuring one-to-one stroke correspondence. Our framework trained on clean line drawings generalizes to rough sketches and the generated results can be imported into inbetweening systems to produce inbetween sequences. Hence, the method is compatible with standard 2D animation workflow. An adaptive spatial transformation module (ASTM) is introduced to handle non-rigid motions and stroke distortion. We collect a dataset for training, with 10k+ pairs of raster frames and their vector drawings with stroke correspondence. ACM Transactions on Graphics (Presented at SIGGRAPH 2024) (CCF-A) [Paper] [Code] [Project Page]
	Text-based Vector Sketch Editing with Image Editing Diffusion Prior Haoran Mo, Xusheng Lin, Chengying Gao^* and Ruomei Wang Intro: We present a framework for text-based vector sketch editing to improve the efficiency of graphic design. The key idea behind the approach is to transfer the prior information from raster-level diffusion models, especially those from image editing methods, into the vector sketch-oriented task. The framework presents three editing modes and allows iterative editing. To meet the editing requirement of modifying the intended parts only while avoiding changing the other strokes, we introduce a stroke-level local editing scheme that automatically produces an editing mask reflecting locally editable regions and modifies strokes within the regions only. International Conference on Multimedia & Expo (ICME, 2024) (CCF-B) [Paper] [Code]
	Video-Driven Sketch Animation via Cyclic Reconstruction Mechanism Zhuo Xie, Haoran Mo and Chengying Gao^* Intro: Considering the time-consuming manual workflow in 2D sketch animation production, we present an automatic solution by using videos as reference to animate the static sketch images. This includes motion extraction from the videos and injection into the sketches to produce animated sketch sequences in which appearance properties from the source sketches should be preserved. To reduce blurry artifact caused by complex motions and maintain stroke line continuity, we propose to incorporate inner masks of the sketches as an explicit guidance to indicate inner regions and ensure component integrality. Moreover, to bridge the domain gap between the video frames and the sketches when modelling the motions, we introduce a cyclic reconstruction mechanism to increase compatibility with different domains and improve motion consistency between the sketch animation and the driving video. International Conference on Multimedia & Expo (ICME, 2024) (CCF-B)
	Multi-instance Referring Image Segmentation of Scene Sketches based on Global Reference Mechanism Peng Ling, Haoran Mo and Chengying Gao^* Intro: We propose GRM-Net, a one-stage framework tailored for multi-instance referring image segmentation of scene sketches. We extract the language features from the expression and fuse it into a conventional instance segmentation pipeline for filtering out the undesired instances in a coarse-to-fine manner and keeping the matched ones. To model the relative arrangement of the objects and the relationship among them from a global view, we propose a global reference mechanism (GRM) to assign references to each detected candidate to identify its position. Pacific Graphics (PG 2022) *(oral) (CCF-B)** [Paper] [Code]
	Line Art Colorization Based on Explicit Region Segmentation Ruizhi Cao, Haoran Mo and Chengying Gao^* Intro: We introduce an explicit segmentation fusion mechanism to aid colorization frameworks in avoiding color bleeding artifacts. This mechanism is able to provide region segmentation information for the colorization process explicitly so that the colorization model can learn to avoid assigning the same color across regions with different semantics or inconsistent colors inside an individual region. The proposed mechanism is designed in a plug-and-play manner, so it can be applied to a diversity of line art colorization frameworks with various kinds of user guidances. Computer Graphics Forum (Pacific Graphics 2021) *(oral) (CCF-B)** [Paper] [Code]
	General Virtual Sketching Framework for Vector Line Art Haoran Mo, Edgar Simo-Serra, Chengying Gao^, Changqing Zou and Ruomei Wang Intro:* Vector line art plays an important role in graphic design, however, it is tedious to manually create. We introduce a general framework to produce line drawings from a wide variety of images, by learning a mapping from raster image space to vector image space. Our approach is based on a recurrent neural network that draws the lines one by one. A differentiable rasterization module allows for training with only supervised raster data. We use a dynamic window around a virtual pen while drawing lines, implemented with a proposed aligned cropping and differentiable pasting modules. Furthermore, we develop a stroke regularization loss that encourages the model to use fewer and longer strokes to simplify the resulting vector image. Ablation studies and comparisons with existing methods corroborate the efficiency of our approach which is able to generate visually better results in less computation time, while generalizing better to a diversity of images and applications. ACM Transactions on Graphics (SIGGRAPH 2021, Journal track) *(oral) (CCF-A)** [Paper] [Code] [Project Page]
	SketchyCOCO: Image Generation from Freehand Scene Sketches Chengying Gao, Qi Liu, Qi Xu, Jianzhuang Liu, Limin Wang, Changqing Zou* Intro: We introduce the first method for automatic image generation from scene-level freehand sketches. Our model allows for controllable image generation by specifying the synthesis goal via freehand sketches. The key contribution is an attribute vector bridged generative adversarial network called edgeGAN which supports high visual-quality image content generation without using freehand sketches as training data. We build a large-scale composite dataset called SketchyCOCO to comprehensively evaluate our solution. We validate our approach on the task of both objectlevel and scene-level image generation on SketchyCOCO. We demonstrate the method’s capacity to generate realistic complex scene-level images from a variety of freehand sketches by quantitative, qualitative results, and ablation studies. Computer Vision and Pattern Recognition (CVPR, 2020) *(oral) (CCF-A)** [Paper] [Code]
	Language-based Colorization of Scene Sketches Changqing Zou^#, Haoran Mo^#(joint first author), Chengying Gao^, Ruofei Du and Hongbo Fu Intro:* This paper for the first time presents a language-based system for interactive colorization of scene sketches, based on semantic comprehension. The proposed system is built upon deep neural networks trained on a large-scale repository of scene sketches and cartoonstyle color images with text descriptions. Given a scene sketch, our system allows users, via language-based instructions, to interactively localize and colorize specific foreground object instances to meet various colorization requirements in a progressive way. We demonstrate the effectiveness of our approach via comprehensive experimental results including alternative studies, comparison with the state-of-the-art methods, and generalization user studies. Given the unique characteristics of language-based inputs, we envision a combination of our interface with a traditional scribble-based interface for a practical multimodal colorization system, benefiting various applications. ACM Transactions on Graphics (SIGGRAPH Asia 2019, Journal track) *(oral) (CCF-A)** [Paper] [Code]
	SketchyScene: Richly-Annotated Scene Sketches Changqing Zou^#, Qian Yu^#, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen^, and Hao Zhang Intro:* This paper constructed the first large-scale dataset of scene sketches called SketchyScene. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches. European Conference on Computer Vision (ECCV, 2018) (CCF-B) [Paper] [Code]

Image Editing and Synthesis

Including: image inpainting, color restoration, color transfer and non-photorealistic rendering.

	CAP-VSTNet: Content Affinity Preserved Versatile Style Transfer Linfeng Wen, Chengying Gao, Changqing Zou Intro:* Content affinity loss including feature and pixel affinity is a main problem which leads to artifacts in photorealistic and video style transfer. This paper proposes a new framework named CAP-VSTNet, which consists of a new reversible residual network and an unbiased linear transform module, for versatile style transfer. This reversible residual network can not only preserve content affinity but not introduce redundant information as traditional reversible networks, and hence facilitate better stylization. Empowered by Matting Laplacian training loss which can address the pixel affinity loss problem led by the linear transform, the proposed framework is applicable and effective on versatile style transfer. Computer Vision and Pattern Recognition (CVPR, 2023) (CCF-A) [Paper] [Code]
	Structural Prior Guided Image Inpainting for Complex Scene Shuxin Wei, Chengying Gao Intro: Existing deep-learning based image inpainting methods have reach plausible results for small corrupted regions with rich context information. However, these methods fail to generate semantically reasonable results and clear boundaries. In this paper, we disentangle inpainting for complex scene into two stages: semantic segmentation map inpainting and segmentation guided texture inpainting. We use feature correspondence matrix to find correlation between segmentation maps and known region of corrupted images and realize texture generation of corrupted region. International Conference on Multimedia & Expo (ICME, 2021) *(oral) (CCF-B)** [Paper]
	基于稀疏结构的复杂物体修复高成英，徐仙儿，罗燕媚，王栋计算机学报，2019
	An edge-refined vectorized deep colorization model for grayscale-to-color images Zhuo Su, Xiangguo Liang, Jiaming Guo, Chengying Gao, Xiaonan Luo Neurocomputing, 2018 [Paper]
	PencilArt: A Chromatic Penciling Style Generation Framework Chengying Gao, Mengyue Tang, Xiangguo Liang, Zhou Su, Changqing Zou Computer Graphics Forum (CGF), 2018 (CCF-B) [Paper]
	Data-Driven Image Completion for Complex Object Chengying Gao, Yanmei Luo, Hefeng Wu, Dong Wang Signal Processing: Image Communication*, 2017 [Paper]
	L0 Gradient-Preserving Color Transfer Dong Wang, Changqing Zou, Guiqing Li, Chengying Gao, Zhuo Su, Ping Tan Computer Graphics Forum (CGF), 2017 (CCF-B) [Paper]

3D Pose Estimation and Motion Generation

Unpaired Motion Style Transfer with Motion-oriented Projection Flow Network

Yue Huang, Haoran Mo, Xiao Liang, Chengying Gao*

Intro: In this paper, we propose a novel unpaired motion style transfer framework that generates complete stylized motions with consistent content. We introduce a motion-oriented projection flow network (M-PFN) designed for temporal motion data, which encodes the content and style motions into latent codes and decodes the stylized features produced by adaptive instance normalization (AdaIN) into stylized motions. The M-PFN contains dedicated operations and modules, e.g., Transformer, to process the temporal information of motions, which help to improve the continuity of the generated motions.

International Conference on Multimedia & Expo (ICME, 2022) (*oral) (CCF-B)
[Paper]

3D interacting hand pose and shape estimation from a single RGB image

Chengying Gao*, Yujia Yang, Wensheng Li

Intro: This paper proposes a network called GroupPoseNet using a grouping strategy to address this problem. GroupPoseNet extracts the left- and right-hand features respectively and thus avoids the mutual affection between the interacting hands. Empowered by a novel up-sampling block called MF-Block predicting 2D heat-maps in a progressive way by fusing image features, hand pose features, and multi-scale features, GroupPoseNet is effective and robust to severe occlusions. To achieve an effective 3D hand reconstruction, we design a transformer mechanism based inverse kinematics module(termed TikNet) to generate 3D hand mesh.

Neurocomputing, 2022
[Paper]

Garment Modeling and Virtual Try-on

	Controllable Garment Image Synthesis Integrated with Frequency Domain Features Xinru Liang, Haoran Mo, Chengying Gao* Intro: We propose a controllable garment image synthesis framework that takes as inputs an outline sketch and a texture patch and generates garment images with complicated and diverse texture patterns. To improve the performance of global texture expansion, we exploit the frequency domain features in the generative process, which are from a Fast Fourier Transform (FFT) and able to represent the periodic information of the patterns. We also introduce a perceptual loss in the frequency domain to measure the similarity of two texture pattern patches in terms of their intrinsic periodicity and regularity. Computer Graphics Forum (Pacific Graphics, 2023) *(oral) (CCF-B)** [Paper]
	FashionGAN: Display your fashion design using Conditional Generative Adversarial Nets Yirui Cui, Qi Liu, Chengying Gao, Zhuo Su Computer Graphics Forum (Pacific Graphics, 2018) (oral) (CCF-B)** [Paper] [Code] [Dataset]
	Automatic 3D Garment Fitting Based on Skeleton Driving Haozhong Cai, Guangyuan Shi, Chengying Gao, Dong Wang Pacific-Rim Conference on Multimedia (PCM, 2018) (oral) (CCF-C)** [Paper]
	Automatic Garment Modeling From Front And Back Images Lifeng Huang, Chengying Gao* Pacific Graphics (PG, 2014) (CCF-B) [Paper]

Multimedia Processing & 3D Rendering and Modeling

Multimedia Processing: generation and understanding of music and dance.
3D Rendering and Modeling: fast fluid surface reconstruction based on narrow band method, fabric modeling and rendering.

	Video-Driven Sketch Animation via Cyclic Reconstruction Mechanism Xiao Liang, Zijian Zhao, Weichao Zeng, Yutong He, Fupeng He, Yiyi Wang and Chengying Gao Intro: Learning musical structures and composition patterns is necessary for both music generation and understanding, but current methods do not make uniform use of learned features to generate and comprehend music simultaneously. In this paper, we propose PianoBART, a pre-trained model that uses BART for both symbolic piano music generation and understanding. We devise a multi-level object selection strategy for different pre-training tasks of PianoBART, which can prevent information leakage or loss and enhance learning ability. The musical semantics captured in pre-training are fine-tuned for music generation and understanding tasks. Experiments demonstrate that PianoBART efficiently learns musical patterns and achieves outstanding performance in generating high-quality coherent pieces and comprehending music. International Conference on Multimedia & Expo (ICME, 2024) (CCF-B)
	A Completely Parallel Surface Reconstruction Method for Particle-Based Fluids Wencong Yang, Chengying Gao Intro: In this paper, a fast, simple and extremely accurate narrow-band method of fluid surface is proposed firstly, which makes the surface reconstruction algorithm (such as marching cube) accurately process the valid fluid surface area, which greatly avoids the useless calculation process. At the same time, we analyze the potential race conditions and conditional branching in the reconstruction process, by using mutual exclusive prefix sum algorithm, the whole process of fluid surface reconstruction is completely parallelized, which greatly speeds up the efficiency of surface reconstruction. Computer Graphics International (CGI, 2020) (CCF-C) [Paper]
	Fully automatic algorithm on yarn model generation Zekun Zhang [Introduction (PPT)]
	Microscopic model based real time algorithm on fabric rendering Xingrong Luo [Introduction (PPT)]