Intelligent and Multimedia Science Laboratory

Sketch Generation and Applications
Image Editing and Synthesis
Non-photorealistic Rendering
Garment Modeling and Virtual Try-on
3d Rendering & Modeling

Sketch Generation and Applications

	Line Art Colorization Based on Explicit Region Segmentation Ruizhi Cao, Haoran Mo and Chengying Gao^* Intro: We introduce an explicit segmentation fusion mechanism to aid colorization frameworks in avoiding color bleeding artifacts. This mechanism is able to provide region segmentation information for the colorization process explicitly so that the colorization model can learn to avoid assigning the same color across regions with different semantics or inconsistent colors inside an individual region. The proposed mechanism is designed in a plug-and-play manner, so it can be applied to a diversity of line art colorization frameworks with various kinds of user guidances. Computer Graphics Forum (Pacific Graphics 2021) *(oral)** [Paper] [Code]
	General Virtual Sketching Framework for Vector Line Art Haoran Mo, Edgar Simo-Serra, Chengying Gao^, Changqing Zou and Ruomei Wang Intro:* Vector line art plays an important role in graphic design, however, it is tedious to manually create. We introduce a general framework to produce line drawings from a wide variety of images, by learning a mapping from raster image space to vector image space. Our approach is based on a recurrent neural network that draws the lines one by one. A differentiable rasterization module allows for training with only supervised raster data. We use a dynamic window around a virtual pen while drawing lines, implemented with a proposed aligned cropping and differentiable pasting modules. Furthermore, we develop a stroke regularization loss that encourages the model to use fewer and longer strokes to simplify the resulting vector image. Ablation studies and comparisons with existing methods corroborate the efficiency of our approach which is able to generate visually better results in less computation time, while generalizing better to a diversity of images and applications. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2021) *(oral)** [Paper] [Code] [Project Page]
	SketchyCOCO: Image Generation from Freehand Scene Sketches Chengying Gao, Qi Liu, Qi Xu, Jianzhuang Liu, Limin Wang, Changqing Zou* Intro: We introduce the first method for automatic image generation from scene-level freehand sketches. Our model allows for controllable image generation by specifying the synthesis goal via freehand sketches. The key contribution is an attribute vector bridged generative adversarial network called edgeGAN which supports high visual-quality image content generation without using freehand sketches as training data. We build a large-scale composite dataset called SketchyCOCO to comprehensively evaluate our solution. We validate our approach on the task of both objectlevel and scene-level image generation on SketchyCOCO. We demonstrate the method’s capacity to generate realistic complex scene-level images from a variety of freehand sketches by quantitative, qualitative results, and ablation studies. Computer Vision and Pattern Recognition (CVPR) *(oral)** , 2020 [Paper] [Code]
	Language-based Colorization of Scene Sketches Changqing Zou^#, Haoran Mo^#(joint first author), Chengying Gao^, Ruofei Du and Hongbo Fu Intro:* This paper for the first time presents a language-based system for interactive colorization of scene sketches, based on semantic comprehension. The proposed system is built upon deep neural networks trained on a large-scale repository of scene sketches and cartoonstyle color images with text descriptions. Given a scene sketch, our system allows users, via language-based instructions, to interactively localize and colorize specific foreground object instances to meet various colorization requirements in a progressive way. We demonstrate the effectiveness of our approach via comprehensive experimental results including alternative studies, comparison with the state-of-the-art methods, and generalization user studies. Given the unique characteristics of language-based inputs, we envision a combination of our interface with a traditional scribble-based interface for a practical multimodal colorization system, benefiting various applications. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2019) *(oral)** [Paper] [Code]
	SketchyScene: Richly-Annotated Scene Sketches Changqing Zou^#, Qian Yu^#, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen^, and Hao Zhang Intro:* This paper constructed the first large-scale dataset of scene sketches called SketchyScene. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches. European Conference on Computer Vision (ECCV), 2018 [Paper] [Code]

Image Editing and Synthesis

Including: image inpainting, color restoration and color transfer.

	Structural Prior Guided Image Inpainting for Complex Scene Shuxin Wei, Chengying Gao Intro: Existing deep-learning based image inpainting methods have reach plausible results for small corrupted regions with rich context information. However, these methods fail to generate semantically reasonable results and clear boundaries. In this paper, we disentangle inpainting for complex scene into two stages: semantic segmentation map inpainting and segmentation guided texture inpainting. We use feature correspondence matrix to find correlation between segmentation maps and known region of corrupted images and realize texture generation of corrupted region. International Conference on Multimedia & Expo (ICME), 2021 *(oral)** [Paper]
	基于稀疏结构的复杂物体修复高成英，徐仙儿，罗燕媚，王栋计算机学报，2019
	An edge-refined vectorized deep colorization model for grayscale-to-color images Zhuo Su, Xiangguo Liang, Jiaming Guo, Chengying Gao, Xiaonan Luo Neurocomputing, 2018 [Paper]
	Data-Driven Image Completion for Complex Object Chengying Gao, Yanmei Luo, Hefeng Wu, Dong Wang Signal Processing: Image Communication*, 2017 [Paper]
	L0 Gradient-Preserving Color Transfer Dong Wang, Changqing Zou, Guiqing Li, Chengying Gao, Zhuo Su, Ping Tan Computer Graphics Forum (CGF), 2017 [Paper]

Non-Photorealistic Rendering

PencilArt: A Chromatic Penciling Style Generation Framework

Chengying Gao, Mengyue Tang, Xiangguo Liang, Zhou Su, Changqing Zou

Computer Graphics Forum (CGF), 2018
[Paper]

Garment Modeling and Virtual Try-on

FashionGAN: Display your fashion design using Conditional Generative Adversarial Nets

Yirui Cui, Qi Liu, Chengying Gao*, Zhuo Su

Computer Graphics Forum (Proceedings of Pacific Graphics 2018) (*oral)
[Paper] [Code] [Dataset]

Automatic 3D Garment Fitting Based on Skeleton Driving

Haozhong Cai, Guangyuan Shi, Chengying Gao*, Dong Wang

Pacific-Rim Conference on Multimedia (PCM) (*oral), 2018
[Paper]

Automatic Garment Modeling From Front And Back Images

Lifeng Huang, Chengying Gao*

Pacific Graphics (PG), 2014
[Paper]

3d Rendering & Modeling

Including: fast fluid surface reconstruction based on narrow band method, fabric modeling and rendering.

A Completely Parallel Surface Reconstruction Method for Particle-Based Fluids

Wencong Yang, Chengying Gao

Intro: In this paper, a fast, simple and extremely accurate narrow-band method of fluid surface is proposed firstly, which makes the surface reconstruction algorithm (such as marching cube) accurately process the valid fluid surface area, which greatly avoids the useless calculation process. At the same time, we analyze the potential race conditions and conditional branching in the reconstruction process, by using mutual exclusive prefix sum algorithm, the whole process of fluid surface reconstruction is completely parallelized, which greatly speeds up the efficiency of surface reconstruction.

Computer Graphics International (CGI) , 2020
[Paper]

Fully automatic algorithm on yarn model generation

Zekun Zhang

[Introduction (PPT)]

Microscopic model based real time algorithm on fabric rendering

Xingrong Luo

[Introduction (PPT)]