Sketch Generation and Applications

 

 

Multi-instance Referring Image Segmentation of Scene Sketches based on Global Reference Mechanism

Peng Ling, Haoran Mo and Chengying Gao*

Intro: We propose GRM-Net, a one-stage framework tailored for multi-instance referring image segmentation of scene sketches. We extract the language features from the expression and fuse it into a conventional instance segmentation pipeline for filtering out the undesired instances in a coarse-to-fine manner and keeping the matched ones. To model the relative arrangement of the objects and the relationship among them from a global view, we propose a global reference mechanism (GRM) to assign references to each detected candidate to identify its position.

Pacific Graphics (PG 2022) (*oral)  (CCF-B)
[Paper]

Line Art Colorization Based on Explicit Region Segmentation

Ruizhi Cao, Haoran Mo and Chengying Gao*

Intro: We introduce an explicit segmentation fusion mechanism to aid colorization frameworks in avoiding color bleeding artifacts. This mechanism is able to provide region segmentation information for the colorization process explicitly so that the colorization model can learn to avoid assigning the same color across regions with different semantics or inconsistent colors inside an individual region. The proposed mechanism is designed in a plug-and-play manner, so it can be applied to a diversity of line art colorization frameworks with various kinds of user guidances.

Computer Graphics Forum (Pacific Graphics 2021) (*oral)  (CCF-B)
[Paper] [Code]

General Virtual Sketching Framework for Vector Line Art

Haoran Mo, Edgar Simo-Serra, Chengying Gao*, Changqing Zou and Ruomei Wang

Intro: Vector line art plays an important role in graphic design, however, it is tedious to manually create. We introduce a general framework to produce line drawings from a wide variety of images, by learning a mapping from raster image space to vector image space. Our approach is based on a recurrent neural network that draws the lines one by one. A differentiable rasterization module allows for training with only supervised raster data. We use a dynamic window around a virtual pen while drawing lines, implemented with a proposed aligned cropping and differentiable pasting modules. Furthermore, we develop a stroke regularization loss that encourages the model to use fewer and longer strokes to simplify the resulting vector image. Ablation studies and comparisons with existing methods corroborate the efficiency of our approach which is able to generate visually better results in less computation time, while generalizing better to a diversity of images and applications.

ACM Transactions on Graphics (SIGGRAPH 2021, Journal track) (*oral)  (CCF-A)
[Paper] [Code] [Project Page]

SketchyCOCO: Image Generation from Freehand Scene Sketches

Chengying Gao, Qi Liu, Qi Xu, Jianzhuang Liu, Limin Wang, Changqing Zou*

Intro: We introduce the first method for automatic image generation from scene-level freehand sketches. Our model allows for controllable image generation by specifying the synthesis goal via freehand sketches. The key contribution is an attribute vector bridged generative adversarial network called edgeGAN which supports high visual-quality image content generation without using freehand sketches as training data. We build a large-scale composite dataset called SketchyCOCO to comprehensively evaluate our solution. We validate our approach on the task of both objectlevel and scene-level image generation on SketchyCOCO. We demonstrate the method’s capacity to generate realistic complex scene-level images from a variety of freehand sketches by quantitative, qualitative results, and ablation studies.

Computer Vision and Pattern Recognition (CVPR, 2020) (*oral)  (CCF-A)
[Paper] [Code]

Language-based Colorization of Scene Sketches

Changqing Zou#, Haoran Mo#(joint first author), Chengying Gao*, Ruofei Du and Hongbo Fu

Intro: This paper for the first time presents a language-based system for interactive colorization of scene sketches, based on semantic comprehension. The proposed system is built upon deep neural networks trained on a large-scale repository of scene sketches and cartoonstyle color images with text descriptions. Given a scene sketch, our system allows users, via language-based instructions, to interactively localize and colorize specific foreground object instances to meet various colorization requirements in a progressive way. We demonstrate the effectiveness of our approach via comprehensive experimental results including alternative studies, comparison with the state-of-the-art methods, and generalization user studies. Given the unique characteristics of language-based inputs, we envision a combination of our interface with a traditional scribble-based interface for a practical multimodal colorization system, benefiting various applications.

ACM Transactions on Graphics (SIGGRAPH Asia 2019, Journal track) (*oral)  (CCF-A)
[Paper] [Code]

SketchyScene: Richly-Annotated Scene Sketches

Changqing Zou#, Qian Yu#, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen*, and Hao Zhang

Intro: This paper constructed the first large-scale dataset of scene sketches called SketchyScene. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches.

European Conference on Computer Vision (ECCV, 2018)  (CCF-B)
[Paper] [Code]

Back to top

Image Editing and Synthesis

Including: image inpainting, color restoration, color transfer and non-photorealistic rendering.


Structural Prior Guided Image Inpainting for Complex Scene

Shuxin Wei, Chengying Gao

Intro: Existing deep-learning based image inpainting methods have reach plausible results for small corrupted regions with rich context information. However, these methods fail to generate semantically reasonable results and clear boundaries. In this paper, we disentangle inpainting for complex scene into two stages: semantic segmentation map inpainting and segmentation guided texture inpainting. We use feature correspondence matrix to find correlation between segmentation maps and known region of corrupted images and realize texture generation of corrupted region.

International Conference on Multimedia & Expo (ICME, 2021) (*oral)  (CCF-B)
[Paper]

基于稀疏结构的复杂物体修复

高成英，徐仙儿，罗燕媚，王栋

计算机学报，2019

An edge-refined vectorized deep colorization model for grayscale-to-color images

Zhuo Su, Xiangguo Liang, Jiaming Guo, Chengying Gao, Xiaonan Luo

Neurocomputing, 2018
[Paper]

PencilArt: A Chromatic Penciling Style Generation Framework

Chengying Gao, Mengyue Tang, Xiangguo Liang, Zhou Su, Changqing Zou

Computer Graphics Forum (CGF), 2018  (CCF-B)
[Paper]

Data-Driven Image Completion for Complex Object

Chengying Gao, Yanmei Luo, Hefeng Wu*, Dong Wang

Signal Processing: Image Communication, 2017
[Paper]

L0 Gradient-Preserving Color Transfer

Dong Wang, Changqing Zou, Guiqing Li, Chengying Gao, Zhuo Su, Ping Tan

Computer Graphics Forum (CGF), 2017  (CCF-B)
[Paper]

Back to top

3D Pose Estimation and Motion Generation

Unpaired Motion Style Transfer with Motion-oriented Projection Flow Network

Yue Huang, Haoran Mo, Xiao Liang, Chengying Gao*

Intro: In this paper, we propose a novel unpaired motion style transfer framework that generates complete stylized motions with consistent content. We introduce a motion-oriented projection flow network (M-PFN) designed for temporal motion data, which encodes the content and style motions into latent codes and decodes the stylized features produced by adaptive instance normalization (AdaIN) into stylized motions. The M-PFN contains dedicated operations and modules, e.g., Transformer, to process the temporal information of motions, which help to improve the continuity of the generated motions.

International Conference on Multimedia & Expo (ICME, 2022) (*oral)  (CCF-B)
[Paper]

3D interacting hand pose and shape estimation from a single RGB image

Chengying Gao*, Yujia Yang, Wensheng Li

Intro: This paper proposes a network called GroupPoseNet using a grouping strategy to address this problem. GroupPoseNet extracts the left- and right-hand features respectively and thus avoids the mutual affection between the interacting hands. Empowered by a novel up-sampling block called MF-Block predicting 2D heat-maps in a progressive way by fusing image features, hand pose features, and multi-scale features, GroupPoseNet is effective and robust to severe occlusions. To achieve an effective 3D hand reconstruction, we design a transformer mechanism based inverse kinematics module(termed TikNet) to generate 3D hand mesh.

Neurocomputing, 2022
[Paper]

Back to top

Garment Modeling and Virtual Try-on

FashionGAN: Display your fashion design using Conditional Generative Adversarial Nets

Yirui Cui, Qi Liu, Chengying Gao*, Zhuo Su

Computer Graphics Forum (Pacific Graphics, 2018) (*oral)  (CCF-B)
[Paper] [Code] [Dataset]

Automatic 3D Garment Fitting Based on Skeleton Driving

Haozhong Cai, Guangyuan Shi, Chengying Gao*, Dong Wang

Pacific-Rim Conference on Multimedia (PCM, 2018) (*oral)  (CCF-C)
[Paper]

Automatic Garment Modeling From Front And Back Images

Lifeng Huang, Chengying Gao*

Pacific Graphics (PG, 2014)  (CCF-B)
[Paper]

Back to top

3D Rendering & Modeling

Including: fast fluid surface reconstruction based on narrow band method, fabric modeling and rendering.


A Completely Parallel Surface Reconstruction Method for Particle-Based Fluids

Wencong Yang, Chengying Gao

Intro: In this paper, a fast, simple and extremely accurate narrow-band method of fluid surface is proposed firstly, which makes the surface reconstruction algorithm (such as marching cube) accurately process the valid fluid surface area, which greatly avoids the useless calculation process. At the same time, we analyze the potential race conditions and conditional branching in the reconstruction process, by using mutual exclusive prefix sum algorithm, the whole process of fluid surface reconstruction is completely parallelized, which greatly speeds up the efficiency of surface reconstruction.

Computer Graphics International (CGI, 2020)  (CCF-C)
[Paper]

Fully automatic algorithm on yarn model generation

Zekun Zhang

[Introduction (PPT)]

Microscopic model based real time algorithm on fabric rendering

Xingrong Luo

[Introduction (PPT)]

Back to top