Texture Memory-Augmented Deep Patch-Based Image Inpainting

Rui Xu1      Minghao Guo 1      Jiaqi Wang 1      Xiaoxiao Li1      Bolei Zhou1      Chen Change Loy2

Abstract


Patch-based methods and deep networks have been employed to tackle image inpainting problem, with their own strengths and weaknesses. Patch-based methods are capable of restoring a missing region with high-quality texture through searching nearest neighbor patches from the unmasked regions. However, these methods bring problematic contents when recovering large missing regions. Deep networks, on the other hand, show promising results in completing large regions. Nonetheless, the results often lack faithful and sharp details that resemble the surrounding area. By bringing together the best of both paradigms, we propose a new deep inpainting framework where texture generation is guided by a texture memory of patch samples extracted from unmasked regions. The framework has a novel design that allows texture memory retrieval to be trained end-to-end with the deep inpainting network. In addition, we introduce a patch distribution loss to encourage high-quality patch synthesis. The proposed method shows superior performance both qualitatively and quantitatively on three challenging image benchmarks, i.e., Places, CelebA-HQ, and Paris Street-View datasets.

Framework



As shown in this figure, our two-stage framework follows a coarse-to-fine style. Taking the whole image as input, the first stage predicts the coarse results containing global structure guidance. Different from previous works, the second stage only adopts corrupted patches in a parallel manner. In a word, our T-MAD first recovers the coarse structure, and then synthesizes high-quality textures at patch level with the help of the texture memory.

Visualization


The qualitative comparison with existing models. From left to right: Corrupted input image, results of PatchMatch, PICNet, Edge-Connect, DeepFill, CRA, our T-MAD and ground-truth. (Best viewed with zoom-in.)

Materials


Citation

@article{xu2020texture,
title={Texture Memory-Augmented Deep Patch-Based Image Inpainting},
author={Xu, Rui and Guo, Minghao and Wang, Jiaqi and Li, Xiaoxiao and Zhou, Bolei and Loy, Chen Change},
journal={arXiv preprint arXiv:2009.13240},
year={2020}
}

Contact

If you have any question, please contact Rui Xu at xr018@ie.cuhk.edu.hk.