Novel techniques for auto-inpainting in heritage reconstruction

Abstract

Digital reconstruction of ruined historic monuments and heritage sites can help in visualizing how these may have existed in the past. Also, such a process requires no physical alteration to the existing monuments. This facilitates in avoiding their further accidental damage. A digitally reconstructed heritage site in the form of an immersive walkthrough can serve as a delightful tool for both educational and entertainment purpose. This thesis presents novel approaches for auto-inpainting that involves image inpainting as well as automatic detection of cracks and other damaged regions for inpainting in heritage monuments. As a by-product of one of our inpainting techniques, we are also able to perform resolution enhancement i.e. super-resolution. The purpose is to obtain the digitally reconstructed monuments having enhanced resolution, where the digital reconstruction is performed by automatically detecting and inpainting the damaged regions. The resulting images can serve as an input to immersive walkthrough systems. In our first inpainting approach, we propose an iterative exemplar based method that fills the missing pixels by making use of parameters of an autoregressive (AR) model. These parameters represent the pixel-neighborhood relationship. Considering a set of candidate exemplars, we estimate the parameters of the AR model using the non-negatively constrained least squares (NNLS) method. In our second inpainting approach, we propose a unified framework to perform simultaneous inpainting and super-resolution. Here, we construct dictionaries of image-representative low and high resolution patch pairs from the known regions in the test image and its coarser resolution. Inpainting of the missing pixels is performed using exemplars found by comparing patch details at a finer resolution, where self-learning is used to obtain the finer resolution patches by making use of the constructed dictionaries. The obtained finer resolution patches represent the super-resolved patches in the missing regions. Advantage of our approach when compared to other exemplar based inpainting techniques are (a) additional constraint in the form of finer resolution matching results in better inpainting and (b) inpainting is obtained not only in the given spatial resolution but also at higher resolution leading to super-resolution inpainting. In practice, inpainting techniques perform filling of the missing pixels in known regions i.e., they rely on the user to manually select the regions to be inpainted. This is also the case with our two inpainting methods briefly discussed above. To avoid human interaction i.e., manual selection of region to be inpainted, we propose auto-inpainting techniques wherein the missing regions to be inpainted are not known but are automatically detected prior to inpainting. First, we propose a novel method that detects and inpaints the visually dominant damage regions viz. eyes, nose and lips in facial images of statues. A bilateral symmetry based method is used to identify the eye, nose and lip regions. We then use texton features extracted from each of these regions in a multi-resolution framework to characterize both the damaged and non-damaged regions. For classification, the texton features are matched with those extracted offline from a training set consisting of true damaged and non-damaged regions. If the region is found to be damaged, the best matching non-damaged region from the training set is used to inpaint the identified region using an existing inpainting method. Having proposed an auto-inpainting technique for facial regions in statues, we proceed to auto-inpainting the non-facial regions in heritage monuments by proposing three techniques for detecting and inpainting cracks. Here, we initially propose a simple pixel based approach that uses order-statistics and density based filters for the detection of cracks. We then propose a patch based approach for crack detection by making use of singular value decomposition (SVD) for patch comparison. Further, we propose another effective and more accurate crack detection method based on comparison of patches using a measure derived from the edit distance, which is a popular string metric used in the area of text mining. We extend this crack detection approach to perform inpainting of video frames by making use of the scale invariant feature transform (SIFT) and homography. Here, we consider the camera movement to be unconstrained while capturing video of the heritage site, since such videos are typically captured by novices, hobbyists and tourists. Finally, we provide the temporal consistency measure to quantify the quality of the inpainted video.