论文标题
基于补丁的随机关注图像编辑
Patch-Based Stochastic Attention for Image Editing
论文作者
论文摘要
近年来，注意机制在深度学习中变得至关重要。这些非本地操作类似于图像处理中的传统基于补丁的方法，它补充了本地卷积。但是，计算全部注意力矩阵是带有沉重记忆和计算负载的昂贵步骤。这些局限性遏制网络架构和性能，尤其是对于高分辨率图像的情况。我们基于随机算法贴片摩擦提出了一个有效的注意力层，该层用于确定近似最近的邻居。我们将我们提出的层称为“基于斑块的随机注意层”（PSAL）。此外，我们提出了基于贴片聚集的不同方法，以确保psal的不同性，从而允许对包含我们层的任何网络的端到端培训。 PSAL的内存足迹很小，因此可以扩展到高分辨率图像。它在不牺牲最近邻居的空间精度和全球性的情况下保持了这种足迹，这意味着即使在较浅的层次上，也可以轻松地将其插入任何层次。我们演示了PSAL对几个图像编辑任务的有用性，例如图像介入，引导图像着色和单图像超分辨率。我们的代码可在以下网址找到：https：//github.com/ncherel/psal
Attention mechanisms have become of crucial importance in deep learning in recent years. These non-local operations, which are similar to traditional patch-based methods in image processing, complement local convolutions. However, computing the full attention matrix is an expensive step with heavy memory and computational loads. These limitations curb network architectures and performances, in particular for the case of high resolution images. We propose an efficient attention layer based on the stochastic algorithm PatchMatch, which is used for determining approximate nearest neighbors. We refer to our proposed layer as a "Patch-based Stochastic Attention Layer" (PSAL). Furthermore, we propose different approaches, based on patch aggregation, to ensure the differentiability of PSAL, thus allowing end-to-end training of any network containing our layer. PSAL has a small memory footprint and can therefore scale to high resolution images. It maintains this footprint without sacrificing spatial precision and globality of the nearest neighbors, which means that it can be easily inserted in any level of a deep architecture, even in shallower levels. We demonstrate the usefulness of PSAL on several image editing tasks, such as image inpainting, guided image colorization, and single-image super-resolution. Our code is available at: https://github.com/ncherel/psal
