Inversion by P4: polarization-picture post-processing

Yoav Y. Schechner 0 0 Department of Electrical Engineering , Technion , Israel Institute of Technology , Haifa 32000 , Israel Polarization may be sensed by imaging modules. This is done in various engineering systems as well as in biological systems, specifically by insects and some marine species. However, polarization per pixel is usually not the direct variable of interest. Rather, polarization-related data serve as a cue for recovering task-specific scene information. How should polarization-picture post-processing (P4) be done for the best scene understanding? Answering this question is not only helpful for advanced engineering (computer vision), but also to prompt hypotheses as to the processing occurring within biological systems. In various important cases, the answer is found by a principled expression of scene recovery as an inverse problem. Such an expression relies directly on a physics-based model of effects in the scene. The model includes analysis that depends on the different polarization components, thus facilitating the use of these components during the inversion, in a proper, even if non-trivial, manner. We describe several examples for this approach. These include automatic removal of path radiance in haze or underwater, overcoming partial semireflections and visual reverberations; three-dimensional recovery and distance-adaptive denoising. The resulting inversion algorithms rely on signal-processing methods, such as independent component analysis, deconvolution and optimization. 1. INTRODUCTION An expanding array of animals are found to have a visual system that is polarization-sensitive, using several mechanisms [1,2]. This array includes various marine animals [3 17] as well as air and land species [18 22]. It has been hypothesized and demonstrated that such a capacity can help animals in various tasks, such as navigation (exploiting the polarization field of the sky), finding and discriminating mates, finding prey and communication. Similarly, machine vision systems may benefit from polarization. Thus, computational methods are being developed for polarization-picture post-processing (defined here as P4). Some tasks are enhancement of images, and extraction of features useful for higherlevel operations (segmentation and recognition). In this paper, we focus on inverse problems that can be solved using P4. In these problems, a physical model of effects occurring in the scene can be formulated in a closed form. Inversion of the model quantitatively recovers the scene, overcoming various degradations. In the context of polarization, this approach is used in remote sensing from satellites, astronomy and medical imaging. In contrast, this paper surveys several inverse problems relating to objects, distances and tasks that are encountered or 2. DESCATTERING In atmospheric haze or underwater, scene visibility is degraded in both brightness contrast and colour. The benefit of seeing better in such media is obvious, for animals, human operators and machines. The mechanisms of image degradation both in haze and underwater are driven by scattering within the medium. The main difference between these environments is the distance scale. Other differences relate to the colour and angular dependency of light scattering. Due to the similarity of the effects, image formation in both environments can be formulated using the same parametric equations: the mentioned differences between the media are expressed in the values taken by the parameters of these equations. It is often suggested that P4 can increase contrast in scattering environments. One approach is based on subtraction of different polarization-filtered images [23 25], or displaying the degree of polarization (DOP) [26,27]. This is an enhancement approach, rather than an attempt to invert the image formation process and thus recover the objects. Furthermore, this approach associates polarization mainly with the object radiance. However, rather than the object, light scattered by the medium (atmosphere or water) is often significantly polarized [6,28,29] and dominates the polarization of the acquired light. (a) Model In this section, we describe a simple model for image formation in haze or water, including polarization. Then, this model is mathematically inverted to recover the object. Polarization plays an important role in this recovery task [30 35]. As depicted in figure 1, an image acquired in a medium has two sources. The first source is the scene object at distance z, the radiance of which is attenuated by absorption and scattering. This component is also somewhat blurred by scattering, but we neglect this optical blur, as we explain later. The image corresponding to this degraded source is the signal Sx; y Lobjectx; ytz; where Lobject is the object radiance we would have sensed, had there been no scattering and absorption along the line of sight (LOS), and (x,y) are the image coordinates. Here t(z) is the transmissivity of the medium. It monotonically decreases with the distance z. The second source is ambient illumination. Part of the illumination is scattered towards the camera by particles in the medium. In the literature, this part is termed path radiance [36], veiling light [6,16,37], spacelight [4,6,16,29] and backscatter [38]. In literature dealing with the atmosphere, it is also termed airlight [39]. This component is denoted by B. It monotonically increases with z. The total image irradiance is Clearly, B is a positive additive component. It does not occlude the object. So, how come B appears to veil the scene? Furthermore, the image formation model (equations (2.1) and (2.2)) neglects any optical blur. Thus, how come hazy/underwater images appear blurred? The answer to these puzzles is given in Treibitz & Schechner [40]. Due to the quantum nature of light (photons), the additive component B induces random photon noise in the image. To understand this, recall that photon flux from the scene and the detection of each photon are Poissonian random processes [41]. This randomness yields an effective noise. The overall noise variance [41 43] of the measured pixel intensity is approximately Here k and g are positive constants, which are specific to the sensor. Due to equations (2.2) and (2.3), B increases Itotal and thus the noise1 intensity. The longer the distance to the object, the larger B is. Consequently, the image is more noisy there, making it more difficult to see small object details (veiling), even if contrast-stretching is applied by image postprocessing. As analysed in Treibitz & Schechner [40], image noise creates an effective blur, despite an absence of blur in the optical process: the recoverable signal has an effective spatial cutoff frequency, induced by noise. To recover Lobject by inverting equation (2.1), there is first a need to decouple the two unknowns (per image point) S and B, which are mixed by equation (2.2). This is where P4 becomes helpful. Le (...truncated)