A PDF file should load here. If you do not see its contents
the file may be temporarily unavailable at the journal website
or you do not have a PDF plug-in installed and enabled in your browser.
Alternatively, you can download the file locally and open with any standalone PDF reader:
https://link.springer.com/content/pdf/10.1007%2Fs11263-012-0607-7.pdf
A Variational Approach to Video Registration with Subspace Constraints
Ravi Garg
Anastasios Roussos
Lourdes Agapito
This paper addresses the problem of non-rigid video registration, or the computation of optical flow from a reference frame to each of the subsequent images in a sequence, when the camera views deformable objects. We exploit the high correlation between 2D trajectories of different points on the same non-rigid surface by assuming that the displacement of any point throughout the sequence can be expressed in a compact way as a linear combination of a low-rank motion basis. This subspace constraint effectively acts as a trajectory regularization term leading to temporally consistent optical flow. We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the low-rank manifold. The resulting energy functional can be decoupled into the optimization of the brightness constancy and spatial regularization terms, leading to an efficient optimization scheme. Additionally, we propose a novel optimization scheme for the case of vector valued images, based on the dualization of the data term. This allows us to extend our approach to deal with colour images which results in significant improvements on the registration results. Finally, we provide a new benchmark dataset, based on motion capture data of a flag waving in the wind, with dense ground truth optical flow for evaluation of multiframe optical flow algorithms for non-rigid surfaces. Our experiments show that our proposed approach outperforms state of the art optical flow and dense non-rigid registration algorithms.
-
Optical flow in the presence of non-rigid deformations is a
challenging task and an important problem that continues to
attract significant attention from the computer vision
community. It has wide ranging applications from medical imaging
and video augmentation to non-rigid structure from motion.
Given a template image of a non-rigid object and an input
image of it after deforming, the task can be described as one
of finding the displacement field (warp) that relates the input
image back to the template. In this paper we consider long
video sequences instead of a single pair of frameseach of
the images in the sequence must be aligned back to the
reference frame. Our work concerns the estimation of the vector
field of displacements that maps pixels in the reference frame
to each image in the sequence (see Fig. 1).
Two significant difficulties arise. First, the image
displacements between the reference frame and subsequent ones are
large since we deal with long sequences. Secondly, as a
consequence of the non-rigidity of the motion, multiple warps
can explain the same pair of images causing ambiguity. In
this paper we show that a multi-frame approach allows us
to exploit temporal information, resolving these ambiguities
and improving the overall quality of the optical flow. We
make use of the strong correlation between 2D trajectories
of different points on the same non-rigid surface. These
trajectories lie on a lower dimensional subspace and we assume
that the trajectory vector storing 2D positions of a point
across time can be expressed compactly as a linear
combination of a low-rank motion basis. This leads to a
significant reduction in the dimensionality of the problem while
implicitly imposing some form of temporal smoothness.
Figure 2 depicts the lower dimensional trajectory subspace.
Subspace constraints have been used before both in the
context of sparse point tracking (Irani 2002; Brand 2001;
Fig. 1 Video registration is equivalent to the problem of estimating
dense optical flow u(x; n) between a reference frame Iref and each
of the subsequent frames In in a sequence. We propose a multi-frame
optical flow algorithm that exploits temporal consistency by imposing
subspace constraints on the 2D image trajectories
Torresani et al. 2001; Torresani and Bregler 2002) and
optical flow (Irani 2002; Garg et al. 2010) in the rigid and
nonrigid domains, to allow correspondences to be obtained in
low textured areas. While Iranis original rigid (Irani 2002)
formulation along with its non-rigid extensions (Torresani
et al. 2001; Brand 2001; Torresani and Bregler 2002) relied
on minimizing the linearized brightness constraint without
smoothness priors, Garg et al. (2010) extended the subspace
constraints to the continuous domain in the non-rigid case
using a variational approach. Nir et al. (2008) propose a
variational approach to optical flow estimation based on a
spatio-temporal model. However, all of the above approaches
impose the subspace constraint as a hard constraint. Hard
constraints are vulnerable to noise in the data and can be
avoided by substituting them with principled robust
constraints.In this paper we extend the use of multi-frame
temporal smoothness constraints within a variational framework by
providing a more principled energy formulation with a robust
soft constraint which leads to improved results. In
practice, we penalize de (...truncated)