Motion-blur-free video shooting system based on frame-by-frame intermittent tracking

Inoue et al. Robomech J Motion-blur-free video shooting system based on frame-by-frame intermittent tracking Michiaki Inoue 0 Mingjun Jiang 0 Yuji Matsumoto 0 Takeshi Takaki 0 Idaku Ishii 0 0 Department of System Cybernetics, Hiroshima University , Higashi-Hiroshima , Japan In this paper, a concept of frame-by-frame intermittent tracking to achieve motion-blur-free and high-brightness images when video shooting fast moving scenes is proposed. In our tracking concept, two control methods are applied alternately at hundreds of hertz, according to the open or closed shutter state of the camera. When the shutter is open, the target's speed in images is controlled at zero and visual feedback is transmitted to achieve motion blur reduction, and when the shutter is closed, the camera returns to its home position. We developed a prototype of our motion-blur-free video shooting system, which consists of our tracking method implemented on a high-speed two degrees-of-freedom tracking vision platform that controls the pan and tilt directions of the camera view by using high-speed video processing in order to reduce motion blur. Our motion-blur-free video shooting system can capture gray-level 512 × 512 images at 125 fps with frame-by-frame intermittent tracking. Its performance is verified by the experimental results for several video sequences of fast moving objects. In the experiments, without a decrease in the exposure times our method reduced image degradation caused by motion blur. Target tracking; High-speed vision; Active vision; Motion deblurring Introduction Motion blur is a well-known phenomenon that occurs when shooting images of fast moving scenes. The degradation degree of images caused by motion blur depends on the duration of the camera exposure, as well as on the apparent speed of the target scenes, and the camera’s exposure time is often decreased in order to reduce motion blur. However, a trade-off exists between brightness and motion blur in image shooting, because it is difficult to obtain non-blurred bright images with a decreased exposure time, since less light is then projected onto the image sensor. This trade-off is extremely aggravated in highly magnified observations of fast moving scenes in various application fields, such as flowing cells in microscopic fields, precise inspection of products on a moving conveyor line, and road surface and tunnel wall inspection from a moving car, because the apparent speed of the scene increases in the magnified camera view and the light that is projected on the image sensor diminishes when the magnification is increased. Motion deblurring is a frequently used approach for reducing this image degradation resulting from motion blur in image shooting. In many studies, approaches [ 1, 2 ] were developed that apply blur kernels that express motion blur in the input images; the blurred images are restored by deconvolving the images with the estimated blur kernels. These approaches include single-image deblurring methods [ 3–5 ] and multi-image deblurring methods [ 6–8 ]. In the former, the blur kernels are estimated from a single image using parametric models for maximum a-posteriori estimation and in the latter the illposed problems in deconvolution are reduced by estimating the blur kernels from multiple images. Several papers have reported motion deblurring methods that consider the camera’s egomotion while the shutter is open, which is estimated by using gyro sensors and accelerometers [9] or the camera’s geometric location [ 10 ]. However, most of these methods adopt a software-based approach for image restoration and do not consider the acquisition of blur-free input images. There are limitations to the extent to which images can be improved, in particular when significant changes in the target scene occur in the images captured while the camera shutter is open. To reduce motion blur resulting from camera shake, a large number of digital cameras with camera-shake reduction systems have been developed that can stabilize input images by shifting their optical systems mechanically. These image stabilization systems are categorized into two types according to the approach that is applied: the lens-shift approach, which shifts a floating lens to move the optical axis [ 11, 12 ] and the sensorshift approach, which shifts the image sensor [13]. These image stabilizers can stabilize input images by controlling the optical path with a floating lens or the position of the image sensor with the camera’s internal sensors, such as its gyro sensor. The camera-shake reduction system is a camera-stabilization approach that uses the camera internal sensors for reducing motion blur resulting from camera shake; it is unsuitable for shooting blur-free images of fast moving scenes when the camera is fixed, because the internal sensors cannot detect any apparent motion in the captured images. Many high-speed vision systems operating at 1000 fps or more have been developed [ 14, 15 ], and visual tracking algorithms, such as optical-flow systems [16] and face-tracking systems [ 17 ], have been implemented in high-speed vision systems. The effectiveness of real-time high-speed image processing has been verified in many types of applications, such as robot manipulation [ 18 ], flying-object tracking [ 19 ], and micro-organism tracking [ 20, 21 ]. Such high-speed tracking systems can reduce motion blur without decreasing the exposure time, because they can continuously adjust the position of an object to the center of the camera view by using highspeed visual feedback control. However, in such systems, the camera view is adjusted only for a single target object, and the viewpoints cannot be freely changed when the object is tracked in the camera view. For viewpoint-free video shooting, Hayakawa et al. [ 22, 23 ] developed a galvano-mirror-based tracking system that can compensate motion blur in images by controlling the amplitude of the sinusoid trajectory with highspeed visual feedback using the Bayer block matching method in order to stop background scenes appearing in images while the shutter is open, and conducted highframe-rate (HFR) video shooting with motion blur reduction for the purpose of highway inspection conducted from a car traveling at 100  km/h. However, the motion blur reduction was limited when the object speed suddenly changed, because of the limited time resolution of the pan-and-tilt mirror control in the galvano-mirror system; it was not so considered for a programable sawtooth-like wave trajectory that enables alternative switching of the mirror speed from the target object’s speed to zero in frame-by-frame intermittent tracking so that both a moving target object and a static background are clearly observed without blurring at the same time. For microscopic observation with a fixed camera view, Ueno et  al. [ 24 ] developed a motion-blur-free microscope that can shoot non-blurred videos of unidirectionally moving objects at a high frame rate using a piezo actuator-based microscopic tracking system, in which a concept similar to the frame-by-frame intermittent tracking introduced in this study was applied for motion blur reduction; however, the object speed for motion blur reduction was limited to 10 mm/s or less at submillimeter-level due to the upper limit of the movable range of the 1-DOF linear piezo stage, and it can not use for motion-blur-free video-shooting of general objects fastmoving in real space, which are two-dimensionally moving at several meters per second. In this paper, we introduce the concept of frame-byframe intermittent tracking and extend it to a mirrordrive 2-degrees-of-freedom (DOF) piezo actuator-based tracking system for motion-blur-free video shooting of objects moving fast in two dimensions when a high magnification ratio is used. Thus, in this paper, we introduce a frame-by-frame intermittent tracking method to shoot non-blurred and bright videos of fast moving scenes with a fixed camera position by alternating the tracking control methods according to whether the shutter is open or closed. We developed a motion-blur-free video shooting system that simultaneously controls the angles of the pan and tilt mirrors on a 2-DOF piezo actuator-based active vision system for HFR video shooting through the implementation of a frame-by-frame intermittent tracking algorithm in real time. Its performance was verified by the experimental results for several moving scenes. Frame‑by‑frame intermittent tracking Motion blur in video shooting of moving objects depends on their apparent motions on the image sensor when the shutter is open, that is, when the incident light accumulates on the image sensor, whereas their apparent motions cause no motion blur when the shutter is closed, that is, when the image sensor is blind to any incident light. Thus, we introduce a frame-by-frame intermittent tracking method [ 24 ] that can reduce motion blur in video shooting by alternating control methods in a high-speed active vision system, from vision-based tracking control to back-to-home control, according to whether the camera shutter is open or closed; the active vision system changes the optical path to the image sensor. This concept is illustrated in Fig. 1. The vision-based tracking control is activated to maintain the relative velocity between the coordinate systems of the object and the image sensor at zero when the shutter is open; fast-moving target scene motion-blur-free camera frame k tracking target active vision back-to-home tracking target active vision no motion blur don’t care no motion blur don’t care open exposure close exposure open exposure close exposure e x p o s u r e & f r a m e t i m i n g captured captured ultrafast viewpoint control high-speed actuator high-speed vision motion-blur-free camera frame-by-frame intermittent tracking frame k+1 target back-to-home target active vision active vision it is operated by estimating the apparent velocity of the objects in images in real time. The back-to-home control is activated to reset the optical path of the camera to its home position when the shutter is closed. This control, while requiring no information from the image sensor, ensures that the movable range of the active vision system is not exceeded. As compared with methods presented in related papers on motion deblurring and image stabilization, our frameby-frame intermittent tracking method has the following advantages. 1. Motion-blur-free video shooting Without decreasing the exposure time of the camera, high-brightness images of fast moving objects can be captured without motion blur. 2. Vision-based frame-by-frame image stabilization Without any internal sensor being required, the apparent speed of fast moving objects on the image sensor can be controlled at zero in every frame with real-time motion estimation, which is accelerated by high-speed video processing. 3. Free-viewpoint observation Users can freely alter the viewpoint of the camera, when it is controlled by frame-by-frame intermittent tracking. The method includes fixed-viewpoint observation. In our method, the frame-by-frame switching viewpoint control from vision-based tracking control to back-to-home control can be expressed with the sawtooth-like trajectory of the position of the image sensor desired at time t, p(t), according to whether the shutter is open or closed: p(t) = p0 + v(ts(t)) · (t − ts(t)) p0 (0 ≤ t − ts(t) < τo) , (otherwise). (1) where p0 is the home position of the image sensor. τo and τc are the open and closed shutter duration, respectively. ⌊a⌋ is the maximum integer that does not exceed a and ts(t) = ⌊t/τ ⌋τ is the time at which the image is shot at every frame; it is quantized by the frame-cycle time τ = τo + τc. Figure  2 illustrates the saw-tooth-like desired trajectory and control chart of our frame-byframe intermittent tracking method. In Eq. (1), the upper line expresses the control target for vision-based tracking control when the shutter is open, and the lower line expresses the control target for back-to-home control when the shutter is closed. It is assumed that the relative velocity between the coordinate systems for the object and the image-sensor at time ts(t), v(ts(t)), is estimated by processing the captured images in real time. Thus, the state of the shutter periodically changes between open and closed in every frame, and the two control methods should switch when the video is shot with frame-byframe intermittent tracking, corresponding to the framecycle time τ. For HFR video shooting of moving objects with frameby-frame intermittent tracking, a high-speed actuator that can periodically drive the motion of an image sensor at hundreds of hertz or more should be accelerated for frame-by-frame switching of the control methods; its switching frequency perfectly corresponds to the frame rate of a high-speed vision system that can estimate the apparent speeds of moving objects on the image sensor in real time. To design a motion-blur-free video shooting system with frame-by-frame intermittent tracking of moving objects, the following constraints pertaining to a high-frequency response actuator should be considered, as well as the frame rate of the vision system, as illustrated in Fig. 3. 1. Limited moving speed The response of an actuator is determined by its dynamic parameters, such as its mechanical time constant. The speed svmax at which a high-frequency response actuator with a small time constant can move has a certain upper limit. It is difficult to track the target object with no motion blur frame/ exposure timing exposure timings synchronization highspeed actuator frame cycle time o p e n c l o s e o p e n c l o s e o p e n c l o s e o p e n high-speed vision camera position apparent object’s speed actuator positions speed = 0 home position (3) non-linear trajectory (with ripples) close open close open close open close time (1) limited move speed when its apparent speed v on the image sensor is higher than the upper-limit speed svmax of the actuator. 2. Limited moving range The range in which a highfrequency response actuator can move is in general limited because of the trade-off between its frequency response and the range in which it can move. In frame-by-frame intermittent tracking with exposure time τ, the actuator should track a target object, the apparent speed of which on the image sensor is v, over the moving distance vτ while the shutter is open; however, the motion blur cannot be perfectly reduced when the apparent speed v is higher than r vmax = Amax/τ. 3. Nonlinear trajectory Assuming that a target object moves at a fixed speed while the shutter is open, to achieve motion blur reduction the linear trajectory of an actuator should be generated by controlling its inclination such that the apparent speed of the target object is cancelled. Most high-frequency response actuators achieve their high-frequency drives with low damping ratios by reducing their viscosities, such as frictions, whereas it takes a certain time to attenue g ited ran limleb ) a (2 vo m ate their ringing responses once resonant oscillation starts. In frame-by-frame intermittent tracking at hundreds of hertz or more, the interval of the intermittent drive of a high-frequency response actuator is not significantly larger than its damping ratio, and there remain certain nonlinear deviations with ripples in the actuator’s trajectory; these may still leave motion blurs in images. Motion‑blur‑free video‑shooting system System configuration We developed a prototype motion-blur-free video-shooting system, which is designed for frame-by-frame intermittent tracking to allow zoom-in imaging of fast moving objects without incurring motion blur. The system consists of a high-speed vision platform (IDP Express) [ 15 ], a CCTV zoom lens, two piezo tilt stages (PT1M36500S-N, Nano Control Co., Japan) with mirror surfaces, and a personal computer (PC) with an ASUSTeK P6T7 WS Supercomputer mainboard, Intel Core i7 960 3.20GHz CPU, 6-GB memory, Windows 7 Professional 32-bit OS, and a D/A board (PEX-340416, Interface Co., Japan). Figure  4 provides an overview of the prototype system when the HTZ-11000 (Joble Co., Japan) was used as the CCTV zoom lens. IDP Express includes a camera head and an FPGA image processing board (IDP Express board). The camera head has a 512 × 512 pixel CMOS image sensor, the sensor and pixel size of which are 5.12 × 5.12 mm and 10 × 10 μm, respectively. The camera head was mounted on the camera port of the CCTV zoom lens. The IDP Express board was designed for high-speed video processing and recording, and we could implement image processing algorithms by hardware logic on the FPGA (Xilinx XC3S5000); it was mounted using a PCI-e 2.0 × 16 bus I/F on the PC. The 8-bit grayscale 512 × 512 images and 25 mm IDP Express camera head CCTV zoom lens (HTZ11000) 7 5 m m tsoh ttceo o b j Fig. 4 Overview of motion-blur-free video-shooting system processed results could be simultaneously transferred at 2000 fps to the allocated memory in the PC. Two piezo tilt stages were used for a mirror-drive 2-DOF active vision system to realize frame-by-frame intermittent tracking in pan and tilt directions. The piezo tilt stage can shift its surface in the rotation direction with a 2.78 × 10−6◦ resolution, and its size, weight, resonant frequency, and the range within which it can move are 36 × 42 × 29 mm, 100 g, 3900 Hz, and 0.173◦, respectively, when no objects are mounted on it. On the surface of the piezo stage, a 30 × 30 × 5mm-size aluminum mirror (TFA-30S05-1, Sigma Koki Co., Japan) weighing 20 g was mounted. The piezo stage for the pan angle was installed 25  mm in front of the CCTV-zoom lens, and that for the tilt angle was installed 75  mm in front of that for the pan angle; the light from the target object passes to the tilt-mirror stage and the pan-mirror stage, and then is captured on the image sensor on the camera head. The drive voltage for the piezo stages, supplied by a high-capacity piezo driver (PH601, Nano Control Co., Japan), was 0–150 V, and the motor commands from the PC were amplified in the piezo driver in order to operate the piezo stages periodically. In this study, a frame-by-frame intermittent-tracking algorithm was software-implemented on the PC. The apparent speed of the objects in images was estimated in real time using the results processed on the IDP Express board, and motor commands were transferred to the θˆ d (t) = (φˆd (t), ψˆ d (t)) 1. Binarization A grayscale 512 × 512 input image I(x,  y,  t) is captured at time t = kτ at an interval τ with an exposure time τo. I(x,  y,  t) is binarized with the threshold IB into B(x,  y,  t). The apparent velocity of the object in the image at time t is estimated as v(t) = (c(t) − c(t − τ ))/τ with the image centroids c(t) = (M10/M00, M01/M00) at time t and t − τ. The apparent angular velocity of the object in the pan and tilt directions of a zooming optical system, ω(t) = (ωφ (t), ωψ (t)), is proportional to v(t) as ω(t) = A(c(t) − c(t − τ ))/τ , where A is a constant parameter determined by the magnification ratio of the zooming optical system, the pixel pitch of the image sensor, and the distance between the object and the optic center of the optical system. M00, M10, and M01 are the zero- and firstorder moment features of B(x, y, t) defined as Mmn(t) = xmyn · B(x, y, t), (m, n) = (0, 0), (1, 0), (0, 1). X,Y 2. Trajectory generation for intermittent tracking The desired angular trajectory in the pan and tilt directions of the mirror-drive 2-DOF active vision system, θ d (t) = (φd (t), ψd (t)), is generated using the apparent angular velocity of the target object, ω((k − 1)τ ), which is estimated at t = (k − 1)τ, in order to cancel its apparent motion on the image sensor when the shutter is open from t = kτ to kτ + τo: (2) (3) (4)  ω((k − 1)τ ) (t − kτ − τr ) + θ 0 = (f (t; φd (kτ + τo), φ0), f (t; ψd (kτ + τo), ψ0))  θ 0 (− τr ≤ t − kτ < τo) (τo ≤ t − kτ < τb + τo) , (otherwise) piezo stage via the D/A board to reduce motion blur in the images. Corresponding to the drive voltage 0–150 V of the piezo stage, analog voltage signals in the range of 0–10.24 V were outputted from the D/A board mounted on the PC; these signals were converted at a high rate from the 12-bit digital sequences stored in the buffer of the D/A board. The details are described in the following subsection. Integrated algorithms Assuming that a single object to be captured on video is moving two-dimensionally at a certain velocity on a plane that is parallel to the image sensor’s plane, and the object’s apparent velocity in images is proportional to its actual velocity on the plane, the following algorithm was implemented in the prototype system in order to observe a single object in an image. θd(t) = med(φmin, φˆd(t), φmax), med (ψmin, ψˆd(t), ψmax) , (5) where med (a, b, c) indicates the median value of a, b, and c. θ 0 = (φ0, ψ0) indicates the pan and tilt angles for the home position, and [φmin, φmax] and [ψmin, ψmax] indicate the movable ranges of the pan and tilt angles of the mirror-drive 2-DOF active vision system, respectively. τt and τb refer to the duration times of the vision-based tracking control and back-to-home control, respectively. τr = τt − τo is the delay time required for the mirror-drive 2-DOF active vision system to match the apparent motion of the object in the image, corresponding to its rise time. f (t; φd (kτ + τo), φ0) and f (t; ψd (kτ + τo), ψ0) are the fifth-order polynomial trajectory functions that ensure that the back-to-home control moves the mirror-drive 2-DOF active vision system smoothly frame − 1 frame camera shutter open close open close desired trajectory = ( − 1) = = ( + 1) from θ d (kτ + τo) to θ 0 to avoid a large acceleration. Figure 5 shows the timing chart for the generation of the trajectory. 3. Control of mirror-drive 2-DOF active vision system After storing the desired trajectory for the duration time τt + τb, the D/A board begins sending motor commands at t = kτ − τr in synchronization with the image capture timing. The motor commands for the pan and tilt angles are amplified to the drive voltages in the mirror-drive 2-DOF active vision system as V (t) = (Vφ(t), Vψ (t)) = (bφφd(t) + cφ, bψ ψd(t) + cψ ), (6) where the parameters (bφ , cφ ) and (bψ , cψ ) are determined by verifying the actual pan and tilt trajectories of the mirror-drive 2-DOF active vision system. Specifications Using this prototype system, 512 × 512 input images were captured at 125  fps, with a frame interval of τ = 8 ms and exposure time of τo = 4ms. The duration times of the vision-based tracking control and back-to-home control were set to τt = 4.5 and τb = 2.0 ms, respectively; the tracking time τt included a delay time of τr = 0.5ms. The desired trajectory of the pan and tilt angles of the mirror-drive 2-DOF active vision system was generated as two 16-bit digital sequences at 200 kHz for a duration of τt + τb = 6.5 ms; 1310 16-bit data for each angle were updated with an 8-ms cycle time. The home positions of the pan and tilt angles of the mirror-drive 2-DOF active vision system were set to one end of their movable range such that θ 0 = 0, where their drive voltages were 0  V. Because of the narrow movable ranges of the pan and tilt angles of the 2-DOF active vision system, 0.17◦ and 0.14◦, respectively, the maximum speed of objects under observation without motion blur being incurred was determined theoretically by the ratio of the movable range of the duration time of the open exposure. Considering that the variations in the view angles via the mirrors correspond to twice those of the mirror angles, the max◦imum angu◦lar speeds for the pan and tilt angles are 67.1 /s and 49.7 /s, respectively. When the focal length of the zoom lens and the pixel pitch of the image sensor are f [mm] and x = 0.01  mm, respectively, one pixel corresponds 1to◦ co5r7r.e3stpaonn−d1s( tox/1f.)75≈f p0i.5xe7l3.f W−1h◦,enasfsu=m1i1n2g.5  mf≫m, thxe; maximum apparent speeds in the x and y directions on the image sensor for objects under observation without motion blur being incurred are 13.0 and 9.7 pixel/ms, respectively, corresponding to the displacements of 52.2 and 38.6 pixels in the x and y directions during an exposure time of 4  ms. When f = 650  mm, the maximum apparent speeds in the x and y directions on the image sensor for objects under observation without motion blur being incurred are 76.1 and 56.4 pixel/ms, respectively, corresponding to the displacements of 304 pixels and 225 pixels in the x and y directions during an exposure time of 4 ms. The binarization in Step 1 and the calculation of the moment features in Step 2 were implemented with parallel hardware logic for 8-bit gray-level images on the user-specific FPGA of the IDP Express board. The other steps were software-implemented as multithreaded processes with parallel executions on the PC. The execution time for Steps 1 and 2 was 0.108 ms and for Steps 3 and 4 0.887 ms. The total execution time was 1.01  ms. We confirmed that all the processes could be executed for 512 × 512 images in real time at 125 fps with an exposure time of 4 ms. Experiments Preliminary trajectory evaluation First, we conducted a preliminary experiment to verify the relationship between the input voltages to the piezo stages of the mirror-drive 2-DOF active vision system and its angular displacements in the pan and tilt directions when the active vision system was periodically operated on a designed trajectory at a frequency of 125  Hz. We determined the parameters (bφ , cφ ) and (bψ , cψ ), which are expressed in Eq. (6), of the trajectory of the active vision system during the time when the shutter is open, and quantified the nonlinear deviations with ripples in the pan and tilt trajectories. In the experiment, the periodic voltage wave at a cycle time of τ = 8 ms was inputted to the piezo stage, as shown in Fig. 6; the input voltage wave was set to a linear waveform from 0  V to a maximum voltage Vmax in a period τt = 4.5  ms, where Vmax was set to 15, 30, 45, 60, 75, 90, 105, 120, 135, and 150  V. To measure the pan and tilt angles of the active vision system, a laser beam spot for observation was redirected by the mirrors of the active vision system, and the locations of the laser beam spot projected on a screen at a distance of 4350 mm from the active vision system were extracted offline by capturing an HFR video at 10,000 fps. Figure  7 shows the angular displacements of the pan and tilt angles of the active vision system for 30 ms when the periodic input voltage waves, the maximum voltages of which varied from 0 to 150 V, at 125 Hz were applied to the piezo stages. In both the pan and tilt angles, the angular displacements were periodically changed at a frequency of 125 Hz in proportion to the amplitudes of the input voltage waves, whereas they involved certain ripple waves because of their resonant vibrations. The observed resonant frequencies in the pan and tilt angles were approximately 730 and 850  Hz, respectively; they were one-fifth or less of 3900  Hz, which is the resonant frequency of the piezo stage when no object is mounted on it. The decrease in the resonant frequencies was caused mainly the mirror attached to the piezo stage. It can be observed that the resonant frequency in the tilt angle was less than that in the pan angle, and the amplitude of a the ripple in the tilt angle was more than that in the pan angle, because the tilt angular motion was more strongly affected by gravity than the pan angular motion. When the angular trajectories during the exposure time τ = 4  ms were linearized by the least squares method, Fig.  8 shows the relationship between their inclinations and the input voltages to the piezo stages. It can be observed that the inclinations of the angular trajectories, which correspond to the apparent angular velocity of the target object, varied linearly with the amplitudes of the input voltages; the parameters in Eq.  (6) were estimated as (bφ , cφ ) =  (2.10, 4.47) and (bψ , cψ ) = (2.91, 1.61) for the pan and tilt angles, respectively. Figure  9 shows the relationship between the estimated angular speeds ω˜ = (ω˜ φ , ω˜ ψ ) and the averaged deviations ( φd , ψd ) from the approximated lines during 4  ms. In the figure, the ratio of the averaged deviation ( φd , ψd ) to the estimated angular displacement (φmv, ψmv) = (ω˜ φ τ , ω˜ ψ τ ) during the exposure time τ = 4ms, ( φd /φmv, ψd /ψmv), is also plotted; the ratio indicates the percentage by which our frame-byframe intermittent tracking method can reduce motion blur in shooting fast moving objects. When the maximum voltage of the input image was 150  V,◦the angular speeds and the averaged deviations are 49.7 /s and 1.91 × 10−2◦ for the pan angle and 67.1◦/s and 3.64 ×10−2◦ for the tilt angle; the ratios ( φd /φmv, ψd /ψmv) were 9.3 and 12.8  %. It can be observed that the deviation error from the approximate line becomes larger as the angular speeds become larger in both the pan and tile angles. The ratio ( φd /φmv, ψd /ψmv) was not so significantly changed with the estimated angular speeds, whereas the ratio of the tilt angle was larger than that of the pan angle because of the effect of gravity. Thus, we should consider image degradation with a certain motion blur with the above-mentioned ripple deviations in motion-blur-free video shooting at 125 fps. Circle‑dot motion at constant speeds Next, we conducted video shooting experiments for a circle-dot pattern to verify the relationship between the speed of an object and its motion blur. The pattern was moved along (1) the horizontal direction and (2) ◦ the oblique direction with an inclination of 20 , at constant speeds of 0, 250, 500, 750, and 1000  mm/s using a 1-DOF linear slider. In the experiment, the HTZ-11000 (Joble Co., Japan) was used as the CCTV zoom lens; its focal length was set to f = 650mm. The linear slider was located at a distance of 4350  mm from the mirrordrive 2-DOF active vision system; the 35 × 35 mm area on a plane at a distance of 4350 mm corresponded to an image region of 512 × 512 pixels, and 6.84 × 10−2 mm corresponded to one pixel. We can cancel motion blur during the 4 ms exposure when shooting a target object moving at 5.21 and 3.86  m/s on a plane 4350  mm in front of the mirror-drive 2-DOF active vision system in the vertical and horizontal direction, respectively, corresponding to its apparent motions at 304 and 225 pixels during the 4 ms exposure time in the x and y direction on the image sensor. Figure 10 shows (a) an overview of the experimental environment, (b) the circle-dot pattern to be observed, and (c) the configuration of the experimental setting. The 4-mm-diameter circle dots were blackprinted at intervals of 50 mm on a white sheet of paper. Figure  11 shows the 227 × 227 images cropped from the 512 × 512 input images so that the circle dot is located at their centers, and Fig. 12 shows the brightness profiles of 256 pixels on a horizontally intersected line of images when the circle dot moved at 0, 250, 500, 750, and 1000 mm/s in the horizontal direction. The threshold for binarization was IB = 50. As observed in Figs. 11 and 12, the input images captured with frame-by-frame intermittent tracking (IT) were compared with those captured without mechanical tracking (NT) and their motion deblurring (MD) images. The MD images were obtained by processing the NT images offline using a non-blind convolution method with a line kernel function [ 25 ]. The NT images became increasingly blurred in the horizontal direction as the speed of the circle dot increased, whereas the IT images remained almost entirely free of blurring regardless of the speed. Figure  13 shows the 227 × 227 images cropped from the 512 × 512 input images when the circle dot moved in the oblique direction. It can be seen that frame-by-frame intermittent tracking achieves motion-blur-free video shooting of the object moving in the oblique direction, as well as in the horizontal direction; the NT images are blurred in the 20◦ oblique direction, whereas the IT images are without blur at all the slider speeds. In the MD images, most of motion blurs were remarkably reduced, whereas certain ghost errors remained in the moving directions especially when the circle-dot moved by dozens of pixels during the camera shutter was open. This is because it is difficult for deconvolution-based methods to completely reduce large motion blurs for nonlinear brightness images with zero or saturation. To evaluate the degree of motion blur of the observed the circle dot, the index = + − − was introduced; + a b c target object (2) oblique direction circle dot CCTV zoom lens (HTZ11000) camera head linear slider mirror-drive 2-DOF active vision 50 mm 4 mm diameter 20 deg (1) horizontal direction mirror-drive active vision zoom lens tilt-mirror camera head 4350 mm pan-mirror Fig. 10 Experimental environment and circle-dot pattern to be evaluated. a Overview. b Circle dots to be evaluated. c Experimental setting 0 mm/s and − represent the lengths of the major and minor axes of the approximated ellipse of the circle dot in the image. The index increases as the motion blur increases in the image, and is zero when the dot is a perfect circle in the image. + and − were estimated offline by calculating the zero-, first-, and second-order moment features for the circle-dot region in the image, which was extracted by binarization with a threshold of 63. Considering the offset 0 = 2.6 pixel when no motion is present, the blur index ′ = − 0 was evaluated for the IT and NT images in Figs.  11 and 13. Figure  14 shows the relationsshpiep′efdbosretotwhfee5e0InTstehalenecdstpeNdeeTidmimoafgaeagsec.siT;rchlee ′dbwolutarsainnadvdeeirtxasgbeldu′rffooirrndtthhexee IT images was remarkably low at all the speeds as compared with that for the NT images; it became larger as tmhoevsepdeeind tohfethheocriizrcolnetadlodtiirneccrteioanse,dt.hWebhleunr itnhdeecxircle′ dfoort the IT images was 0.9, 2.3, 3.1, and 2.7 pixel at 250, 500, 7153.03,, 1an4.d3, 11010.50,  amnmd7/s.4,%reosfptehcetivreeslyp;ecthtiivse cvoarlureesopfonds′ fotor the NT images. When the circle-dot moved in the oblique direction, ′ for the IT images was 0.1, 1.9, 2.6, and 2.3 pixel at 250, 500, 750, and 1000  mm/s, respectively; this cvoalrureesopfonds′ tfoor1.t6h,e14N.9T, 1i m2.2ag,easn.dIn7.5th%e oefxtpheerirmesepnetc,ttihvee speed of the circle dot was 1 m/s or less, which is considerably lower than the maximum motion-blur-free speeds of 5.21 m/s in the horizontal direction and 3.86 m/s in the vertical direction, and our frame-by-frame intermittent tracking method noticeably reduced motion blur of circle dots moving at all the speeds in video shooting with the exposure time of 4  ms, whereas slight motion blur remained in the IT images because of nonlinear deviations with ripples on the trajectory of the mirror-drive 2-DOF active vision system. Table tennis ball motion at constant speeds Next, we conducted video shooting experiments for fast moving table tennis balls launched by a table tennis machine to verify motion blur when the speed of the object to be observed is larger than the maximum motion-blur-free speed of our mirror-drive 2-DOF active vision system. Figure  15 shows (a) an overview of the experimental environment, and (b) the 40-mm-diameter table tennis balls that were observed. The table tennis machine (TSP Hyper S-2, Yamato Takkyu Co., Japan) was installed 4350  mm in front of the mirror-drive 2-DOF active vision system, and a table tennis ball (plain) was launched in (1) the horizontal direction, ◦and (2) the oblique direction with an inclination of 20 at constant 0 mm/s speeds of 3, 4, 5, 6, and 7 m/s. In the experiment, a CCTV lens of f = 75  mm was used with a 1.5× extender; the 200 × 200 mm area on a plane at a distance of 4350 mm corresponded to an image region of 512 × 512 pixels and 0.391 mm corresponded to one pixel. When observing an object moving fast on a plane of 4350 mm in front of the mirror-drive 2-DOF active vision system, the maximum motion-blur-free speeds were 5.21 m/s in the horizontal piezo tilt-stage (tilt angle) IDP Express camera head direction and 3.86  m/s in the vertical direction, corresponding to its apparent motions at 52.2 and 38.6 pixels during the exposure time of 4 ms. Figures 16 and 17 show the 227 × 227 images cropped from the 512 × 512 input images [(a) IT images, (b) NT images] so that the table tennis ball is located at their centers when it is thrown in the horizontal direction and oblique direction. As compared with the input images captured when a table tennis ball was thrown at 3, 4, 5, 6, and 7 m/s, the input image of a motionless table tennis ball (0  m/s) is illustrated. The threshold for binarization in frame-by-frame intermittent tracking was IB = 50. It can be seen that the IT images remained almost blurfree, regardless of the speed, and they were similar to the input images captured when the ball speed was 0  m/s, whereas the motion blur of the table tennis balls in the NT images increased in both their moving directions as their speed increased. tabFlieguteren n18issbhaollwasntdheitrseblalutironinsdheipx betw′feoernththeeITspaenedd oNfTa 0 m/s 3 m/s 4 m/s 5 m/s 6 m/s 7 m/s 0 m/s 3 m/s 4 m/s 5 m/s 6 m/s 7 m/s 0 m/s 3 m/s 4 m/s 5 m/s 6 m/s 7 m/s 0 m/s 3 m/s 4 m/s 5 m/s 6 m/s 7 m/s images. Considering the offset 0 = 1.22  pixel in the case of no motion, ′ of those for 50 selected images, which were binarized with a threshold of 55, was averfcaogirrecdtleh,-eidnoNatTmmiaomntianogener.ss,Aimtshieclaobrmlutporatirhneaddteiwxnitthhe′thfeoexrpbtelhrueirmIiTennditemxuasginegs′ was remarkably low at all the speeds in the horizontal and oblique directions. The blur index ′ for the IT images at 3, 4, 5, 6, and 7 m/s in the horizontal direction was 0.05, 0.26, 0.20, 0.70, and 2.20 pixel, respectively, which corresponds to 1.1, 2.6, 1.1, 2.7, and 6.5% of the resp′feocrtitvheevIaTluimeoafges a′tfo3r, 4th,5e,N6,Ta nimd a7g  mes/.sTinhethbeluorbilniqdueex direction was 0.05, 0.42, 0.71, 1.63, and 2.33 pixel, respectively, which corresponds to 0.8, 3.2, 3.8, 6.4, and 6.8% of the respective value of ′ for the NT images. The ′ blur index for the IT images showed a tendency to increase slightly when shooting a video of a table tennis ball thrown at 6 and 7 m/s in the horizontal and oblique directions. This is mainly because the speed of the table tennis ball was so much higher than the maximum motion-blur-free speed (5.21 m/s in the horizontal direction, 3.86  m/s in the vertical direction) that the moving distance during an exposure time of 4  ms exceeded the upper limit of the movable range of the 2-DOF mirrordrive active vision system. Table tennis ball motion at variable speeds Next, we show the experimental results for the video of a table tennis ball identical to that used in the previous subsection, when the table tennis ball was alternately launched from the table tennis ball machine at different speeds of 3 and 5  m/s at intervals of 0.5  s. Figure  19a shows the 2-s temporal changes of the estimated speed and blur index ′ for the IT images with frame-by-frame intermittent tracking, as compared with (b) those for the NT images when the table tennis balls were passing in front of the mirror-drive 2-DOF active vision system in a manner similar to that when capturing the IT images. Corresponding to the launching interval of a table tennis ball and its passing time duration over a whole image region of 512 × 512 pixels, the speeds of the table tennis balls in images were discontinuously estimated in time; the passing time durations were 66.7 and 40.0 ms when a table tennis ball was thrown at 3 and 5 m/s, respectively; they correspond to the duration times for capturing eight and five frame images at 125 fps, respectively. It can be seen that the ball speed was estimated as a pulse wave, in which a 3-m/s-amplitude pulse of 66.7-msawltiderthnataenlyd aat in5-temrsv-aalsmopflit0u.5d es,oafnd40t-hmesb-wluirdtihndaepxpear′ for the NT images also alternated between 7 and 20 pixels. The blur index ′ for the IT images became a certain large value of 7 and 20 pixels exactly when the table tennis ball thrown at 3 and 5 m/s appeared in the image, whereas it was remarkably reduced around 1 pixel dozens of milliseconds after its appearance in the image; this corresponds to the duration time for capturing two frame images at 125  fps. The latency in motion blur reduction is caused mainly by (1) the time delay in frame-by-frame intermittent tracking, involving a one-frame-delay in estimating the ball speed using image features computed at the previous frame and a one-frame delay in reflecting it to the pan-tilt actuation of the 2-DOF mirrordrive active vision system, and (2) underestimated speed exactly when the table tennis ball appears in the field of camera view because of its partial appearance at the right side of the images. Figure  20 shows (a) a sequence of the images with frame-by-frame intermittent tracking, and (b) a sequence of the images without tracking when a table tennis ball with printed patterns, as illustrated in Fig.  15b, thrown at 3  m/s in the horizontal direction, was passing over the whole image region from right to left, taken at intervals of 16  ms; the upper images are the 512 × 512 input images, and the lower ones are the 132 × 132 images cropped from them so that the table tennis ball is located at their centers. It can be seen that the NT images are too heavily blurred to allow recognition of the letter patterns printed on the table tennis ball in all the frames. For the IT images, the input image was largely blurred at the start frame when the table tennis ball appeared at the right side in the image, whereas the blurring of the input images in all the remaining frames was reduced to the extent that the letter pattern of “hello, world!” at the center of the table-tennis ball can always be recognized. Nevertheless, a two-frame delay remains in frame-byframe intermittent tracking for motion blur reduction. Our system can capture less-blurred input images with a dozens-of-millisecond delay for a table tennis ball thrown a b Fig. 19 Estimated speed and motion-blur index when table tennis balls were thrown at variable speeds. a With tracking (IT). b Without tracking (NT) a b whole images of 512x512 pixels cropped images of 133x133 pixels cropped images of 133x133 pixels 0.000 s 0.016 s 0.032 s 0.048 s 0.064 s Fig. 20 Series of images captured when a table tennis ball was thrown in the horizontal direction. a With tracking (IT). b Without tracking (NT) at 8.0  m/s or less; its passing time over a whole image region of 512 × 512 pixels was larger than 24 ms for capturing three frame images at 125 fps. Conclusion In this study, we developed a motion-blur-free video shooting system based on a concept of frame-by-frame intermittent tracking, in which the control of the camera shutter state is alternated at a rate of hundreds of fps. The target’s speed in images is controlled at zero during exposure; otherwise, the camera’s position returns to its home position. Our system can capture 512 × 512 images of fast moving objects at 125 fps with an exposure time of 4  ms without motion blur being incurred by controlling the pan and tilt directions of a mirror-drive 2-DOF active vision system using high-speed video processing. The system’s performance was verified by conducting several experiments using fast moving objects. We focused on motion blur reduction for moving objects in uniform backgrounds in this study, whereas both moving objects and static backgrounds can be clearly observed without blurring when the mirror speed is alternatively switched from the target object’s speed to zero in frame-by-frame intermittent tracking so that image capturing “with tracking” (IT) and “without tracking” (NT) can be simultaneously conducted. Currently, the limited responses of the piezo actuators become the major bottleneck in frame-by-frame intermittent tracking at a higher frame rate; the duration time for back-to-home-control is 2 ms or more on our system, whereas the duration time for vision-based tracking control was set to approximately 4 ms, as shown in the angular displacements in Fig. 7. On the basis of these results, we plan to improve our motionblur-free video shooting system by adapting it for video shooting of fast moving objects in complex scenes with improved accuracy using fast general-purpose motion detection algorithms and faster frame-by-frame intermittent tracking using a free-vibration-type actuator such as a resonant mirror vibrating at hundreds or thousands of hertz, to apply our motion-blur-free video shooting system to highly magnified observations of fast moving scenes in various applications, such as the precise inspection of products moving fast on a conveyor line and tunnel and road inspection from a car moving at a high speed. Authors’ contributions MI carried out the main part of this study and drafted the manuscript. MJ and YM set up the experimental system of this study. TT and II contributed concepts of this study and revised the manuscript. All authors read and approved the final manuscript. Acknowledgements and funding A part of this research was supported by the Adaptable and Seamless Technology Transfer Program through Target-Driven R&D (No. AS2615002J), JST, Japan. Competing interests The authors declare that they have no competing interests. Availability of data and materials Not applicable. Consent for publication Not applicable. Ethics approval and consent to participate Not applicable. Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. 1. Kundur D , Hatzinakos D ( 1996 ) Blind image deconvolution . IEEE Signal Process Mag 13 ( 3 ): 43 - 64 2. Campisi P , Egiazarian K ( 2007 ) Blind image deconvolution: theory and applications . CRC Press, Boca Raton 3. KuKim S , KiPaik J ( 1998 ) Out-of-focus blur estimation and restoration for digital auto-focusing system . Electron Lett 34 ( 12 ): 1217 - 1219 4. Fergus R , Singh B , Hertzmann A , Roweis ST , Freeman WT ( 2006 ) Removing camera shake from a single photograph . ACM Trans Graphic 25 ( 3 ): 787 - 794 5. Tai Y-W , Tan P , Brown MS ( 2011 ) Richardson-Lucy deblurring for scenes under projective motion path . IEEE Trans Pattern Anal Mach Intell 33 ( 8 ): 1603 - 1618 6. Bascle B , Blake A , Zisserman A ( 1996 ) Motion deblurring and super-resolution from an image sequence . In: Proceedings of European conference on computer vision , pp 571 - 582 7. Farsiu S , Robinson MD , Elad M , Milanfar P ( 2004 ) Fast and robust multiframe super resolution . IEEE Trans Image Proc 13 ( 10 ): 1327 - 1344 8. Chen J , Yuan L , Tang C-K , Quan L ( 2008 ) Robust dual motion deblurring . In: Proceedings IEEE conference computer vision pattern recognition , pp 1 - 8 9. Joshi N , Kang SB , Zitnick CL , Szeliski R ( 2010 ) Image deblurring using inertial measurement sensors . ACM Trans Graphic 29 ( 4 ): 30 10. Kim MD , Ueda J ( 2015 ) Dynamics-based motion de-blurring for a PZTdriven, compliant camera orientation mechanism . Int J Robotics Res 34 : 653 - 673 11. Canon IS technology. http://www.usa.canon.com/cusa/consumer/ standard_display/Lens_Advantage_IS/ 12. Nikon 's effective VR system . http://imaging.nikon.com/lineup/lens/ concept/vr/en/ 13. Sony SteadyShot™ INSIDE . http://www.sony-mea.com/microsite/dslr/10/ tech/steadyshot.html 14. Watanabe Y , Komuro T , Ishikawa M ( 2007 ) 955 -fps real-time shape measurement of a moving/deforming object using high-speed vision for numerous-point analysis . In: Proceedings IEEE international conference on robotics and automation , pp 3192 - 3197 15. Ishii I , Tatebe T , Gu Q , Moriue Y , Takaki T , Tajima K ( 2010 ) 2000 fps real-time vision system with high-frame-rate video recording . In: Proceedings IEEE international conference robotics and automation , pp 1536 - 1541 16. Ishii I , Taniguchi T , Yamamoto K , Takaki T ( 2012 ) High-frame-rate optical flow system . IEEE Trans Circuits Syst Video Technol 22 ( 1 ): 105 - 112 17. Ishii I , Ichida T , Gu Q , Takaki T ( 2013 ) 500 -fps face tracking system . J Real Time Image Process 8 ( 4 ): 379 - 388 18. Namiki A , Hashimoto K , Ishikawa M ( 2003 ) A hierarchical control architecture for high-speed visual servoing . Int J Robotics Res 22 ( 10 -11): 873 - 888 19. Okumura K , Yokoyama K , Oku H , Ishikawa M ( 2015 ) 1 ms auto pan-tiltvideo shooting technology for objects in motion based on Saccade Mirror with background subtraction . Adv Robotics 29 ( 7 ): 457 - 468 20. Oku H , Ishii I , Ishikawa M ( 2000 ) Tracking a protozoon using high-speed visual feedback . In: Proceedings 1st annual conference on microtechnologies in medicine and biology , pp 156 - 159 21. Gu Q , Aoyama T , Takaki T , Ishii I ( 2015 ) Simultaneous vision-based shape and motion analysis of cells fast-flowing in a microchannel . IEEE Trans Automat Sci Eng 12 ( 1 ): 204 - 215 22. Hayakawa T , Watanabe T , Ishikawa M ( 2015 ) Real-time high-speed motion blur compensation system based on back-and-forth motion control of galvanometer mirror . Opt Express 23 ( 25 ): 31648 - 31661 23. Hayakawa T , Ishikawa M ( 2016 ) Development of motion-blur-compensated high-speed moving visual inspection vehicle for tunnels . Int J Struct Civil Eng Res 5 ( 2 ): 151 - 155 24. Ueno T , Gu Q , Aoyama T , Takaki T , Ishii I , Kawahara T ( 2015 ) Motion-blurfree microscopic video shooting based on frame-by-frame intermittent tracking . In: Proceedings IEEE international conference automation science engineering , pp 837 - 842 25. Levin A , Sand P , Cho TS , Durand F , Freeman WT ( 2008 ) Motion-invariant photography . ACM Trans Graph 27 ( 3 ): 71