ImageTracker is software that can be used to stabilize a series of color or gray scale images that were taken using a non-stable (vibrating or moving) camera. ImageTracker can perform the following operations:
ImageTracker cannot remove motion blur.
The image transformations that ImageTracker applies to match images onto the reference image assume that the observed scene is flat. This assumption is usually not quite true (even in the Netherlands). A non-flat scenery combined with a moving camera location makes perfect registration impossible and may cause several local optimal sets of parameter values to exist. In practice the results can be quite impressive for non flat scenery and a moving camera location.
The stabilization operates on the entire image. This requires that most of the objects visible in the scene are fixed. For traffic observations (the primary purpose of this software), only vehicles and pedestrians are supposed to move and most other objects in the scene are stationary. The ImageTracker software allows the camera to move along the scene. This is done by extending the internal copy of the reference image as new scenery comes into view.
ImageTracker is written in Java and should run on any computer for which there is a Java Virtual Machine. The software can take advantage of multiple core and multiple CPU computers; in fact it should be able to scale nicely to hundreds of cores when such hardware becomes available. Image stabilization is a very CPU-intensive operation. On a 2008 quad-core desktop computer, stabilization of 5 mega-pixel images can take one minute per image. Larger jobs can be split into multiple smaller ones and divided over several computers. In this case approximate initial values for the search parameters must be provided by the user.
ImageTracker can read almost any image file type. If the images were acquired with a video camera, it may be necessary to de-interlace them first. De-interlacing should be done by interpolating the skipped lines or doubling the non-skipped lines. Simply leaving out the skipped lines will not work as the remaining image would have non-sqpare pixels (resulting in problems when the camera rotates during the image sequence).
The stabilized images are written in PNG format. The output file names are the same as the input ones, but prefixed with an "s" (for stabilized). The extension is changed to "png" if the input one was different.
On start-up the program displays a single window with 5 tabs:
Normally the user would visit these tabs in this order.
This tab contains all the tools to compute the required lens correction. Without proper lens correction, stabilization can be problematic. Even the best lenses exhibit significant distortion near the edges of the image. The ImageTracker program can be used without lens correction (by not loading a calibration image in this tab), but the results may be disappointing.
The calibration image should be a checkerboard pattern. A sheet of vinyl floor covering is suitable for this purpose. The reference image must be taken with the camera pointing straight onto the checkerboard pattern (the sheet must be flat and the line from the camera to the center of the image must be perpendicular to the sheet. The checkerboard grid must run approximately parallel to the width and height directions of the camera. The image should enclose about 30 x 20 squares of the pattern. As you can see on the calibration image below, lightning need not be perfectly uniform. (This image was reduced in size to 25%; the original is 2448x2050 pixels.)
In the Lens calibration tab click on the Open button and select the lens calibration image. After loading the image the software will locate all the cross-points in the image and try to line those up in rows and columns. Once finished, the user can inspect the results by selecting any of the six radio buttons:
|Raw image||The original image|
|Analized image||A gray-scale image with dark and light patches near the cross-points of the raw image The pixels with a locally extreme value are indicated with red and green cross-hair markers.|
|Image with calibration points||The original image with diagonal cross hair markers.|
|Corrected image||The straightened image. Usually some pixels in the straightened images do not reverse-map onto the raw image. These pixels are rendered in Red.|
|Statistics||Some indicators of the quality of the calibration; see below.|
|Table with calibration points||A table with the coordinates (in pixels) of the calibration points.|
The statistics information gives a short summary of the quality of the lens correction that was derived. The Standard deviation of distance should be less than 1% of the Mean distance. The difference between the Longest distance and the Shortest distance should be much smaller than the Mean distance. The number of incomplete pairs should be small relative to the number of point pairs.
Divide the width by the number of pixels in that direction of the image. This gives the size of a pixel in mm.
Besides the lens correction image, ImageTracker needs to know the viewing angle of the camera. This is expressed in a peculiar way; the distance from the camera to the center of the image measured in pixels. This can be measured while taking the lens calibration image, but it can also be done in a separate setup. To measure the size of a pixel in mm, point the camera straight (perpendicular) to a flat surface. Now attach two sticky memo stickers on the surface near the extreme left and right of the area observed by the camera. Put those stickers roughly along the center line of the image. Use a tape measure to measure the length of the observed center line in mm (call this value width). Also use the tape measure to measure the distance of the camera to the middle of that center line (call this value distance).
The image that will be used as reference image must be in the same directory as the images that will be stabilized.
Click on the button Select first image file ... and navigate to the directory containing the image files to be stabilized. Select the first image file to be stabilized. Then click Select last image file ... and select the last image file to be stabilized.
If the reference image is not within the series of images to be stabilized, you must select the reference image as the first (or last) image.
This tab contains all the controls that affect how a stabilized image is saved. The user should use these settings to optimize the size of the stabilized images and select only the region of interest. The computation time of the subsequent processing (such as searching for moving objects, that can be achieved by the ObjectFinder software) is directly impacted by the size of the stabilized images. To set the parameters of the output images, the user should select the reference image in the Select input image files tab. Going back on the Select output directory and image properties, the reference image should be displayed. The user can now set the properties of the stabilized images. The appearance of the output image is updated once a parameter is changed. Please note that no stabilization is made at this point, so the user may be disappointed when selecting another image to display, because it may not display the region of interest he chose. But it does not matter as long as he set it on the reference image. The settings are divided in 5 groups (sometimes the window must be widened in order to show all the groups; this is a bug in Java FlowLayout).
This group contains only one button. Click it to select the directory where the stabilized images must be written. In addition to the stabilized images, the ImageTracker also creates a series of small images that, together, constitute the reference image and a text file result.txt that contains the values of the parameters used to correct each image. The current version of ImageTracker also writes some other files to this directory for debugging purposes. These can be ignored.
This group contains two boxes to set the Width and Height of the output images. If the camera was moved along the landscape, make sure that the output images will be large enough to contain the entire "flight". Pixels that were not observed in a particular image will be output as Red pixels (R,G,B = 255,0,0).
This group contains two boxes to change Contrast and Brightness. Contrast is a floating point value that is used as a multiplier for the pixel values. Brightness is added to the pixel values after multiplication with Contrast. Finally the values for R, G and B are clipped and rounded to obtain integer values in the range 0..255.
This group contains four boxes to set a translation (X offset and Y offset in pixels), Rotation (counter-clockwise in degrees) and Enlargement.
This group contains two boxes for wedge (key stone) distortion (Wedge X, Wedge Y). The Wedge values can be used to correct if the pictures of the camera were not taken straight down. Wedge X is a rotation along the X-axis of the camera, Wedge Y is a rotation along the Y-axis of the camera. Orthorectification can be performed by setting these values. With the correct values, objects on the ground that were the same distance apart should be the same distance apart and objects on the ground that are at right angles should be at right angles. The elevation value entered in the Lens calibration tab must be correct for orthorectification to work. (It is, in fact, possible to find a reasonably close approximation of the elevation by trial and error.)
This tab allows the user to change the ranges and step-sizes of the parameters that are varied to match an image on the reference image. The program uses the values used for the preceding image as initial guess for the current image. So these maximum changes must be sufficient to handle the difference between two successive image. The parameters are changed in rounds. Each round, the step size of each parameter is halved. For each parameter there is a maximum precision at which it can reasonably be tuned. This precision corresponds to a change of about a tenth of the size of a pixel.
The parameters are:
|Parameter||Step size in first round||First round||Last round||Maximum change||Smallest step size|
|X-Offset||2.0 pixels||0||5||120 pixels||0.06 pixels|
|Y-Offset||2.0 pixels||0||5||120 pixels||0.06 pixels|
|Enlargement||0.0002||4||5||0.01 (1%)||0.0001 (0.01%)|
|X-Speed||0.00005 pixels/line-time||4||5||0.02 pixels/line-time||0.000025 pixels/line-time;|
|Y-Speed||0.00005 pixels/line-time||4||5||0.02 pixels/line-time||0.000025 pixels/line-time;|
If these default settings perform badly (tracking is lost, or does not quickly enough catch up with changes of the camera view) the user can change these.
The last two parameters are needed to correct image distortion that happens in video cameras with CMOS sensors when the camera tilts or pans. If the images were not acquired with such a camera, change the value of first round for these parameters into a value that exceeds the value of last round. This prevents ImageTracker from attempting to optimize these parameters.
This tab contains the controls to set up tracking, start it and observe the results. On the right is displayed the difference between the reference image and the current processed image (or the start image before running the program). If the reference image is the same as the current image, the difference is a uniform gray panel. If they are different, the scene should be visible (because of lighting difference from one image to another), but it should appear in gray scale.
The ImageTracker can only track limited changes in camera position between two successive images. If the reference image is not the first image in the series, the user should enter the approximate initial values for the six parameters in the first six boxes in order to align the reference image with the start image. The alignment would be correct when most of the pixels in the difference image would be gray.
The last three boxes allow the user to change the Reference image (default 0), Start image (default 0) and the End image (default 99999).
Clicking on the Run button starts the stabilization process. Clicking on Cancel stops it (it can take a couple of seconds to terminate the operation).
The Refresh rate can be altered to limit the maximum number of intermediate difference images that are shown on the screen. A larger interval reduces the CPU time that is consumed to display these difference images.
If the ImageTracking operation is restarted with a different reference image, the user must remove the tile files in the Output directory (or change the output directory under the Select output directory and image properties tab to a new, empty, directory).
The ImageTracker writes the found values for the six alignment parameters in a file results.txt in the Output directory. These values can be considered checkpoints for restarting (after e.g. a power failure).
Set up the ImageTracker exactly like it was before the run was aborted. (Same lens calibration image, input image list, output directory, output image properties, same reference image.) In the Run tab enter the rank number of the last successfully stabilized image as Start image. Copy the six values for X-Offset etc. from the results.txt file. This should result in good registration of that image. Finally click Run to resume the image stabilization process.
The ImageTracker program can almost entirely be set up and started using command line parameters. The only exception are the values under Alignment settings.
The following program arguments are understood:
|FirstImage=full_path_and_file_name||Set the first image file name|
|LastImage=full_path_and_file_name||Set the first image file name|
|OutputDirectory=path_name||Set the output directory|
|OutputWidth=width_in_pixels||Set the width of the output images|
|OutputHeight=height_in_pixels||Set the height of the output images|
|OutputContrast=contrast_value||Set the contrast factor of the output images|
|OutputBrightness=brightness_value||Set the brightness value of the output images|
|OutputXOffset=X_offset_in_pixels||Set the X-Offset of the output images|
|OutputYOffset=Y_offset_in_pixels||Set the Y-Offset of the output images|
|OutputRotation=rotation_in_degrees||Set the rotation value of the output images|
|OutputEnlargement=factor||Set the enlargement factor of the output images|
|OutputWedgeX=rotation_in_degrees||Set the X-Wedge value of the output images|
|OutputWedgeY=rotation_in_degrees||Set the Y-Wedge value of the output images|
|Elevation=elevation_in_pixels||Set the elevation value. Warning: this value is ignored if a lens correction image is loaded that has it's own elevation in an accompanying .ele file.|
|InitialXOffset=X_offset_in_pixels||Set the initial X-offset value (Run tab)|
|InitialYOffset=Y_offset_in_pixels||Set the initial Y-offset value (Run tab)|
|InitialRotation=rotation_in_degrees||Set the initial rotation value (Run tab)|
|InitialEnlargement=factor||Set the initial enlargement value (Run tab)|
|InitialWedgeX=wedge_X_in_degrees||Set the initial wedge-X value (Run tab)|
|InitialWedgeY=wedge_Y_in_degrees||Set the initial wedge-Y value (Run tab)|
|ReferenceImage=rank_of_reference_image||Set the reference image|
|StartImage=rank_of_start_image||Set the start image|
|EndImage=rank_of_end_image||Set the end image|
|LensCalibration=full_path_and_file_name||Load the lens calibration image (and, if present, the elevation from the accompanying .ele file)|
|MaxThreads=number_of_threads||Set the maximum number of threads to use in the internal image processing functions. This value cannot be changed in the graphical user interface. Currently the default is 20 which seems to work well for quad and hex core CPUs.|
|Run||Start the stabilization process. There should be no = at the end of this program argument.|