ImageTracker is software that can be used to stabilize a series of color or gray scale images that were taken using a non-stable (vibrating or moving) camera. ImageTracker can perform the following operations:

1. Correct lens distortion (requires an image of a checkerboard pattern and knowledge of the viewing angle of the camera; the camera hardware and lens settings must be the same as used for the images that will be lens-corrected and stabilized).
2. Search for the image transformation that matches each image of a series as close as possible onto a reference image.
3. Apply an optional output transformation to each stabilized image and write (part of) the observed scene to a numbered file in a user specified output directory. The output transformation can perform orthorectification; correct the image in case the camera was not pointing straight down onto the scene.

ImageTracker cannot remove motion blur.

The image transformations that ImageTracker applies to match images onto the reference image assume that the observed scene is flat. This assumption is usually not quite true (even in the Netherlands). A non-flat scenery combined with a moving camera location makes perfect registration impossible and may cause several local optimal sets of parameter values to exist. In practice the results can be quite impressive for non flat scenery and a moving camera location.

The stabilization operates on the entire image. This requires that most of the objects visible in the scene are fixed. For traffic observations (the primary purpose of this software), only vehicles and pedestrians are supposed to move and most other objects in the scene are stationary. The ImageTracker software allows the camera to move along the scene. This is done by extending the internal copy of the reference image as new scenery comes into view.

ImageTracker is written in Java and should run on any computer for which there is a Java Virtual Machine. The software can take advantage of multiple core and multiple CPU computers; in fact it should be able to scale nicely to hundreds of cores when such hardware becomes available. Image stabilization is a very CPU-intensive operation. On a 2008 quad-core desktop computer, stabilization of 5 mega-pixel images can take one minute per image. Larger jobs can be split into multiple smaller ones and divided over several computers. In this case approximate initial values for the search parameters must be provided by the user.

Input files

ImageTracker can read almost any image file type. If the images were acquired with a video camera, it may be necessary to de-interlace them first. De-interlacing should be done by interpolating the skipped lines or doubling the non-skipped lines. Simply leaving out the skipped lines will not work as the remaining image would have non-sqpare pixels (resulting in problems when the camera rotates during the image sequence).

Output files

The stabilized images are written in PNG format. The output file names are the same as the input ones, but prefixed with an "s" (for stabilized). The extension is changed to "png" if the input one was different.

Use of the ImageTracker program

On start-up the program displays a single window with 5 tabs:

1. Lens calibration
2. Select input image files
3. Select output directory and image properties
4. Alignment settings
5. Run

Normally the user would visit these tabs in this order.

Lens calibration

This tab contains all the tools to compute the required lens correction. Without proper lens correction, stabilization can be problematic. Even the best lenses exhibit significant distortion near the edges of the image. The ImageTracker program can be used without lens correction (by not loading a calibration image in this tab), but the results may be disappointing.

The calibration image should be a checkerboard pattern. A sheet of vinyl floor covering is suitable for this purpose. The reference image must be taken with the camera pointing straight onto the checkerboard pattern (the sheet must be flat and the line from the camera to the center of the image must be perpendicular to the sheet. The checkerboard grid must run approximately parallel to the width and height directions of the camera. The image should enclose about 30 x 20 squares of the pattern. As you can see on the calibration image below, lightning need not be perfectly uniform. (This image was reduced in size to 25%; the original is 2448x2050 pixels.)

In the Lens calibration tab click on the Open button and select the lens calibration image. After loading the image the software will locate all the cross-points in the image and try to line those up in rows and columns. Once finished, the user can inspect the results by selecting any of the six radio buttons:

The statistics information gives a short summary of the quality of the lens correction that was derived. The Standard deviation of distance should be less than 1% of the Mean distance. The difference between the Longest distance and the Shortest distance should be much smaller than the Mean distance. The number of incomplete pairs should be small relative to the number of point pairs.

Camera viewing angle

Besides the lens correction image, ImageTracker needs to know the viewing angle of the camera. This is expressed in a peculiar way; the distance from the camera to the center of the image measured in pixels. This can be measured while taking the lens calibration image, but it can also be done in a separate setup. To measure the size of a pixel in mm, point the camera straight (perpendicular) to a flat surface. Now attach two sticky memo stickers on the surface near the extreme left and right of the area observed by the camera. Put those stickers roughly along the center line of the image. Use a tape measure to measure the length of the observed center line in mm (call this value width). Also use the tape measure to measure the distance of the camera to the middle of that center line (call this value distance).

Divide the width by the number of pixels in that direction of the image. This gives the size of a pixel in mm.
Divide the measured distance by the computed size of a pixel to obtain the distance from the camera to the surface in pixels (call this value elevation)
This elevation is required when ImageTracker computes the distortion due to movement of the camera location. We recommend that a text file is created with the same name as the calibration image, but extension ".ele". This file must be stored in the same directory as the calibration image. When ImageTracker loads the lens calibration image it will look for the corresponding ".ele" file and - if it exists - load that too. Otherwise the user must enter the value manually in the Elevation box in the Lens calibration tab.

Select input image files

The image that will be used as reference image must be in the same directory as the images that will be stabilized.
Click on the button Select first image file ... and navigate to the directory containing the image files to be stabilized. Select the first image file to be stabilized. Then click Select last image file ... and select the last image file to be stabilized.
If the reference image is not within the series of images to be stabilized, you must select the reference image as the first (or last) image.

After selecting the first and last images the list under Image files will be populated. Clicking on a file name causes that file to be displayed. Checking the check box Apply lens correction causes the image to be displayed with lens correction. It is instructive to see how the image changes when this check box is (un-)checked. The effect resembles the "popping" of a color slide that heats up in a projector.

The status of the check box only affects the preview image in this tab.

Select output directory and image properties

This tab contains all the controls that affect how a stabilized image is saved. The user should use these settings to optimize the size of the stabilized images and select only the region of interest. The computation time of the subsequent processing (such as searching for moving objects, that can be achieved by the ObjectFinder software) is directly impacted by the size of the stabilized images. To set the parameters of the output images, the user should select the reference image in the Select input image files tab. Going back on the Select output directory and image properties, the reference image should be displayed. The user can now set the properties of the stabilized images. The appearance of the output image is updated once a parameter is changed. Please note that no stabilization is made at this point, so the user may be disappointed when selecting another image to display, because it may not display the region of interest he chose. But it does not matter as long as he set it on the reference image. The settings are divided in 5 groups (sometimes the window must be widened in order to show all the groups; this is a bug in Java FlowLayout).

Output directory

This group contains only one button. Click it to select the directory where the stabilized images must be written. In addition to the stabilized images, the ImageTracker also creates a series of small images that, together, constitute the reference image and a text file result.txt that contains the values of the parameters used to correct each image. The current version of ImageTracker also writes some other files to this directory for debugging purposes. These can be ignored.

Output image size

This group contains two boxes to set the Width and Height of the output images. If the camera was moved along the landscape, make sure that the output images will be large enough to contain the entire "flight". Pixels that were not observed in a particular image will be output as Red pixels (R,G,B = 255,0,0).

Color correction

This group contains two boxes to change Contrast and Brightness. Contrast is a floating point value that is used as a multiplier for the pixel values. Brightness is added to the pixel values after multiplication with Contrast. Finally the values for R, G and B are clipped and rounded to obtain integer values in the range 0..255.

Translate, rotate, enlargement

This group contains four boxes to set a translation (X offset and Y offset in pixels), Rotation (counter-clockwise in degrees) and Enlargement.

Wedge distortion

This group contains two boxes for wedge (key stone) distortion (Wedge X, Wedge Y). The Wedge values can be used to correct if the pictures of the camera were not taken straight down. Wedge X is a rotation along the X-axis of the camera, Wedge Y is a rotation along the Y-axis of the camera. Orthorectification can be performed by setting these values. With the correct values, objects on the ground that were the same distance apart should be the same distance apart and objects on the ground that are at right angles should be at right angles. The elevation value entered in the Lens calibration tab must be correct for orthorectification to work. (It is, in fact, possible to find a reasonably close approximation of the elevation by trial and error.)

Alignment settings

This tab allows the user to change the ranges and step-sizes of the parameters that are varied to match an image on the reference image. The program uses the values used for the preceding image as initial guess for the current image. So these maximum changes must be sufficient to handle the difference between two successive image. The parameters are changed in rounds. Each round, the step size of each parameter is halved. For each parameter there is a maximum precision at which it can reasonably be tuned. This precision corresponds to a change of about a tenth of the size of a pixel.
The parameters are:

If these default settings perform badly (tracking is lost, or does not quickly enough catch up with changes of the camera view) the user can change these.

X-Speed and Y-Speed

The last two parameters are needed to correct image distortion that happens in video cameras with CMOS sensors when the camera tilts or pans. If the images were not acquired with such a camera, change the value of first round for these parameters into a value that exceeds the value of last round. This prevents ImageTracker from attempting to optimize these parameters.

Run

This tab contains the controls to set up tracking, start it and observe the results. On the right is displayed the difference between the reference image and the current processed image (or the start image before running the program). If the reference image is the same as the current image, the difference is a uniform gray panel. If they are different, the scene should be visible (because of lighting difference from one image to another), but it should appear in gray scale.
The ImageTracker can only track limited changes in camera position between two successive images. If the reference image is not the first image in the series, the user should enter the approximate initial values for the six parameters in the first six boxes in order to align the reference image with the start image. The alignment would be correct when most of the pixels in the difference image would be gray.
The last three boxes allow the user to change the Reference image (default 0), Start image (default 0) and the End image (default 99999).
Clicking on the Run button starts the stabilization process. Clicking on Cancel stops it (it can take a couple of seconds to terminate the operation).
The Refresh rate can be altered to limit the maximum number of intermediate difference images that are shown on the screen. A larger interval reduces the CPU time that is consumed to display these difference images.
If the ImageTracking operation is restarted with a different reference image, the user must remove the tile files in the Output directory (or change the output directory under the Select output directory and image properties tab to a new, empty, directory).

Restarting the ImageTracker

The ImageTracker writes the found values for the six alignment parameters in a file results.txt in the Output directory. These values can be considered checkpoints for restarting (after e.g. a power failure).
Set up the ImageTracker exactly like it was before the run was aborted. (Same lens calibration image, input image list, output directory, output image properties, same reference image.) In the Run tab enter the rank number of the last successfully stabilized image as Start image. Copy the six values for X-Offset etc. from the results.txt file. This should result in good registration of that image. Finally click Run to resume the image stabilization process.

Command line parameters

The ImageTracker program can almost entirely be set up and started using command line parameters. The only exception are the values under Alignment settings.
The following program arguments are understood: