Friday, April 11, 2014

Testing the calibration accuracy

This is step 6 of 10 in the tutorial Collecting 3D shape data using StereoMorph

The previous section demonstrated how to use a checkerboard pattern of known square size and the dltCalibrateCameras() function to calibrate two cameras in stereo. While dltCalibrateCameras() returns calibration and reconstruction RMS errors, these are measures of how well the DLT camera model fits the calibration points and not the reconstruction accuracy per se. Moreover, the RMS errors are scale-independent. If we have not measured the calibration checkerboard square size correctly, the RMS errors will be unaffected but our reconstructions will be improperly scaled. This section will demonstrate how to use a checkerboard (ideally having a square size different from that used in the calibration step) to test the accuracy of a calibrated stereo camera setup.

1. Repeat the steps in “Creating a Checkerboard Pattern” to create another checkerboard of a different square size than that used in the calibration. This will allow you to test whether you have the proper scaling for the calibration checkerboard (if they had the same square size, we would not be able to test this). For this tutorial, I used a checkerboard printed at 9% scaling for the calibration and a checkerboard printed at 6% scaling to test the calibration accuracy.

Two checkerboards, printed at 9% and 6% scaling. In this tutorial, the cameras are calibrated with the left pattern and the calibration is tested with the right.
2. Repeat the steps in Measuring Checkerboard Square Size to measure the square size of the test checkerboard pattern. The 6% scaled test checkerboard in this tutorial has a square size of 4.233 mm when measured using a precision rule.

Measuring the square size of the test checkerboard pattern (21 x 14, printed at 6% scaling).
3. Take photographs of the new checkerboard pattern as in Calibrating Stereo Cameras , ensuring a sampling of points throughout the calibrated volume (anywhere the object you're digitizing could conceivably be).

Eight photos of the test checkerboard (a different size than the calibration checkerboard) per camera view used in the test calibration step. The tutorial includes a test checkerboard in a total of 11 different positions.
4. Import the photographs from different views into two different folders. For demonstration, we’ll use the test calibration photographs in the folder ‘Test Calibration images’ (in the StereoMorph Tutorial folder). Photographs from camera one are in the folder ‘v1’ and photographs from camera two are in the folder ‘v2’.

5. Load the StereoMorph library into the current R session and ensure that the StereoMorph Tutorial folder is your current working directory.

> library(StereoMorph)

Just as in the calibration step, we’ll use the findCheckerboardCorners() function to find the internal corners in each calibration image.

6. Specify the number of internal corners in the checkerboard.

> nx <- 21
> ny <- 14


7. Specify the file locations of the test calibration images, where to save the checkerboard corners and, if desired, where to save the verification images.

> test_image_file <- paste0('Test images/v', c(1, 2))
> test_corner_file <- paste0('Test corners/v', c(1, 2))
> test_verify_file <- paste0('Test images verify/v', c(1, 2))


8. Call findCheckerboardCorners().

> corners <- findCheckerboardCorners(image.file=test_image_file, nx=nx, ny=ny, corner.file=test_corner_file, verify.file=test_verify_file)

The function will find the corners successfully for all but three of the images in the tutorial set. Again, don’t worry if the function fails for a couple of images; these will be ignored in subsequent steps.

9. Once findCheckerboardCorners() has finished running, check all of the verification images to be sure that the order is consistent within each camera view and between the two views. For the tutorial images the corners in the first view are in the reverse order relative to the order in the second view (as was also seen with the calibration images). This will be corrected when the corners are read in from the corner files.

10. Import the corners just saved to individual files by constructing a two-column matrix of file paths, where the columns correspond to each view.

> corners_by_file <- cbind(
paste0(test_corner_file[1], '/', paste0('DSC_00', 11:20, '.txt')),
paste0(test_corner_file[2], '/', paste0('DSC_00', 11:20, '.txt')))


11. Call readCheckerboardsToArray() to read all of the corner matrices into an array; include all files – empty files will read in as NAs.

> test_corners <- readCheckerboardsToArray(file=corners_by_file, nx=nx, ny=ny, col.reverse=c(F, T), row.reverse=c(F, T))

As in the calibration step, the vector c(F, T) is passed to col.reverse and row.reverse to reverse the order of the corners from the second view relative to the first.

Since we are importing corners from 10 images, the third dimension of the test_corners array is 10.

> dim(test_corners)
[1] 294 2 10 2


12. Set grid_size_test to the square size of the test calibration checkerboard. The tutorial test calibration checkerboard, measured using a precision ruler (see Measuring Checkerboard Square Size ), is approximately 4.323 mm.

> grid_size_cal <- 4.2327

13. Load in the calibration coefficients.

> cal.coeff <- as.matrix(read.table(file="cal_coeffs.txt"))

14. Call dltTestCalibration(). Image pairs for which corners were found in neither or only one image will be ignored.

> dlt_test <- dltTestCalibration(cal.coeff=cal.coeff, coor.2d=test_corners, nx=nx, grid.size=grid_size_test)

15. Use the summary() function to print a summary of the accuracy test.

> summary(dlt_test)

dltTestCalibration Summary
    Number of grids: 7
    Number of points: 1029
    Aligned ideal to reconstructed (AITR) point position errors:
        AITR RMS Errors (X, Y, Z): 0.02343, 0.01799, 0.02063
        Mean AITR Distance Error: 0.03297
        AITR Distance RMS Error: 0.03605
    Inter-point distance (IPD) errors:
        IPD RMS Error: 0.03090397
        IPD Mean Absolute Error: 0.02392322
        Mean IPD error: -0.002694208
    Adjacent-pair distance errors:
        Mean adjacent-pair distance error: 0.0004335479
        Mean adjacent-pair absolute distance error: 0.01592422
        SD of adjacent-pair distance error: 0.01840659
    Epipolar errors:
        Epipolar RMS Error: 1.594549 px
        Epipolar Mean Error: 1.555131 px
        SD of Epipolar Error: 0.352441 px


Now to unpack this summary. One of the challenges to assessing the accuracy of a DLT calibration is that any reconstructed points will be in the coordinate system of the points used to calibrate the cameras. This means that even if we had an object with points of known 3D position, reconstructing the object using the DLT coefficients would yield 3D points arbitrarily translated and rotated to somewhere in 3D space. We'd have to perform some alignment step in order to compare the reference 3D points to the reconstructed points. The alignment will cause underestimation of larger errors and overestimation of smaller errors. The best solution is to use a variety of different accuracy metrics that, when taken together provide a complete assessment of accuracy.

dltTestCalibration() provides four assessments of calibration accuracy:
  1. Aligned ideal to reconstructed (AITR) error. For every checkerboard that is reconstructed, the function takes an ideal checkerboard of the same dimensions (uniform square sizes and planar) and aligns the ideal corner points to the reconstructed corner points using least squares alignment. Then, the distance is measured between each ideal point and its corresponding reconstructed points. If the points were perfectly reconstructed, the ideal and reconstructed points would overlap perfectly. The “AITR RMS (root mean square) errors” are first measured along each axis (x, y and z in the coordinate system of the calibration points). This is one way to quantify how accuracy differs along different dimensions. The “mean AITR distance error” is the mean 3D distance between ideal and reconstructed points. This will usually be larger than any of the single axis errors since it incorporates error along all axes. This is also returned as RMS error. One disadvantage of this measure is that the ideal grid will be pulled toward areas of high error to minimize the total alignment error. This can cause underestimation of larger errors and overestimation of smaller errors.
  2. Inter-point distance (IPD) error. This summarizes distance rather than positional errors. For every reconstructed checkerboard random pairs of points (without re-sampling) are chosen and the distance between them is compared to the actual distance in an ideal grid (again, uniform square sizes and planar). This measure avoids the problems with the alignment step in AITR error but doesn't readily provide any information of error along a particular dimension (although this could perhaps be assessed by taking into account the positions of the reconstructed points). The distance errors are returned as “IPD RMS Error” and “IPD Mean Absolute Error”. The reconstructed distances can be either shorter or longer than the actual distance. The "Mean IPD error" takes the mean of these errors. If there is no bias toward over- or underestimation of distance this should be nearly zero. The results will differ slightly at each run because the point pairs are chosen randomly.
  3. Adjacent-pair distance errors. This is identical to IPD error except that randomly chosen points are adjacent on the grid. This means the ideal distances are uniform and the minimum possible distance for IPD error assessment. This is a common stereo camera error assessment used in the literature (e.g. Tashman & Anderst 2003; Brainerd et al. 2010). Since the points in each pair are uniformly close together, their mean position (the mid-point) can be used to look at how IPD error varies as a function of position in the calibration volume. These errors will usually be slightly less than the general IPD errors since error is likely to be greater for points at a greater distance from one another.
  4. Epipolar errors. In a stereo camera setup, a point in one camera view must fall along a line in a second camera view. This line is that point's epipolar line. The distance between a point's epipolar line and its corresponding point in that second camera view is the epipolar error. Since the input to dltTestCalibration() includes the same point in two or more camera views, we can use epipolar error to assess calibration accuracy. Epipolar error must be low to identify corresponding points along curves in different views. The mean and standard deviation of epipolar error is returned in pixels and should be less than a couple of pixels.
So how accurate is the calibration? Millimeters were used as units in measuring the checkerboard square size so all values not in pixels are in millimeters. The positional errors along all three dimensions are around 20 microns on average and the different distance errors range from 16-36 microns. The "mean IPD error" shows a slight bias towards underestimating inter-point distances but only by 3 microns on average. And for points closer together, this bias is less than a micron (0.0003 mm). Lastly, the epipolar error is less than two pixels on average, with a low standard deviation.

It's important to compare these errors to the total calibrated volume. Since the calibration checkerboard is 21 x 14 and the square size is approximately 6.36 mm, each dimension of the calibrated volume is at least 89 mm (14*6.36). 20 microns represents 0.02% positional error (0.020 mm/89 mm), an extremely low error.

dltTestCalibration() returns all of the values used to calculate the stats in the summary output. So plot() and hist() can be used to look at the full error values. For instance, we can look at a histogram of all the IPD errors.

16. Create a histogram of all the inter-point distance errors, using hist().

> hist(dlt_test$ipd.error, breaks=20)
A histogram of inter-point distance errors (in mm); N = 1029.
The histogram shows that nearly all of the distances measured between random pairs of points across each checkerboard are within 0.100 mm of their actual distance.

17. Create a histogram of all the inter-point distance errors, considering only adjacent internal corners (in this case, points within about 4 mm of each other).

> hist(dlt_test$adj.pair.ipd.error, breaks=20)
A histogram of adjacent inter-point distance errors (in mm); N = 980.
When considering only random pairs of adjacent points on each checkerboard, nearly all of the inter-point distances are within 0.040 mm of their actual distance.

18. To test how reconstruction error varies as a function of the distance from the center of the calibrated volume (i.e. do reconstructed points in the periphery of the calibrated space have a higher error than points in the middle?), plot the adjacent inter-point distance as a function of the distance of each adjacent pair from the centroid of all adjacent pairs (an approximate center of the calibrated volume).

> plot(dlt_test$adj.pair.centroid.dist, dlt_test$adj.pair.ipd.error)
IPD error for adjacent points versus the mean position of adjacent pairs along the z-axis.
There is no obvious trend in the plot, indicating that reconstruction error is uniform throughout the calibrated volume.

19. To test how reconstruction error varies as a function of the position along a particular axis, plot the adjacent inter-point distance as a function of the mean position of each adjacent pair along an axis.

> plot(dlt_test$adj.pair.mean.pos[, 3], dlt_test$adj.pair.ipd.error)
IPD error for adjacent points versus the distance of the adjacent pairs from the centroid.
There is also no obvious trend in this plot, providing a second confirmation that reconstruction error is uniform throughout the calibrated volume. The axes here are defined in the same coordinate system as the 3D calibration coordinates estimated in the calibration step. Thus, the orientation of these axes relative to the cameras is arbitrary and will depend on the orientations of the checkerboard patterns used in the calibration.

Now that the cameras are accurately calibrated, the next section will provide instructions on photographing an object for the collection of shape data.

Go to the next step: Photographing an object
Go back to the previous step: Calibrating stereo cameras

No comments:

Post a Comment