This is step 6 of 10 in the tutorial Collecting 3D shape data using StereoMorph
The previous section demonstrated how to use a checkerboard pattern of known square size and the dltCalibrateCameras() function to calibrate two cameras in stereo. While dltCalibrateCameras() returns calibration and reconstruction RMS errors, these are measures of how well the DLT camera model fits the calibration points and not the reconstruction accuracy per se. Moreover, the RMS errors are scale-independent. If we have not measured the calibration checkerboard square size correctly, the RMS errors will be unaffected but our reconstructions will be improperly scaled. This section will demonstrate how to use a checkerboard (ideally having a square size different from that used in the calibration step) to test the accuracy of a calibrated stereo camera setup.
1. Repeat the steps in “Creating a Checkerboard Pattern” to create another checkerboard of a different square size than that used in the calibration. This will allow you to test whether you have the proper scaling for the calibration checkerboard (if they had the same square size, we would not be able to test this). For this tutorial, I used a checkerboard printed at 9% scaling for the calibration and a checkerboard printed at 6% scaling to test the calibration accuracy.
|Two checkerboards, printed at 9% and 6% scaling. In this tutorial, the cameras are calibrated with the left pattern and the calibration is tested with the right.|
|Measuring the square size of the test checkerboard pattern (21 x 14, printed at 6% scaling).|
|Eight photos of the test checkerboard (a different size than the calibration checkerboard) per camera view used in the test calibration step. The tutorial includes a test checkerboard in a total of 11 different positions.|
5. Load the StereoMorph library into the current R session and ensure that the StereoMorph Tutorial folder is your current working directory.
Just as in the calibration step, we’ll use the findCheckerboardCorners() function to find the internal corners in each calibration image.
6. Specify the number of internal corners in the checkerboard.
> nx <- 21
> ny <- 14
7. Specify the file locations of the test calibration images, where to save the checkerboard corners and, if desired, where to save the verification images.
> test_image_file <- paste0('Test images/v', c(1, 2))
> test_corner_file <- paste0('Test corners/v', c(1, 2))
> test_verify_file <- paste0('Test images verify/v', c(1, 2))
8. Call findCheckerboardCorners().
> corners <- findCheckerboardCorners(image.file=test_image_file, nx=nx, ny=ny, corner.file=test_corner_file, verify.file=test_verify_file)
The function will find the corners successfully for all but three of the images in the tutorial set. Again, don’t worry if the function fails for a couple of images; these will be ignored in subsequent steps.
9. Once findCheckerboardCorners() has finished running, check all of the verification images to be sure that the order is consistent within each camera view and between the two views. For the tutorial images the corners in the first view are in the reverse order relative to the order in the second view (as was also seen with the calibration images). This will be corrected when the corners are read in from the corner files.
10. Import the corners just saved to individual files by constructing a two-column matrix of file paths, where the columns correspond to each view.
> corners_by_file <- cbind(
paste0(test_corner_file, '/', paste0('DSC_00', 11:20, '.txt')),
paste0(test_corner_file, '/', paste0('DSC_00', 11:20, '.txt')))
11. Call readCheckerboardsToArray() to read all of the corner matrices into an array; include all files – empty files will read in as NAs.
> test_corners <- readCheckerboardsToArray(file=corners_by_file, nx=nx, ny=ny, col.reverse=c(F, T), row.reverse=c(F, T))
As in the calibration step, the vector c(F, T) is passed to col.reverse and row.reverse to reverse the order of the corners from the second view relative to the first.
Since we are importing corners from 10 images, the third dimension of the test_corners array is 10.
 294 2 10 2
12. Set grid_size_test to the square size of the test calibration checkerboard. The tutorial test calibration checkerboard, measured using a precision ruler (see Measuring Checkerboard Square Size ), is approximately 4.323 mm.
> grid_size_cal <- 4.2327
13. Load in the calibration coefficients.
> cal.coeff <- as.matrix(read.table(file="cal_coeffs.txt"))
14. Call dltTestCalibration(). Image pairs for which corners were found in neither or only one image will be ignored.
> dlt_test <- dltTestCalibration(cal.coeff=cal.coeff, coor.2d=test_corners, nx=nx, grid.size=grid_size_test)
15. Use the summary() function to print a summary of the accuracy test.
Number of grids: 7
Number of points: 1029
Aligned ideal to reconstructed (AITR) point position errors:
AITR RMS Errors (X, Y, Z): 0.02343, 0.01799, 0.02063
Mean AITR Distance Error: 0.03297
AITR Distance RMS Error: 0.03605
Inter-point distance (IPD) errors:
IPD RMS Error: 0.03090397
IPD Mean Absolute Error: 0.02392322
Mean IPD error: -0.002694208
Adjacent-pair distance errors:
Mean adjacent-pair distance error: 0.0004335479
Mean adjacent-pair absolute distance error: 0.01592422
SD of adjacent-pair distance error: 0.01840659
Epipolar RMS Error: 1.594549 px
Epipolar Mean Error: 1.555131 px
SD of Epipolar Error: 0.352441 px
Now to unpack this summary. One of the challenges to assessing the accuracy of a DLT calibration is that any reconstructed points will be in the coordinate system of the points used to calibrate the cameras. This means that even if we had an object with points of known 3D position, reconstructing the object using the DLT coefficients would yield 3D points arbitrarily translated and rotated to somewhere in 3D space. We'd have to perform some alignment step in order to compare the reference 3D points to the reconstructed points. The alignment will cause underestimation of larger errors and overestimation of smaller errors. The best solution is to use a variety of different accuracy metrics that, when taken together provide a complete assessment of accuracy.
dltTestCalibration() provides four assessments of calibration accuracy:
- Aligned ideal to reconstructed (AITR) error. For every checkerboard that is reconstructed, the function takes an ideal checkerboard of the same dimensions (uniform square sizes and planar) and aligns the ideal corner points to the reconstructed corner points using least squares alignment. Then, the distance is measured between each ideal point and its corresponding reconstructed points. If the points were perfectly reconstructed, the ideal and reconstructed points would overlap perfectly. The “AITR RMS (root mean square) errors” are first measured along each axis (x, y and z in the coordinate system of the calibration points). This is one way to quantify how accuracy differs along different dimensions. The “mean AITR distance error” is the mean 3D distance between ideal and reconstructed points. This will usually be larger than any of the single axis errors since it incorporates error along all axes. This is also returned as RMS error. One disadvantage of this measure is that the ideal grid will be pulled toward areas of high error to minimize the total alignment error. This can cause underestimation of larger errors and overestimation of smaller errors.
- Inter-point distance (IPD) error. This summarizes distance rather than positional errors. For every reconstructed checkerboard random pairs of points (without re-sampling) are chosen and the distance between them is compared to the actual distance in an ideal grid (again, uniform square sizes and planar). This measure avoids the problems with the alignment step in AITR error but doesn't readily provide any information of error along a particular dimension (although this could perhaps be assessed by taking into account the positions of the reconstructed points). The distance errors are returned as “IPD RMS Error” and “IPD Mean Absolute Error”. The reconstructed distances can be either shorter or longer than the actual distance. The "Mean IPD error" takes the mean of these errors. If there is no bias toward over- or underestimation of distance this should be nearly zero. The results will differ slightly at each run because the point pairs are chosen randomly.
- Adjacent-pair distance errors. This is identical to IPD error except that randomly chosen points are adjacent on the grid. This means the ideal distances are uniform and the minimum possible distance for IPD error assessment. This is a common stereo camera error assessment used in the literature (e.g. Tashman & Anderst 2003; Brainerd et al. 2010). Since the points in each pair are uniformly close together, their mean position (the mid-point) can be used to look at how IPD error varies as a function of position in the calibration volume. These errors will usually be slightly less than the general IPD errors since error is likely to be greater for points at a greater distance from one another.
- Epipolar errors. In a stereo camera setup, a point in one camera view must fall along a line in a second camera view. This line is that point's epipolar line. The distance between a point's epipolar line and its corresponding point in that second camera view is the epipolar error. Since the input to dltTestCalibration() includes the same point in two or more camera views, we can use epipolar error to assess calibration accuracy. Epipolar error must be low to identify corresponding points along curves in different views. The mean and standard deviation of epipolar error is returned in pixels and should be less than a couple of pixels.
It's important to compare these errors to the total calibrated volume. Since the calibration checkerboard is 21 x 14 and the square size is approximately 6.36 mm, each dimension of the calibrated volume is at least 89 mm (14*6.36). 20 microns represents 0.02% positional error (0.020 mm/89 mm), an extremely low error.
dltTestCalibration() returns all of the values used to calculate the stats in the summary output. So plot() and hist() can be used to look at the full error values. For instance, we can look at a histogram of all the IPD errors.
16. Create a histogram of all the inter-point distance errors, using hist().
> hist(dlt_test$ipd.error, breaks=20)
|A histogram of inter-point distance errors (in mm); N = 1029.|
17. Create a histogram of all the inter-point distance errors, considering only adjacent internal corners (in this case, points within about 4 mm of each other).
> hist(dlt_test$adj.pair.ipd.error, breaks=20)
|A histogram of adjacent inter-point distance errors (in mm); N = 980.|
18. To test how reconstruction error varies as a function of the distance from the center of the calibrated volume (i.e. do reconstructed points in the periphery of the calibrated space have a higher error than points in the middle?), plot the adjacent inter-point distance as a function of the distance of each adjacent pair from the centroid of all adjacent pairs (an approximate center of the calibrated volume).
> plot(dlt_test$adj.pair.centroid.dist, dlt_test$adj.pair.ipd.error)
|IPD error for adjacent points versus the mean position of adjacent pairs along the z-axis.|
19. To test how reconstruction error varies as a function of the position along a particular axis, plot the adjacent inter-point distance as a function of the mean position of each adjacent pair along an axis.
> plot(dlt_test$adj.pair.mean.pos[, 3], dlt_test$adj.pair.ipd.error)
|IPD error for adjacent points versus the distance of the adjacent pairs from the centroid.|
Now that the cameras are accurately calibrated, the next section will provide instructions on photographing an object for the collection of shape data.
Go to the next step: Photographing an object
Go back to the previous step: Calibrating stereo cameras