Tuesday, April 8, 2014

Calibrating stereo cameras

This is step 5 of 10 in the tutorial Collecting 3D shape data using StereoMorph

This section demonstrates how to photograph the checkerboard created in Creating a Checkerboard Pattern in different positions and orientations and use these photos to calibrate two cameras arranged in stereo.

Example of checkerboard photographed in different positions for stereo camera calibration.
Camera calibration using the direct linear transformation (DLT) method requires two sets of points: (1) a set of 3D coordinates and (2) their corresponding 2D pixel coordinates in each camera view. These two point sets are used to calculate calibration coefficients. These coefficients can then be used to reconstruct any point in 3D given its 2D pixel coordinates in at least two camera views. DLT calibration has traditionally been done using a "calibration object", typically a 3D box-shaped structure filled with markers at known 3D positions.

Example calibration object. From XROMM Wiki at wiki.brown.edu.

First, designing and building a 3D calibration object is not trivial. The object must have several points of known 3D position relative to one another and a sufficient number of these points, filling the 3D volume, should be visible from a single camera view. Secondly, to achieve high accuracy, these objects must be made using high precision machining. Thirdly, the calibration points must be digitized manually every time the cameras are calibrated.

Rather than use a 3D calibration object photographed in a single position, StereoMorph performs a DLT calibration using a 2D checkerboard pattern photographed in several positions and orientations. Photographs from two camera views of a planar checkerboard in a single orientation are insufficient to calibrate cameras in stereo because all of the checkerboard points lie in a single plane. Photographing a checkerboard in several different positions throughout the volume solves this problem by providing a sample of points throughout the volume. However, this poses a new problem. Each plane of points is in a different coordinate system; the 3D position and orientation of one plane relative to another is unknown.

StereoMorph solves this by using the nlminb() function (in the R package ‘stats’) to estimate the six transformation parameters (three translation, three rotation) required to transform the first checkerboard into each subsequent checkerboard in 3D space. The transformation parameters that minimize the calibration error are chosen as the optimal parameters. These transformation parameters are then used to generate the 3D coordinates needed to compute the DLT calibration coefficients.

1. Once you have arranged two cameras in a stereo setup (see the previous section), photograph the checkerboard pattern in several different positions and orientations within the volume to be calibrated (the volume in which the camera views overlap).

Eight photos of the checkerboard per camera view used in this tutorial for camera calibration.

In the image above the first camera view corresponds to the camera positioned on the floor (in the sketch to the right) and the second camera view corresponds to the camera sitting on the top of the table. Note in particular that the second camera is viewing the checkerboard pattern upside-down relative to the first camera.

2. Import the photographs from different views into two different folders. For demonstration, we’ll use the calibration photographs in the folder ‘Calibration images’ (in the StereoMorph Tutorial folder). Photographs from camera one are in the folder ‘v1’ (view 1) and photographs from camera two are in ‘v2’. For the rest of this tutorial, views 1 and 2 will be used to refer to photographs taken from camera 1 and 2, respectively.

The camera arrangement used in this tutorial.

3. Load the StereoMorph library into the current R session and ensure that the StereoMorph Tutorial folder is your current working directory.

> library(StereoMorph)

We’ll use the findCheckerboardCorners() function, introduced in the ‘Auto-detecting Checkerboard Corners’ section, to find the internal corners in each calibration image.

4. First, specify the number of internal corners in the checkerboard.

> nx <- 21
> ny <- 14

5. Then, specify the file locations of the calibration images, where to save the checkerboard corners and, if desired, where to save the verification images showing the automatically detected corners.

> image_file <- paste0('Calibration images/v', c(1, 2))
> corner_file <- paste0('Calibration corners/v', c(1, 2))
> verify_file <- paste0('Calibration images verify/v', c(1, 2))

Since the images are in two different folders (‘v1’ and ‘v2’), the paste0() function is used to create a vector (example below) for the two different folders within ‘Calibration images’.

> paste0('Calibration images/v', c(1, 2))
[1] "Calibration images/v1" "Calibration images/v2"

The findCheckerboardCorners() will automatically assign save-as filenames.

6. Call findCheckerboardCorners(). Note that this function can take several seconds per photo to run.

> corners <- findCheckerboardCorners(image.file=image_file, nx=nx, ny=ny, corner.file=corner_file, verify.file=verify_file)

By default, the findCheckerboardCorners() function will print its progress to the R console as each image is loaded and whether or not the specified number of internal corners were found (note that if the number of internal corners found does not match the number specified, no corners are saved).

Loading image 1 (DSC_0002.JPG)...
    294 corners found successfully.
Loading image 2 (DSC_0003.JPG)...
    294 corners found successfully.

The function will find the corners successfully for all of the images in the tutorial set. When using your own images don’t worry if the function fails for a couple of images; these will be ignored in subsequent steps. Four to five pairs of calibration images are usually sufficient for an accurate calibration.

7. Once findCheckerboardCorners() has finished running, check all of the verification images to be sure that the order is consistent within each camera view and between the two views.

The checkerboard in two different positions from the same camera view (view 1)

Within the same view, the corners will be returned in the same order as long as the orientation of the checkerboard does not change radically within the same set. However, in the case of this calibration the order is not the same between the two views.

An image pair from two different camera views

Although it looks like the corners in view 1 and 2 have been returned in the same order, they are actually in the reverse order. Camera 1 is viewing the checkerboard head-on while camera 2 is viewing the checkerboard from the top. The point circled in red in view 2 and closest to the red mat on the table top is the same as the point circled in blue in view 1. We can easily fix this when we import in the corners using the readCheckerboardsToArray() function.

8. Import the corners just saved to individual files by constructing a two-column matrix of file paths, where the columns correspond to each view. The file paths must be specified as a matrix because the readCheckerboardsToArray() function will use the matrix structure to separate the corners by view.

> corners_by_file <- cbind(
paste0(corner_file[1], '/', paste0('DSC_000', 2:9, '.txt')),
paste0(corner_file[2], '/', paste0('DSC_000', 2:9, '.txt')))

9. Call readCheckerboardsToArray() to read all of the corner matrices into an array; include all files – empty files will read in as NAs. The array will have four dimensions: the first two dimensions correspond to the rows and columns of each corner matrix (294 x 2), the third corresponds to the number of positions of each checkerboard (8) and the fourth corresponds to the number of camera views (2).

> corners <- readCheckerboardsToArray(file=corners_by_file, nx=nx, ny=ny, col.reverse=c(F, T), row.reverse=c(F, T))

In passing the vector c(F, T) to the col.reverse parameter (‘F’ for FALSE and ‘T’ for TRUE), the function will maintain the current order of the corners in the first view and reverse the column order of the corners in second view. The same is done for the row order via the row.reverse parameter. This reverses the order of all the corners for each image from the second camera view. This is essential – the same row in each corner matrix must correspond to the same point across all of the images (among and between views).

10. Set grid_size_cal to the square size of the calibration checkerboard. The tutorial calibration checkerboard, measured using a precision ruler (see “Measure Checkerboard Square Size”), is approximately 6.365 mm.

> grid_size_cal <- 6.3646

11. Call dltCalibrateCameras(). This can take from 30 seconds to five minutes depending on the number of images and the speed of your computer. We’ll just use the first six image pairs of the checkerboard. Image pairs for which corners were found in neither or only one image will be ignored.

> dlt_cal_cam <- dltCalibrateCameras(coor.2d=corners[, , 1:6, ], nx=nx, grid.size=grid_size_cal, print.progress=TRUE)

The function begins by reducing the grid point density.

Reduce grid point density (12 total)
    1) Aspect 1, View 1; Mean fit error: 0.45 px; Max: 1.78 px
    2) Aspect 1, View 2; Mean fit error: 0.65 px; Max: 2.85 px

This fits a camera perspective model to the checkerboard corner points and reduces this corner set to nine points (3x3). These nine points contain the same information as the original number of points (here, 293), but allows the optimization function to proceed much more quickly. The mean fit error should be less than a pixel.

The function then searches for the optimal parameters to transform each checkerboard, yielding a set of 3D coordinates that results in the lowest calibration error.

Full Transform RMSE Minimization
Number of parameters: 30
Number of points: 54

The first checkerboard (or grid) is fixed in 3D space at (0, 0, 0). dltCalibrateCameras() then searches for a set of six parameters per grid (three for translation and three for rotation) to transform each grid relative to the first. Here, we have six image pairs corresponding to a grid in six different orientations. Because the first grid is fixed, the six transformation parameters are only needed for five grids. This means a total of 30 (6 times 5) parameters must be estimated. Since the point density was reduced to nine per checkerboard, there are 54 total points (9 times 6).

The minimization begins with three grids. Once the minimization converges to an appropriate solution, the minimization adds another grid, using the previously optimized parameters as starting parameters. This proceeds sequentially until all of the input checkerboards have been included (the sequential addition of grids helps in proper convergence).

12. Use the summary() function to print a summary of the calibration run.

> summary(dlt_cal_cam)

dltCalibrateCameras Summary
    Minimize RMS Error by transformation
        Total number of parameters estimated: 30
        Final Minimum Mean RMS Error: 0.885
        Total Run-time: 35.97 sec

    Mean Reconstruct RMSE: 0.414
    Calibration Coefficient RMS Error:
        1: 0.666
        2: 0.66
    3D Coordinate Matrix dimensions: 1764 x 3

The reconstruction and calibration error are decent proxies for the accuracy of the calibration; an RMS error less than one is good. If the RMS error is greater than one, there might be a problem somewhere in the calibration. Testing the accuracy (covered in the next section) can show this more definitively.

The function returns the calibration coefficients in cal.coeff:

> dlt_cal_cam$cal.coeff
               [,1]          [,2]
 [1,]  2.772121e+01 -2.557153e+01
 [2,] -4.918245e-02 -5.183902e+00
 [3,]  6.226293e+00  1.498422e+00
 [4,]  2.329910e+03  2.187563e+03
 [5,]  1.128266e+00 -9.899814e-01
 [6,] -2.795001e+01  7.701885e+00
 [7,] -1.246494e+00  2.450539e+01
 [8,]  1.608486e+03  6.728743e+02
 [9,] -3.854642e-06 -4.570336e-05
[10,] -5.200086e-04 -2.396548e-03
[11,]  2.593912e-03  1.183541e-03

Note that there are 11 rows and two columns. The 11 rows correspond to the 11 DLT coefficients and the two columns correspond to the two camera views. This 11x2 matrix can be used to reconstruct any point, given its 2D pixel coordinates in both camera views. That is, the calibration coefficients are the only values we need from this section to perform the rest of the steps in this tutorial.

13. Save the calibration coefficients to a text file.

> write.table(x=dlt_cal_cam$cal.coeff, file="cal_coeffs.txt", row.names=F, col.names=F, sep="\t")

The next section will detail how to determine the accuracy of the calibration.

Go to the next step: Testing the calibration accuracy
Go back to the previous step: Arranging the cameras

1 comment:

  1. I recently came accross your blog and have been reading along. I thought I would leave my first
    comment. I dont know what to say except that I have enjoyed reading. Nice blog. I will keep
    visiting this blog very often. I am an Glassware calibration lab in Chennai.
    I feel great after reading this information. Please make update I will be regular rss to this site.
    Mass Calibration Lab in Chennai