CALIBRATION OF SMARTPHONE’S REAR DUAL CAMERA SYSTEM

. This paper aims to calibrate smartphone’s rear dual camera system which is composed of two lenses, namely; wide-angle lens and telephoto lens. The proposed approach handles large sized images. Calibration was done by capturing 13 photos for a chessboard pattern from different exposure positions. First, photos were captured in dual camera mode. Then, for both wide-angle and telephoto lenses, image coordinates for node points of the chessboard were extracted. After-wards, intrinsic, extrinsic, and lens distortion parameters for each lens were calculated. In order to enhance the accuracy of the calibration model, a constrained least-squares solution was applied. The applied constraint was that the relative extrinsic parameters of both wide-angle and telephoto lenses were set as constant regardless of the exposure position. Moreover, photos were rectified in order to eliminate the effect of lens distortion. For results evaluation, two oriented photos were chosen to perform a stereo-pair intersection. Then, the node points of the chessboard pattern were used as check points.


Introduction
The rapid evolution of smartphones technology in the last few years directly impacted the smartphone's embedded cameras. Recently, most smartphones are provided with rear dual or multi camera system. However, the majority of the smartphones come up with dual camera system. Calibrating this dual camera system facilitates the possibility of using smartphone in daily applications such as measuring dimensions of some piece of furniture, computing the internal dimensions of some room, recognizing the size of mechanical tools such as bolts and washers, and also, for several biomedical applications. Dual camera is a camera provided with two lenses that help in capturing photos. Many types of dual cameras can be found in the market. However, the type used by most manufacturers is the "Wide-Angle + Telephoto" type. Accordingly, this is the type of dual cameras utilized in this research. The "Wide-Angle + Telephoto" dual camera consists of two lenses: wide-angle lens and telephoto lens. The focal length of telephoto lens is longer than that of wide-angle one. Consequently, telephoto lens is used for differentiation between the background and foreground of the captured images (TechXplore, 2016).
Camera calibration approach using chessboard pattern was first introduced by Zhang (1998) and (2000), parameters of all the elements of a smartphone or tablet (screen, front camera, and back camera) was proposed by Delaunoy et al. (2014). The authors used a single static planar mirror to accomplish the calibration process. By moving the smartphone in front of a planar mirror, it was possible to establish the correspondences between the images and a pattern displayed on the screen. Therefore, the geometric relationship between the non-overlapping cameras with respect to the screen location was estimated. An improved calibration method was introduced by Chen et al. (2018). The proposed method decreased the calibration effort by simplifying the extrinsic calibration of one camera-projector pair. A variant iterative closest point (ICP) algorithm was studied to match 3D cloud data sets for each camera-projector, and to reject outliers and invisible data automatically at each iterative step.
In this paper, the chessboard pattern camera calibration approach -also called Zhang's approach and introduced in Zhang (1998Zhang ( , 2000, and Burger (2019) -was exploited for calibrating smartphone's rear dual camera system which consists of wide-angle lens and telephoto lens. Unlike other commercial softwares, the introduced algorithm was designed to deal with photos having large size. The calibration approach is mainly based on observing 2D chessboard pattern from different exposure positions. For all exposure positions, the relative extrinsic parameters between the two lenses composing the dual camera system are the same. In other words, the geometric distances between perspective centers of both wide-angle and telephoto lenses are constant regardless the exposure position. In addition, the relative rotation angles between the camera coordinate systems of both lenses are constant for all exposure positions. The utilized approach was modified by adding constraint equations in order to satisfy the above two conditions. The paper proceeds by explaining the calibration procedure including necessary mathematical equations. Then, experimental work and results analysis are illustrated. Finally, conclusion and recommendations for future work are presented.

Calibration procedure
Calibration is performed through creating the calibration model by capturing photos in the dual camera mode and then labeling them, followed by estimating the homography between ground and image coordinate systems. Afterward, intrinsic parameters are calculated, where then the extrinsic parameters are extracted. After that, radial lens distortion parameters are calculated. Finally, the entire set of calculated parameters is refined by a constrained leastsquares solution.

Creating calibration model
Calibration model is constructed by capturing photos in dual camera mode from different capturing positions. In dual camera mode and for each capturing position, a pair of photos is captured simultaneously. Each pair of photos consists of a photo that is captured by the wide-angle lens and other one captured by the telephoto lens. Therefore, the captured photos are divided into two groups, namely; the wide-angle lens photos group and the telephoto lens photos group.

Extracting image coordinates of Chessboard-Pattern's node points
In this sub-section, image coordinates of the chessboard pattern node points in each photo are extracted. First, four corners of chessboard pattern are manually assigned in each photo for estimating the vanishing points of parallel chess lines (see Li et al., 2010;Ge et al., 2016;Caprile & Torre, 1990). Then, the cross ratio theory (Cao et al., 2010) was applied for detecting the initial position of the node points in each photo. Finally, the detected nodes are refined according to a corner detection algorithm, which is adopted by Harris and Stephens (1988) and Bouguet (2015).

Estimation of homography between ground coordinate system and image coordinate system
In this research, we define the model plane as the model that consists of ( ) m photos and relating the camera coordinate system to ground coordinate system. For each photo in the model plane, the relationship between ground point coordinates ( , , ) X Y Z , in metric units, and camera point coordinates ( , ) u v , in pixels, is investigated using the pinhole camera model suggested by Zhang (2000), Burger (2019), and Burger and Burge (2016): where ( ) λ is an arbitrary scale factor. The extrinsic parameters matrix R T     is the matrix which relates the ground coordinate system to the camera coordinate system. This matrix consists of the rotation matrix X Y Z are the ground coordinates, in metric units, of the camera perspective center. In addition, ( ) A , known as camera intrinsic parameters matrix, is given in pixels by: o o u v are the camera coordinates of principal point, ( , ) α β are the scale factors in ( , ) u v axes directions, respectively, and ( ) γ represents the skewness coefficient of the two image axes.
For the 2D chessboard pattern ( 0) Z = , the formula given by Equation (1) becomes: or: where ( ) H is the homography between the ground coordinate system and image coordinate system and is defined up to a scale factor as:

Extraction of intrinsic parameters
This sub-section is devoted for calculating the camera intrinsic parameters matrix ( ) A . Since 1 ( ) R and 2 ( ) R are orthonormal, the following two equations can be obtained (Zhang, 2000): In the above two equations, is a symmetrical matrix, which describes the image absolute conic (see Teramoto & Xu, 2002;Luong & Faugeras, 1997), defined by the following vector: First, we solve the ( ) b vector for extracting the intrinsic parameters ( , , , , ) (4) and (5) can be written as follows: or: where: Each photo in the model plane results in 2 equations of forms (9) and (10). Hence, in order to calculate the 6 elements of ( ) b , at least 3 photos are required. If we have ( ) m photos with ( 3) m ≥ , a singular value decomposition solution (Golub & Van Loan, 2013) is applied in order to obtain the 6 elements (actually 5 elements up to a scale factor) of ( ) b vector. Once the ( ) b vector is obtained, a unique solution for the 5 intrinsic parameters is obtained as follows (Burger, 2019):

Extraction of extrinsic parameters
In this sub-section, the ground coordinates of camera perspective center ( , , ) ω φ κ about ( , , ) X Y Z axes, that relates the ground coordinate system to image coordinate system, are calculated. Once the camera intrinsic parameters are obtained, the scale factor ( ) λ is obtained as follows (Zhang, 2000): Then, Since the rotation matrix ( ) R is an orthogonal matrix, then 3 Therefore,  . It should be noted that, there are several methods to extract ( , , ) ω φ κ angles from the ( ) R rotation matrix. In this research we use Matlab® software to extract the ( , , ) ω φ κ angles.
Once ( ) R matrix is obtained, the ground coordinates of the camera perspective center become: (11)

Extraction of radial lens distortion coefficients
In this sub-section, the effect of radial lens distortion is taken into account. Only the first two radial lens distortion coefficients 1 2 ( , ) k k are considered. Knowing the point ground coordinates ( , ) X Y , the ideal normalized image coordinates ( , ) x y are calculated as follows: Furthermore, the ideal pixel image coordinates ( , ) u v are calculated according to Equation (2).
Given ( ) n points in ( ) m photos, Equation (12) leads to a number of (2 ) n m × equations in terms of 1 2 ( , ) k k . Or, in matrix form, we have: with  . The linear least-squares solution for equation (14) is given by Zhang (2000): (15)

Refining of parameters
Two groups of parameters have been calculated so far. The first group contains parameters related to wide-angle lens. Whereas, the second group contains parameters related to telephoto lens. In other words, the two groups are: With subscripts ( , , ) W T m denote wide-angle lens, telephoto lens, and the photo identification number, respectively.
The above parameters will be used as the initial values for the refining procedure described below.
For ( ) n points observed in ( ) m photos, we have the following four minimization equations: where: , ,   (19)). The first constraint is the constancy of the geometric distances between the two perspective centers of both wide-angle and telephoto lenses regardless of the lens position. The second constraint is constancy of the relative rotation angles between the two cameras. The geometric distance between the two perspective centers for the th ( ) j pair of photos is given by: In addition, the relative rotation angles between the two lenses for the th ( ) j pair of photos are given by: Thus, a total number of (4( 1)) m − constraint Equations of forms (20) to (23) are added to the main Equations (16) to (19). The followed procedure of general leastsquares solution with constraints is illustrated in Chapter 9 of Mikhail and Ackermann (1982).

Image rectification
After refining all parameters, images are rectified in order to eliminate the effect of lens distortion. Given the original image ( ) I , the rectified image ( ') I is to be calculated according to Burger (2019) as shown in the following steps: First, the distorted normalized image coordinates ( , ) x y   are calculated as: Then, the distorted radial distance is: The following non-linear equation is solved in order to obtain the non-distorted radial distance ( ) r : It should be noted that the above equation is solved by applying numerical techniques, such as Newton-Raphson's method and assigning an initial value ( ) r r =  . The non-distorted normalized image coordinates ( , ) x y are computed as follows: . / x x r r =  ; . / y y r r =  .
Then, the non-distorted pixel image coordinates ( , ) u v are: Finally, the rectified image ( ') I is computed as follows:

Experimental work
Calibration model was created using Samsung Galaxy Note8 smartphone with rear dual camera. The dual camera system consists of two lenses: wide-angle lens and telephoto lens (Figure 1). Specifications provided by manufacturer for both wide-angle and telephoto lenses are illustrated in Table 1. The table shows that the pixel size is 1.4 mm and 1.0 mm for wide-angle lens and telephoto lens, respectively. In addition, the image size is 4032 × 3024 Pixels with image resolution of 12 MP for both two lenses. The proposed algorithm reads photos one by one in order to handle the large size photos. The used chessboard pattern consists of 6 squares by 9 squares with square size of 30 × 30 mm. Thirteen photos for the chessboard pattern were captured in the dual camera mode from different exposure positions with different orientations. Then, using some option in the smartphone, photos from each of the two lenses of dual camera were separated. This resulted in a total number of photos of 26 photos (13 photos for wideangle lens and 13 photos for telephoto lens). A total number of 40 node points were extracted as described earlier. Figure 2 shows photos sample from both wide-angle and telephoto lenses respectively. Moreover, Figure 3 illustrates the 40 extracted points in some photo. The calibration procedure explained in the above section was applied to the calibration model in order to calculate intrinsic, extrinsic, and radial lens distortion parameters for both wide-angle and telephoto cameras. Two calibration models were examined in this research: Model 1 and Model 2. In Model 1, the value of the skewness coefficient ( ) γ was assigned to zero. Whereas, the value of ( ) γ was calculated during the calibration process in Model 2. The calculated values, in pixels, of intrinsic and radial   Table 2 and Table 3. Table 2 shows results for wide-angle lens applying Model 1 and Model 2, respectively. On the other hand, Table 3 shows results applying the two models for telephoto lens. The tables show that the parameters for each lens are close to each other in both cases of assigning the value of ( ) γ to zero (Model 1) or calculating it during the calibration process (Model 2). In addition, the values of scale factors ( , ) σ β , for both wideangle and telephoto lenses, are close to the values provided by the manufacturer factors shown in Table 1. When using a ruler to manually measure the geometric distance between the two perspective centers of the both telephoto and wide-angle lenses, it was found to be 11 millimeters. In addition, both lenses of the smartphone look parallel to each other, which means that the relative rotation angles ( , , ) ∆ω ∆φ ∆κ between the two lenses are expected to be close to zero. This complies with the results illustrated in Table 4. The table illustrates the calculated values of geometric distance (in millimeters) between the two perspective centers along with the relative rotation angles (in minutes) between the two lenses. Results are shown for both Model 1 and Model 2. It is clear from the table that the calculated values are very close to the manually observed ones. Photos were rectified following the image rectification procedure illustrated previously. A sample rectified photos for both wide-angle lens and telephoto lens are illustrated in Figure 4 and Figure 5, respectively.
In order to evaluate the obtained results more precisely, a stereo-pair intersection was performed using two oriented photos. Then, the 40 node points of the chessboard were utilized as check points. Three intersection cases were considered for each of Model 1 and Model 2. In Case 1, two photos captured simultaneously in the same dual mode (from the same capturing position) were used (Figure 6a). One of the two photos was captured by the wide-angle lens and the other one was captured by the telephoto lens. In Case 2, two photos captured from wide-angle lens exposed from two different dual modes (from different capturing positions) were used ( Figure 6b). Finally, two photos captured from telephoto lens from two different dual modes (from different capturing positions) were utilized in the Case 3 ( Figure 6c). The values of root mean square errors for the 40 check points in ( , , ) X Y Z directions, for the above three cases using both Model 1 and Model 2 are illustrated in Table 5 and Table 6, respectively. The tables show that the results obtained by both Models for each of the three cases are almost similar. In addition, the values of root mean square errors for Case 1 are relatively high. The reason is that the base distance between the wide-angle and telephoto lenses is quite small (about 11 millimeters) with respect to the average distance from the two lenses and the object chessboard pattern (about 600 millimeters). The best configuration of dual camera mode is out of the scope of this research. Moreover, results obtained for Case 2 and Case 3 are reliable and close to each other.

Conclusions and recommendations
In this paper, camera calibration approach using 2D chessboard pattern was applied to the dual camera system of a smartphone. The calibration model has been evaluated using large-sized photos. Obtained results showed that the proposed technique has provided a powerful and reliable tool for dual camera calibration. Mathematical equations necessary for calculating intrinsic, extrinsic, and lens . The three cases for intersection distortion parameters were introduced in the research. All calculated parameters were refined according to a constrained least-squares solution, such that geometric distance and relative rotation angles between the two lenses are given constants. Two calibration models were tested. In Model 1 the value of skewness coefficient was assigned to zero. While, in Model 2, skewness coefficient was calculated during calibration. The values of intrinsic, extrinsic, and radial lens distortion parameters obtained using both models were almost typical. In addition, the values of scale factors ( , ) σ β for both wide-angle and telephoto lenses were close to the values of focal length provided by the manufacturer. Moreover, the values of relative extrinsic parameters of the two lenses were similar to the manually observed ones. Furthermore, photos were rectified in order to eliminate the effect of lens distortion. For the purpose of results evaluation, two oriented photos were selected to perform a stereo-pair intersection using nodes of chessboard as check points. Intersection was done for three cases. Case 1: using wide-angle photo and telephoto photo captured in the same dual mode. Case 2: two wideangle photos captured from different dual modes were used. Case 3: two telephoto photos captured from different dual modes were used. The values of errors in Case 1 were relatively high due to the quite small base distance between the two lenses comparing to the average distance to the object chessboard pattern. Moreover, results obtained for Case 2 and Case 3 were reliable and close to each other.
Future work will focus on studying the best geometric configuration required to minimize the intersection error. Furthermore, applications for using dual camera in tiny object observation will be investigated. Moreover, measuring internal dimension of some room with the help of dual camera of smartphone will be examined. Utilizing dual camera of smartphone in machine applications are to be studied as well. Finally, using dual camera smartphone in biomedical applications will be addressed.