IMPLEMENTATION OF THE SMARTPHONE CAMERA IN THE MEASURES OF NARROW STREET FACADES

. This paper aims to perform metric measurements of narrow street façades using single image captured by smart-phone’s camera. Since tight area accompanied by narrow street limits object to camera distance, object lines perpendicular to façade do not appear in image and consequently their vanishing point (VP) is hard to detect. Accordingly, semi-automated MATLAB ® application was designed depending only on two orthogonal VPs. Novelty of work comes from us-ing smartphone as a cost and time efficient tool for measurements, depending only on two VPs, and applying image line refinement approach exploiting detected VPs. Three single images were captured by three different smartphones. Then, undistorted single images were formed after calibrating cameras. Image lines for horizontal and vertical object lines were extracted semi-automatically. Two VPs were detected applying two models: Model-I solves for vanishing points’ Cartesian coordinates, whereas Model-II solves for angle coordinate peaks of histogram. Image line refinement approach was applied before applying cross-ratio using one horizontal and one vertical reference lines to calculate object lengths of 46 check lines (horizontal and vertical). Proposed models provided reliable and comparable results. Applying line refinement approach improved solution with best overall accuracy of 0.010 m and 0.011 m for Model-I and Model-II, respectively.


Introduction
The task of metric measurements for the purpose of façade observation is one of the highly demanded tasks that is frequently required by surveyors.For narrow streets lying in crowded areas, using traditional land surveying instruments such as total station or theodolite is not the best choice.Besides the land surveying is a time-consuming option, the narrow width of street confines the angle of view of instrument's telescope and thereby prevents the instrument from observing façade points with high altitudes.In addition, the streets that are crowded of people impedes the setting-up of land surveying instrument for long time.In contrary, using the smartphone's camera for façade observation delivers an adequate tool to get over the difficulties accompanied by the ordinary land surveying techniques.The smartphone's camera is free to rotate such that the high-elevated points are possible to be captured and the time taken in site is only the seconds required for image to be captured.Moreover, the use of smartphones, which are currently available at high resolution and affordable prices, will result in high accuracy and points is applied to the extracted raw image lines.Each raw image line is adapted by modifying its endpoint that is nearest to vanishing point to be lying along the line between the farthest endpoint and vanishing point keeping the image line's length the same.Meanwhile, the user is simply required to measure the width and height of an accessible rectangular window on the façade using a measuring tape.The measured width is exploited as the reference length for horizontal object lines, and measured height is exploited as the reference length vertical object lines.The cross-ratio theory is applied for calculating the object length of any horizontal or vertical line belonging to the façade utilizing the horizontal and vertical reference object lines, respectively.Three smartphones' cameras of different brand-names (iPhone_7_Plus, Samsung Galaxy Note8, and HUAWEI_nova_7i) were utilized for data analysis and comparison.
Dissimilar to most of published researches that rely on three orthogonal vanishing points detection such as Li et al. (2010), Vouzounaras et al. (2014), Arslan (2018), the current research relies only on two orthogonal vanishing points detection.These two vanishing points are the vanishing points for horizontal and vertical object lines belonging to the façade in question.Usually, lines perpendicular to building's façade usually do not appear in images captured in narrow streets and, by consequence, the vanishing point related to these perpendicular lines is hard to be assigned.This is caused by one of two reasons.The first reason is that the depth of building (plane perpendicular to façade) is occluded by the other buildings that are next to the building in question.The other reason is that the short object to camera distance (resulting from the small width of narrow street) does not supply a sufficient space for the camera to be moved away enough from the building in purpose to capture such perpendicular lines.In this accordance, the presented research depends only on the two orthogonal vanishing points for the horizontal and vertical object lines appearing in the façade plane.

Related work
Many papers covered the topic of metric measurements from single image including the applications for cross-ratio and vanishing points.One of the early research projects was introduced by Debevec et al. (1996), where a methodology to render and model architectural views using photos was developed combining geometry and imagebased techniques.The cross-ratio variances assuming normal distribution case were formulated in Liu and Chuang (2002).A method to identify cross-ratios with minimum error variances was introduced relying on geometric error analysis.Simulated results showed that the presented approach led to estimate the minimum error variances better than other methods.Lee and Nevatia (2003) integrated various sources including aerial and ground scene photos for performing hierarchical representation of 3D buildings located in urban areas.The results showed that the proposed technique was an efficient and low-cost one.An approach based on cross-ratio invariability for perspective projection was presented by Zhang et al. (2003) for calibrating camera radial distortion.Results obtained by both simulated real data emphasized the simplicity along with efficiency of the proposed approach.In Aguilera et al. (2005), the authors introduced an approach for vanishing points detection taking into account the presence of minisegments resulted from automatic line extraction which leads to unfavorable intersection cases.The introduced approach consists of three main steps: clustering mini-segments, estimation of vanishing points, and computation applying a re-weighted least-squares solution (L.S.S.) support by an M-estimator.A technique for computer vision systems' localization error analysis exploiting cross-ratios was proposed by Chuang et al. (2007).An applied method for selecting images' point features was developed by authors.An approach for extracting geometric information from uncalibrated single image was introduced in Wang et al. (2010).Cross-ratio theory was applied to calculate the line segments' length on planar surface knowing reference lines' length.Both indoor and outdoor images were utilized for verification.In Li et al. (2010), the authors presented a method to estimate the camera focal length along with three orthogonal vanishing points simultaneously from single images depending on image polar coordinate system.Compared to previous methods, the proposed method required less running time and computer memory.A study to directly extract vehicle speeds from video footages utilizing cross-ratio theory was introduced by Wong et al. (2014).In addition, a control study using a calibrated Doppler radar was performed for validation by the authors.A fully automated approach for 3D models creation was introduced by Vouzounaras et al. (2014).The proposed approach combined the extracted image line segments for the identification of vanishing points, image orientation, indoor planes, and outdoor planes.An invariant on the projective plane and its use for reducing the complexity of projective transformations was proposed by Erdnüß (2018).The camera centers and 3D structure of scene were detected through the proposed approach.Arslan (2018) presented a study for extracting metric data from single image.Two methods were examined by the author.The first method relied on cross-ratio, while the second method relied on robust camera calibration and statistical estimators.The results revealed that the second method provided higher accuracy than the first one.

Coordinate systems
Both object and image coordinate systems are defined in this section.In Figure 1, the object space axes are the ( ) , , X Y Z axes perpendicular to façade, parallel to façade width, parallel to façade height, respectively.Additionally, the image coordinate system axes are denoted as ( )

Research methodology
The smartphone's camera is first calibrated in order to create the image undistortion array, then metric measurements are done on undistorted single image.Two MATLAB ® applications were developed: one application for camera calibration and the other one for metric measurements.A summary for the research methodology is given in the flowchart illustrated in Figure 2.
Figure 2. Flowchart for research methodology

Camera calibration MATLAB ® application
A camera calibration MATLAB ® application was created for lens distortion coefficients calculation and image undistortion array generation.A screenshot for the application is given in Figure 3 illustrating the different functions of application.
Figure 3. MATLAB ® application created for camera calibration

Smartphone's camera calibration
The smartphone's camera is calibrated applying Zhang's technique for camera calibration using chessboard pattern (Zhang, 2000;Burger, 2019;Aldelgawy & Abu-Qasmieh, 2021a, 2021b).A number of images changing the angles of view is captured for a well-calibrated chessboard pattern.Then, the image nodes of chessboard pattern are used to establish the calibration model.The pixel image point coordinates (u, v) in terms of metric ground point coordinates (X, Y, Z) are given as follows (Zhang, 2000;Burger, 2019): R R R R is rotation matrix, and represents the metric ground coordinates vector of camera perspective center.In addition, the pixel camera intrinsic parameters matrix ( ) A is given as follows: where ( , ) o o u v are the pixel image coordinates of (pp), a b ( , ) are the image axes scale factors, and g ( ) denotes the skewness coefficient.
For the chess pattern (2D case), putting with ( ) 2) is solved for (H) prior to calculating both intrinsic and extrinsic parameters.

Single image undistortion
The lens distortion effect is considered this sub-section.Based on previous researches such as Zhang (2000), Burger (2019), Aldelgawy and Abu-Qasmieh (2021a), only the radial lens distortion is taken into account through estimating the three radial lens distortion coefficients 1 2 3 ( , , ) k k k .Accordingly, the image undistortion array is generated.Knowing the point ground coordinates of chessboard node ( , ), X Y the normalized (ideal) image coordinates ( , ) x y are given as follows: Then, the undistorted (ideal) pixel image coordinates ( , ) u v are: The relationship between the observed (distorted) pixel image coordinates   ( , ) u v and undistorted ones ( , ) u v is given as follows (Burger & Burge, 2016;Burger, 2019;Wang et al., 2006): where ∆ ∆ ( , ) ur vr are the radial lens distortion components in ( , ) u v directions, respectively that are calculated as follows: Then, given the original image array ( ), I the undistorted image array ( ') I is:

Metric measurements MATLAB® application
An application was developed by MATLAB® for metric measurements (Figure 4).The developed application relies basically on extracting image lines semi-automatically, then applying a refinement approach on the extracted lines based on the firmly detected two orthogonal vanishing points.The procedure used for the developed application is summarized in the following lines.

Image lines extraction
Image lines are extracted from the undistorted single image.First, image noise is reduced applying a median filter.
Then, the raw image line segments are extracted.The developed application is provided with the "Group of Lines" extraction option which allows the user to draw a window on the desired zone in image and manually input the maximum number of raw line segments that are required to be extracted inside the window.Furthermore, the application is provided with the "Line-by-Line" line extraction option which enables the user to pick up each single raw line segment separately by drawing a line close to the demanded line segment.Then, the raw line segment in question is automatically extracted within a selected buffer width (Figure 5a).This option was the one utilized in the proposed research since it allows the line segments to be extracted more precisely.The "Canny" edge detector is applied for edge detection (Canny, 1986;Rong et al., 2014) and, thereby, Hough transform approach (Hough, 1962;Di et al., 2003;Hassanein et al., 2015) is applied for raw line segment extraction (Figure 5b).Finally, the image lines are extracted by detecting their endpoints.Each pair of intersected raw line segments are extended (or trimmed) to their point of intersection simply by a mouse-click on them.Thus, the endpoints of image lines are detected and the image lines are extracted (Figure 5c).This technique for image line extraction helps to reduce the runtime as it allows to exactly extract the desired line segment and ignore the non-desired ones.In addition, the used technique improves the accuracy of line extraction since the user is able to select specific lines in the image.
Figure 4. Application developed by MATLAB ® for metric measurements

Two orthogonal vanishing points detection
The step of locating vanishing points is the most critical step in the single image metric measurements process (Arslan, 2018).As prementioned, the three orthogonal vanishing points are hard to be detected since the lines perpendicular to building's façade usually do not appear in images that are captured in narrow streets (Figure 6).
where ( ) r is the perpendicular distance from principal point ( ) pp to image line, and θ ( ) denotes the angle included by the line perpendicular to image line and the u-axis.
First, the outliers are excluded by applying a RANdom SAmple Consensus (RANSAC) approach (Fischler & Bolles, 1981;Oh & Jung, 2012;Wildenauer & Hanbury, 2012) on image lines corresponding to horizontal and vertical object lines.These outliers could be the object lines on the façade that are not perfectly horizonal or vertical in nature, or they could be the image lines that are not correctly extracted due to image noise.Afterwards, two models (Model-I and Model-II) are used for vanishing points detection.Coordinates of (VPs) detected by RANSAC approach are considered as initial values for detecting (VPs).In Model-I we depend on the Cartesian coordinates of vanishing points, while in Model-II we depend on the peaks of histogram of angle coordinate and, by consequence, the polar coordinates.A detailed explanation for both models is given below.

Model-I:
This model is mainly based on calculating the Cartesian coordinates ( , ) Additionally, we have one constraint equation between Cartesian coordinates of the two (VPs) according to Li et al. (2010), Caprile and Torre (1990) as follows: A constrained least-squares solution (L.S.S.) (Chapter 9 of (Mikhail & Ackermann, 1982)) is applied for Equations ( 9) to (10) with the constraint Equation ( 11).

Model-II:
This model is mainly based on calculating the peaks of histogram of angle coordinates of the two orthogonal vanishing points ( ) In order to solve for the angle coordinates φ φ ( , ) intersection of bundle of convergent lines represented by each vanishing point obeys elliptical Gaussian distribution.In this elliptical Gaussian distribution, the length of minor axis is very short with respect to the major one.Moreover, the angle included between the ellipse's major axis and the line from the pp to VP (radial coordinate of VP) is very small (Li et al., 2010).Accordingly, the angle coordinate of each of the two orthogonal VPs is obtained by looking for the peak of histogram of angle coordinates of the intersection points of image lines corresponding to horizontal and vertical object lines.Then, we solve for the radial coordinates ρ ρ ( , ) vy vz of the two orthogonal vanishing points.The polar form of Equation ( 8) is: Given ( ) (16) Furthermore, we have one constraint between the polar coordinates of the two vanishing points adopted from Equation (11) as follows: A constrained least-squares solution (Chapter 9 of Mikhail and Ackermann (1982)) is applied for Equations ( 15) to ( 16) with the constraint Equation ( 17).
Then, Cartesian coordinates of the two orthogonal VPs are given by: = ρ φ .Cos

Image line refinement
A new approach for refining the extracted raw image lines is applied in this sub-section.The previously detected two orthogonal vanishing points are utilized for the refinement process.In other words, the extracting raw image lines are to be refined in accordance with the detected vanishing point coordinates.For each of the raw image lines of horizontal or vertical object lines, the raw image line is manipulated by modifying the position of its endpoint nearest to the vanishing point.A line is extended from the farthest endpoint of raw image line to the related vanishing point.Then, the nearest endpoint is moved to lie on the extended line such that the length of the refined image line is equal to that of the extracted raw one.Such manipulation should enhance the accuracy resulted by the metric measurement process.In Figure 7, the line (1-2) is the extracted raw image line, the line (1-2′) is the refined image line, and the point (VP) is the vanishing point.Applying the image line refinement approach resulted in modifying the endpoint of the extracted raw line that is nearest to the vanishing point (endpoint (2)) to the refined endpoint (2′).Consequently, the extracted raw image line (1-2) was modified to the refined image line (1-2′).

Length of object lines
The length of any horizontal or vertical object line is calculated applying cross-ratio theory on the refined image lines.The following is the methodology for calculation length of any vertical object line.For any horizontal object line, the object length can be calculated likewise.In Figure 8, given the vanishing point for vertical object lines ( ) z v , object length of vertical reference line (AB), image line of vertical reference line (ab), and two image lines (ae, bf) of object lines perpendicular to reference line in the object space at its two endpoints (A, B).Then, the length of demanded object line (CD) is calculated as follows.First, we extend the two perpendicular image lines (ae, bf) till they intersect the image line in question (cd) at points (a′, b′), respectively.The object length (A′B′) is equal to object length (AB) since both object lines (AB) and (CD) are parallel (vertical) and the two object lines (AE, BF) are perpendicular to line (AB).Then, the crossratio condition is applied for points ( , ', ', ) (Liu & Chuang, 2002;Arslan, 2018)

Experimental work
The dataset was created using the main rear cameras of three smartphones of different brand-names that are denoted as follows.Camera-I: iPhone_7_Plus, Camera-II: Samsung Galaxy Note8, and Camera-III: HUAWEI_ nova_7i.The image resolution of each of the three cameras was 12 mega pixels.Twenty images for chessboard pattern of 30×30 mm size were taken by each smartphone's camera for the purpose of camera calibration.Thereafter, a single image for the façade under study was captured by each camera and then undistorted.The cameras distance to facade was about 8 m in order to simulate the real case of narrow side street.The building's façade under study was about 9 m in width by 24 m in height.A recently calibrated SOKKIA SET05N total station was used for results validation.The distance measuring accuracy for the total station instrument was ±(2 mm + 2 ppm), whereas the angle measuring accuracy was 5".One horizontal and one vertical reference object lines were used for metric measurements along with image lines of two object lines perpendicular to each of reference object lines at their endpoints.In addition, 23 horizontal check lines with object lengths of 1. Model-I-B: based on the Cartesian coordinates of vanishing points applying image line refinement approach on the extracted raw image lines.

Model-II-A: based on the peaks of histogram of angle coordinates of vanishing points using the extracted raw image lines (without refinement).
Model-II-B: based on the peaks of histogram of angle coordinates of vanishing points applying image line refinement approach on the extracted raw image lines.
First, the three smartphones' cameras were calibrated utilized the MATLAB ® application for camera calibration described earlier.The format size along with intrinsic and lens distortion parameters for each camera are demonstrated in Table 1.Afterwards, each single images of façade captured by each camera were undistorted (Figure 10) utilizing the obtained calibration radial and decentring lens distortion coefficients shown in Table 1.Fifty-three image lines corresponding to horizontal object lines along with fifty-eight image lines corresponding to vertical object lines were extracted in each image utilizing the metric measurements MAT-LAB ® application.The results obtained by applying the RANSAC algorithm are illustrated in Table 2.The histograms for angle coordinates of the two orthogonal VPs φ φ ( , ) vy vz for the three cameras used for Model-II-A and Model-II-B are displayed in Figure 11 to Figure 13 where the left column of figures represents histograms of φ ( ) vy and the right column of figures represents histograms of ( ) φ .
vz Moreover, the values of image Cartesian and polar coordinates for the two orthogonal vanishing points for all models are illustrated in Table 3 to Table 5.The shown Cartesian coordinates are calculated with respect to image origin (the upper left corner), while the polar coordinates are calculated with respect to the principal point.Even though, the values of Cartesian and radial coordinates for Model-I-A and Model-II-A along with Model-II-B and Model-II-B are notably different to each other, the angle coordinates are close.This conforms to the elliptical Gaussian distribution that is previously mentioned (Li et al., 2010).However, the calculated object lengths were not significantly affected by those different values of Cartesian and radial coordinates of VPs.It is worth noting that the values of polar coordinates for VPs applying Model-I-A and Model-I-B are those values obtained by solving for Cartesian coordinates, then the radial and angle coordinates were calculated according to Equations ( 12) and ( 13), respectively.On the other hand, the values of angle coordinates for VPs applying Model-II-A and Model-II-B are the peak values for the angle coordinates' histograms illustrated in Figure 11 to Figure 13.Then, after solving for radial coordinates, the values of Cartesian coordinates were obtained according to Equation ( 18    line refinement approach improved the obtained results.The standard deviations for errors for the total 46 lines applying Model-I-B and Model-II-B respectively were 0.010 m and 0.011 m for Camera-I, 0.011 m and 0.012 m for Camera-II, and 0.013 m and 0.014 m for Camera-III. In general, the best results were obtained using Camera-I, Camera-II, and Camera-III, respectively.This complies with the cameras calibration results demonstrated in Table 1 where the values of ( ) a b , , which is equivalent camera focal length, are sorted descending from Camera-I to Camera-III.For the same object to camera distance, the image scale (and, by consequence, the calculated object coordinates' accuracy) is inversely proportional to the camera focal length.All in all, the obtained accuracy is realistic and reasonable especially when compared with traditional land surveying work where in most cases the desired accuracy is around 0.01 m.

Conclusions
This paper investigated the possibility of utilizing the lowcost smartphone's camera for performing metric measurements of building's façades existing in narrow streets.The proposed technique is an adapted semi-automated technique based on detecting only two orthogonal vanishing points on single undistorted image captured by smartphone's main rear camera.Three smartphones of different brand-names were utilized applying two models for vanishing points detection.Model-I solved for the Cartesian coordinates of vanishing points, whereas Model-II solved mainly for the peaks of histogram of angle coordinate.The angle coordinates of vanishing points calculated by both models were close to each other in contrast to Cartesian and radial coordinates.A novel image line refinement approach based on the detected two orthogonal vanishing points was applied.A total number of 46 object lines (23 horizontal and 23 vertical lines) with lengths of 1.513: 3.194 m were used for results validation.The results obtained by both models utilizing the three smartphones' cameras were close to each other with slightly better results for Model-I.The best results were obtained by Camera-I which is the camera with the least equivalent focal length.Additionally, the solution was improved applying the image line refinement approach where the overall standard deviations obtained by Camera-I were 0.010 m and 0.011 m by Model-I and Model-II, respectively.In general, the obtained accuracy was reasonable especially when compared with the accuracy that is usually obtained by the traditional land surveying work.
Future work will develop a methodology for dealing with more complicated façades such as façades with circular and irregular features.In addition, more attention will be paid for extracting image features full automatically.Moreover, stereoscoping with smartphone's camera will be studied.

,
u v .The upper left corner of image represents the image coordinate system's origin.The vanishing point for image lines corresponding to horizontal object lines (lines parallel to Y -Axis) is denoted as ( ) y v .Whereas, the vanishing point for image lines corresponding to vertical object lines (lines parallel to Z -Axis) is denoted as ( ) z v .The principal point is denoted as ( ) pp .

Figure 1 .
Figure 1.Object coordinate system vs. image coordinate system Accordingly, the solution presented in this research relies on detecting only two orthogonal vanishing points.The two orthogonal vanishing points ( ) y v and ( ) z v for horizontal and vertical object lines, respectively are detected.Given the image line parameters θ ( , ), r then the Cartesian coordinates ( , ) v v u v for any of the vanishing points are:

zv
Equation is the equation that relates the image line parameters and vanishing point's coordinates.Given ( ) y n image lines corresponding to horizontal object lines and ( ) z n image lines corresponding to vertical object lines, we solve for the four unknowns ( , , , ) vy vy vz vz u v u v using the following minimization equations:

Figure 5 .Figure 6 .
Figure 5.Samples for extracted image lines: a -line selection buffer; b -raw line segments; c -image lines extended to their endpoints , we have the following minimization equations to solve for the two unknowns ρ Figure 7. Image line refinement − CD DB CB , or: 922 to 2.818 m besides 23 vertical check lines with object lengths of 1.513 to 3.194 m were used for results verification.Image lines for both of horizontal and vertical object lines appearing in the image captured by Camera-II, as a sample for other cameras, are illustrated in Figure 9a and 9b, respectively.In the shown figures, the image reference lines are shown in blue, image perpendicular lines are shown in yellow, image check lines are shown in red, and image endpoints for lines are shown in blue.Four models were applied for metric measurements as follows: Model-I-A: based on the Cartesian coordinates of vanishing points using the extracted raw image lines (without refinement).

Figure 8 .
Figure 8. Length calculation of object line using cross-ratio

Figure 10 .
Figure 10.Undistorted single images of the three cameras: a -Camera-I; b -Camera-II; c -Camera-III

Figure 11 .
Figure 11.Histograms for angle coordinates of two orthogonal VPs for Camera-I

Table 1 .
Camera parameters

Table 2 .
Results obtained applying RANSAC algorithm

Table 3 .
Image cartesian and polar coordinates of vanishing points for Camera-I vy u (pixels) vy v (pixels) vz u (pixels) vz v (pixels)

Table 4 .
Image cartesian and polar coordinates of vanishing points for Camera-II