Ornamental shade trees are used to beautify the l and scape and served as wind barriers. These trees are usually planted singly at a site and consequently are susceptible to be attacked by insects and diseases. Applications of foliar pesticides are an economic and effective control measure to protect these trees from the pest damages. However,spray applications presents special problems because the locations of these trees are usually near residential areas,commercial districts,industrial and recreational parks,water resources or ecological sensitive regions. Conventional spraying systems deliver constant rates of chemicals to these trees regardless of tree size,shape and foliage density as well as the spacing between trees. Thus,a large portion of the pesticide is wasted on non-targeted areas(Zheng et al.,2006). Sensor technologies to detect trees and their configurations and minimize off-target losses of chemicals for target-oriented spraying systems are needed when spraying these trees.

However,target-oriented sprayers have great difficulties to maintain a consistent distance between the sprayer nozzle and the target trees because these trees may have r and om positions. Also,for tall trees the sprayers should have adequate distance away from the trees to allow sprays to cover the entire tree height. Despite distance variability,to ensure sprays are applied at the proper time after sensors detect target trees,different delay times are required to discharge sprays. That is,the sprayers must calculate the delayed time required between the tree detected and the tree sprayed. Thus,critical to the sprayers is sensors that can detect the target tree and determine its distance from the sprayers for target-oriented spray applications.

Ultrasonic sensors are a type of sensors that have been used to detect trees to control spray applications(Giles et al.,1988; Tumbo et al.,2002; Zaman et al.,2004). Jeon et al.(2011b; 2012)used these sensors to detect the distance between the sprayer and the surface of tree canopy to calculate tree size. However,the detection range of a single ultrasonic sensor is limited. If multiple sensors were used,interference between adjacent sensors would compromise their accuracy(Tumbo et al.,2002; Zaman et al.,2004; Jeon et al.,2011a). Another type of sensor,LIDAR(Light Detection and Ranging)has been used to measure target tree distances and canopy sizes(Chen et al.,2012,Wei et al.,2005; Lee et al.,2008; Rosell Polo et al.,2009),but they are relatively expensive.

Machine vision technology implanting image acquisition systems has been widely used to detect crops and weeds(Meyer et al.,1998; Hu et al.,2009; Wang et al.,2011; Steward et al.,2002; Tian et al.,1998; Jeon et al.,2011a). In other areas,this technology incorporating stereoscopic mosaicking technique with multi cameras has been used to determine the depth of view for targets in satellite mapping(Song et al.,2003; Bignalet-Cazalet et al.,2010),marine survey(Nagahdaripour,1998; Rzhanov et al.,2000),remote sensing practice(Bielski et al.,2007; Joshi et al.,2010),medical devices for clinical diagnosis(Miranda-Luna et al.,2008) and other areas(Guestrin et al.,1998).

To achieve target-oriented spraying systems,the Charge-Coupled Device(CCD)technology with image mosaicking technique,which uses one camera to detect trees and measure tree characteristics,has the potential to control spray applications. With this technique,when real-time video images of trees are captured,a set of relative color index is generated to obtain the target tree structure characteristics from the segmentation of target trees and background(Zheng et al.,2005; Zhou et al.,2010). Employing the real-time tree structure information into a control system,a target-oriented sprayer could automatically control spray output based on target needs. However,the use of one camera without a reference target to measure the distance between the camera and trees is a limiting factor(Sawhney et al.,1999). Also,for a tall tree,the images from one camera may not be able to cover the entire tree height if the sprayer is close to the tree.

The use of mosaicking image technique to measure tree distances and increase pesticide spray accuracy for target-oriented sprayers has not been reported. To address these problems,this research proposed a two-camera system to simultaneously capture images of entire trees and then use stereoscopic mosaicking technology to combine these separate images into a new image to obtain the tree structure and distance from the cameras. Thus,the objective of this research was to develop and validate a dynamic tree image mosaicking system derived from a geometrical algorithm based on a configuration of two digital cameras to determine the distance between the target tree and the cameras along with the tree structure.

Binocular stereoscopic vision function states that an object position in three dimensional coordinates can be determined from two perspective images obtained from two angles when the two imaging surfaces form an intersection line. Specially,based on the triangulation principle,the positions of three-dimensional objects could be obtained through the corresponding relationship of the pixels from the two images.

The three dimensional position of the object can be determined by the transformation matrix from the object points in the spatial coordinates to the image coordinate system. In the image mosaicking process,the corresponding points of two images can characterize the relative position of the two images. However,to obtain the object distance information from the images,the first step was to know the relative spatial position of two cameras and its own parameters such as the angle of two camera optical axes,focus distance,and the angle of field-of-view for each camera.

As shown in Fig. 1,when the position of two cameras was assigned,the overlapping area of two images of the same target tree varied with only the distance between the tree and cameras. A shorter distance could have fewer pixels in the overlapping area. The distance information of the target tree could be obtained by calculating the ratio between the height of the overlapping area and the height of two combined images. This ratio is defined as the overlapping coefficient of mosaicked images. If the entire target tree was treated as a plane in the process of image mosaicking,the tree distance was actually the distance between the plane of the target tree and the line across the two focus points of two cameras.

	Fig. 1 The overlapping area of two images for a target at two different distances (a) 0.3 m and (b) 0.5 m from the cameras

Fig. 2 shows the geometry of relative positions of two cameras to the target tree. Parameters in the geometry are defined as: α for the angle of two camera optical axes,b for the distance between the focus points of two cameras,θ for the field-of-view angle of each camera,D for the distance from the line of two focus points to the target tree plane,H for the visual height of target plane,and L for the height of the overlapping area. When the target plane is projected to the plane that is parallel to the imaging plane,the projected visible height is H',and the projected overlapping height is L',then the overlapping coefficient K should be,

	Fig. 2 Geometry of spatial positions of two cameras and target tree plane

Analyzing the geometry shown in figure 2 yields,

and

D actually is the sum of distance d₁(from the plane of two focuses to the interception point of the two camera views) and the distance d₂(from the interception point to the target plane)(Fig. 2). That is,

The two distances d₁ and d₂ have the relationship with parameters b,θ,α,and L as,

Combining equations(1),(2),(3),(4),(6) and (7)into equation(5)yields,

Equation(8)presents the relationship between D and parameters K,θ,α,and b. K is not a constant,but it can be determined through mosaicking two images by matching corresponding points of the same location on the pair of images taken by two cameras. Parameters θ,α,and b are constants when two cameras are mounted on a fixed st and . However,the measurements of θ,α,and b are difficult and often inaccurate. To simplify equation(8),a new constant w is introduced,

By substituting equations(6) and (9)into equation(8),the distance between cameras and the target plane becomes,

Hence,the involvement of four parameters K,θ,α,and b for determination of D is now simplified into three parameters K,d₁ and w. Because d₁ and w are also constants,D only varies with K. Determination of d₁ and w values can be achieved by taking two pairs of target images at two distances(D₁ and D₂)between the target plane and the two cameras,and then determine two K values(K₁ and K₂)from mosaicking the two pairs of images. From equation(10),D₁ and D₂ can be expressed as,

Then,d₁ and w can be calculated from the known values of K₁,K₂,D₁ and D₂,

Therefore,with equation(10)the distance between the camera and the target tree can be determined with the calculation of K values through the process of mosaicking pairs of tree images.

An image mosaicking system was developed(Fig. 3)to capture and mosaic sequential tree images from two cameras for measuring target tree distances from equation(10). The mosaicking system consisted of a video capture module,a camera parameter adjustment module,a tree image mosaicking program module,and a tree distance acquisition module. The functions of these modules included: setting up camera parameters to obtain clear images,capturing sequential images with two cameras,generating tree image sequences,mosaicking and displaying sequential images,refining mosaicked tree images by filtering background noise,calculating overlapping coefficient K,and calculating target tree distance D. The system also had flexibility to mosaic images manually.

	Fig. 3 An image mosaicking system with modules and flow charts to capture and mosaic tree images

A hardware platform was assembled to determine the two parameters d₁ and w with the algorithm derived from equations(13) and (14). The platform consisted of a computer,two factory-calibrated digital cameras(Model # DH-SV1310FC,Da Heng Co.,Ltd,Beijing),a framework for mounting the cameras,a 1394 card and simulated trees. To obtain wider field-of-view of videos and images for a tall tree,a camera st and fixture with a fasten device was designed to mount the two cameras and to adjust the angle of the two camera optical axes(Fig. 4).

	Fig. 4 Real-time mosaicking experimental system to manipulate two cameras for capturing sequential tree images

The cameras were coupled with a CCD to capture and store images in a digital memory. The cameras were able to capture tree images at a maximum speed of 30 frames per second(fps). The process of capturing sequential images included defining critical variables,assigning the callback function,turning on the digital cameras,defining camera parameters(focal length,white balance,exposure time and aperture),capturing and processing video images,and turning off the cameras. During the real-time image mosaicking process,the matching area of each pair of images and the final combined image were shown on the computer screen.

The camera parameters were adjusted before the image mosaicking process. This was because the focal length,white balance and aperture affected image clarity,color and image brightness,respectively,as a result to affect the image segmentation. The white balance was adjusted manually through the color finding table with the G component as a benchmark and the ratio of R and B components as input. During the white balance adjustment,all the field-of-view of the lens was made as white,and the automatic white balance adjustment ratio was repeated with the camera Get-White-Balance-Ratio function until clear and precise images were captured.

For a moving object,the camera exposure time affected the clearness of the captured images. The minimum and maximum exposure times of the cameras used in the test were 20 μs and 1 s,respectively. The target image motion(δ,μm)in the camera display was,

where,v was the camera travel speed when passing the target tree(m·s^-1),t was the exposure time(s),and λ was the optical magnification(or the ratio of target tree image size to the actual tree size). To obtain the satisfied clearness of images,the target image motion(δ)should be shorter than the pixel size of image resolution. For the cameras used in the test,the CCD display height was 3.096 mm and image pixel was 6.45 μm. That is,δ should be shorter than 6.45 μm.

To verify the accuracy of the distance algorithm module,the values for geometry parameters d₁ and w were calibrated and determined. Steps included: placing a 2.0 m tall tree at 1.0 and 7.0 m away from the camera st and fixture,taking two pairs of images with the two cameras for the two tree positions,mosaicking each pair of images and using equation(1)to calculate overlapping coefficient K₁ and K₂,and then using the equations(13) and (14)to calculate the camera setting parameters d₁ and w.

An experiment was conducted to verify the accuracy of the algorithm developed for equation(10)to calculate the tree distance D and the overlapping coefficient K. Steps for the experiment were: placing the target trees at 11 different distances from the cameras with a 0.50 m increase for the distances ranging from 1.50 to 6.50 m,moving the tree at 2 m·s^-1 speed perpendicularly to the camera optical views,taking sequential pairs of images at different distances,automatically mosaicking pairs of images by pixel overlay of overlapping regions(Zhou et al.,2011)to obtain the overlapping coefficient K at each distance,using equation(10)to calculate the target tree distance with the known values of d₁ and w obtained,and then comparing the actual and calculated distances to verify the module accuracy.

As shown in equation(15),the target image motion(δ)linearly increased as the camera travel speed(v),camera exposure time(t) and optical magnification(λ)increased. Choosing values of v,t,and λ must meet the requirement that δ should be shorter than the CCD display pixel resolution 6.45 μm. For example,when the distance between the camera and target tree was 5 m,the maximum tree height that the camera could display was 12 000 mm. That is,λ=3.096/12 000=0.000 258. If the camera travel speed(v)was 5 m·s^-1(18 km·h^-1) and δ was equal to 6.45 μm(the image resolution in CCD),the maximum exposure time(t)from equation(15)should be 0.005 s(1/200 s). Similarly,if t was chosen with the SetExposureTime function in the camera as 0.002 s(1/500 s),the maximum travel speed(v)should be 12.5 m·s^-1 which was much higher than the travel speed normally used for pesticide sprayers. Therefore,the cameras used in the system were fast enough to capture quality sequential images of trees at high camera travel speeds because their lowest exposure time setting was 20 μs(0.000 02 s).

Because an extensive time was required for mosaicking sequential pairs of images,not all the frames captured at the 30 fps speed could be processed. The processing frequency varied with the image processing time which depended on the template radius of the paired images. During each cycle,the most recent frames in the storage from two cameras were coupled as the start point,and then pairs of partial tree images captured by the two cameras were mosaicked with the tree image mosaicking module.

To maximize mosaicking accuracy,a continuous sorting method was used to match the pairs of images while each pair of images was processed independently to form a new mosaicked image. Every mosaicked image was then compared with the previous mosaicked image to obtain an integrated value for the overlapping coefficient(K). After numerous comparisons of sequential images for the same tree,the average K value was calculated. The experimental results demonstrated that three to five pairs of images could reach a stable K value.

Fig. 5 shows the two images(top left and lower left)captured with two cameras and then were mosaicked into a combined image(middle)in real time. The "green extraction" frame shown at the lower right corner in Fig. 5 was used to make decisions on which segments to be selected for the image mosaicking process. The selected segments were also displayed at the lower right bottom of the interface.

	Fig. 5 The interface of dynamic tree image sequence mosaicking system

For calibrated result,the overlapping coefficient K obtained from the image mosaicking module was 0.191 7 at D₁=1.0 m and was 0.458 3 at D₂=7.0 m. Replacing these values into equations(13) and (14)yields the geometry parameters d₁=0.618 8 and w=1.989. Then,equation(10)becomes,

After the values of d₁ and w were determined for the image mosaicking hardware platform,the target tree distance could be calculated with equation(16)after automatically mosaicking pairs of images along with the overlapping coefficient(K)value at that distance. Fig. 6 shows the actual target tree distance(D)as the function of K,presented with equation(16). The overlapping coefficient increased non-linearly as the distance increased. This was because the overlapping area increased as the distance increased for the constant camera optical angles. The change in K value became obvious when the cameras were close to the target tree while the K value tended to become a constant as the distance between the cameras and the target tree increased. Therefore,it was critical to calculate the K values accurately for obtaining the accurate distance(D)when the cameras were close to the target tree.

	Fig. 6 The actual distance between two cameras and the target tree (D) as the function of overlapping coefficient (K) obtained from mosaicked pairs of images

The K values obtained through the real-time mosaicked pairs of target tree images at 11 different distances were illustrated in table 1. The mosaicked K values agreed well with the calculated K values that employed the actual distances into equation(16)(Fig. 7). The linear coefficient for the calculated and mosaicked K values was 0.966 with r² of 0.995. Therefore,the image mosaicking module was able to map the pairs of images with a high accuracy.

	Fig. 7Comparison of K values mosaicked through the real-time tree images and calculated with equation (16) at 11 different distances

Tab. 1 also compares the actual target tree distances and calculated tree distances with equation(16)through the real-time mosaicking pairs of tree images at 11 different distances. Within the range from 1.50 to 6.50 m,the maximum difference between the actual and calculated distances was 0.27 m. The error occurred mainly due to the limitation of image resolution to match the same points on the pairs of images during the mosaicking process. There was no trend showing the error had a relationship with the actual distance(Fig. 8). Within the distance range tested,the maximum error was 7.5%. For shade trees planted on roads,streets,or parks,their height are normally taller than 2.0 m,Compared to these tree heights for target-oriented spray applications,this error should be acceptable.

Tab.1Calculated tree distances and mosaicked overlapping coefficient obtained from the image mosaicking system at different target tree positions

Tab.1Calculated tree distances and mosaicked overlapping coefficient obtained from the image mosaicking system at different target tree positions Target tree position Tree distance to cameras, D/m Difference between actual and calculated distances/m Mosaicked overlapping coefficient K Actual Calculated 1 1.50 1.50 0.00 0.295 8 2 2.00 2.15 0.15 0.358 3 3 2.50 2.60 0.10 0.383 3 4 3.00 3.15 0.15 0.404 2 5 3.50 3.61 0.11 0.416 7 6 4.00 3.99 -0.01 0.425 0 7 4.50 4.77 0.27 0.437 5 8 5.00 5.09 0.09 0.441 7 9 5.50 5.46 -0.04 0.445 8 10 6.00 5.89 -0.11 0.450 0 11 6.50 6.39 -0.11 0.454 1

	Fig. 8Measurement errors of tree distances from the automatic image mosaicking process

Hence,for the pesticide application on high range trees,the sequential pairs of images with the whole tree can be acquired and mosaicked which the integrated tree image can be segmented from the background. Then the high-range target-oriented precision pesticide sprayer will be controlled to spray the pesticide on the target trees according to the mosaicked and segmented tree images.

A geometric algorithm,established with 3-D binocular stereo vision concept was developed to calculate the distance between targeted trees and cameras. The targeted tree distance was the function of the mosaicked image overlapping coefficient,camera mounting position,the angle of two cameras optical axes,and optical view angles. The algorithm simplified the function comprising the overlapping coefficient and two measured geometry parameters at two known distances.

Based on the geometric algorithm,an image acquisition and mosaicking system that included an image mosaicking program and hardware platform was developed to capture,process and mosaic sequential images of trees in real time. Validation of the system demonstrated that it was able to accurately measure the tree distance within a 6.5 m range. The difference between the measured and actual distances was less than 0.27 m or 7.5%. Compared to the spacing between trees,this difference was negligible for high-range target-oriented pesticide spray application systems. More precise delivery of pesticides to target trees would be realized by integration of the image mosaicking system with precision spraying systems.