Golf is one of the most developed and popular sports. The number of active golf players is estimated at 50 million and golf course development has become the fastest growing property sector in the world. However,how to train a golfer to make a proper and accurate golf swing is critical to the success of the golf training course^[1]. Previously,the training was usually guided by professional golfers,which involved extensive human recourse. In recent years,there was a trend to use standard swing procedures for golf training,which considered many important parameters of swing,such as club head trace^[2-5],swing plane^[6],hub path^[7],leg movement^[8],and even wrist angle^[9-10]. However,among these parameters,the key is to accurately capture the swing movement. For this purpose,two kinds of motion capture (Mocap) systems are mainly used in recent years^[11]: optical Mocap (OMocap) system^[12-15] and wearable micro-sensors base Mocap system (MMocap)^[16-20]. The OMocap systems require to attach reflective markers on golfer’s body segment,and then positions of the marker are obtained via multiple high-speed cameras around the golfer. The MMocap systems also require to place micro-sensors on golfer’s body segments for motion reconstruction. Although these two kinds of Mocap systems have been used successfully in many applications,both systems require the golfer to wear some extra devices,which are intrusive and make the golfer uncomfortable during the swing movement.

Recently,newly introduced Microsoft Kinect,which is a contactless motion capture device using an RGB-D sensor providing synchronized color and depth images,has shown some potential to overcome the pitfalls of the traditional Mocap systems^[21]. The main challenge for the Kinect is its poor performance due to occlusions and mixing up of body parts^[22]. To tackle this problem and make it applicable for the golf swing analysis,some ad-hoc solutions to improve the estimation accuracy have been proposed so far. For example,Zhang et al.^[23-24] used the Kinect to capture the 3D skeleton coordination of a golfer who was performing swing,and then used serial correlation model to score and recognize grade of golf swing. Lin et al.^[25] utilized the Kinect to capture the swing motion,and automatically identified the 6 common mistakes on swing motion. Shen et al.^[26] tried to tackle the occlusion problem and presented an exemple-based method to learn and correct the initially estimated poses from Kinect. Although these methods made some progress to improve the wrongly recognized swing posture from Kinect,the challenge is still not very well solved yet due to the ignorance of motion similarity contained in the swing dynamics.

However,there are similarities among swing movements performed by the same person: 1) spatial similarity (the relative movements among joints during swing movement are similar if the swings are performed without external interference) and 2) temporal similarity (the swing periods are similar). These similarities,if applied properly,can be used to further improve the accuracy of motion information against self-occlusion and reconstruct 3D golf swing from low resolution depth image sequences. In this paper,due to the advantage of fusing spatial and temporal information,a dynamic Bayesian network (DBN) model-based golf swing reconstruction algorithm for Kinect is proposed. The algorithm integrates the spatial relationship among joints and their movement dynamics. The DBN model is trained using actual golf swing movement data. The joint positions generated from Kinect are optimized by the model. The reconstruction of golf swing is performed using these optimized positions. We have compared our position outputs with the commercial OMocap system MAT-T^[27] and the method proposed in Ref.^[26]. The good performance results have shown that the proposed algorithm achieves comparable tracking accuracy to the MAT-T system and improves the Kinect outputs much more than the methods mentioned in Refs.^[22] and ^[26].

1 DBN model-based swing movement reconstruction

Generally,the human body can be regarded as a skeleton model shown in Fig. 1 (a). The skeleton model contains five chains. Each chain is a set of segments connected by joints. The joint positions are key elements to the whole swing procedure reconstruction. The segments in a chain are considered as rigid bodies. To reconstruct the golf swing accurately,the exact positions of the key joints should be acquired.

The above golfer model includes 15 key joints include: head,neck,torso,left and right shoulders,left and right elbows,left and right hands,left and right hips,left and right knees,and left and right feet. Their position information during time represents the whole body movement precisely. These 15 key joints construct a hierarchical structure of a golfer containing the five serial chains,which is shown in Fig. 1 (b). In the hierarchical structure,the torso joint is considered as the root of all the chains. To construct our DBN model,the positions of joints are focused on in one arbitrary chain. It can be derived similarly in other four chains by using the DBN model of the chosen chain.

1.1 Model representation

In the DBN model,5 symbols are used to represent the states and observations:

s_tⁱ： The relative position of the i_th joint in chain in its parent joint’s coordinate system (JCS) at time t.

X_tⁱ： The absolute position of the i_th joint in chain in global coordinate system (GCS) at time t.

Y_tⁱ： The position observation of the i_thjoint in chain in GCS at time t.

n： The order number describing the dynamics of s_tⁱ in the whole swing.

m： The number of joints in the chain.

The exact structure of the DBN model is shown in Fig. 2. The static structure of joint chain and the dynamic structure of one joint are shown,respectively. During model construction,the dynamics of the relative position of one joint could be first-order or multi-order Markov. The precise order number “n” is determined through practice.

The purpose of the DBN model is to estimate more precise positions of the joints in GCS during the whole swing,i.e.,to find the most probable position of X_tⁱ by calculating the posterior probability Pr(X_tⁱ|Y_1:t^0:i). Based on the structure of the proposed DBN model,a preliminary iteration sequence is raised to solve the estimation problem X₁⁰,…,X₁^m,X₂⁰,…,X₂^m,.…X_t⁰,…,X_t^m.

$X_1^0, \cdots ,X_1^m,X_2^0, \cdots ,X_2^m,. \cdots X_t^0 \cdots ,X_t^m.$

(1)

This means reconstructing the swing of the golfer accurately,and every joint’s position in GCS from every frame should be acquired precisely. According to (1) ,the position estimation results should be acquired sequentially,following the chain order in one frame,and in the next frame the same order should be followed. The position estimation X_tⁱ is related to its parent X_t^i-1 in the the same frame and its relative position sⁱ_t-1in the previous frame.

1.2 Model inference

As discussion above,the key to acquire ment of precise position of X_tⁱ is to calculate the posterior probability Pr(X_tⁱ|Y_1:t^0:i)， which is related to Pr(X_t^i-1|Y^0:i-1_1:t)and Pr(sⁱ_t-1|Y^0:i_1:t-1). This could be derived step by step from the joint probability distribution which can be written as

$\eqalign{ & Pr(s_{1:t}^{^{0:i}},X_{1:t}^{0:i},Y_{1:t}^{0:i}) = \cr & {{Pr(s_{1:t}^{0:i - 1},X_{1:t}^{0:i - 1},Y_{1:t}^{0:i - 1}){\rm{ }}Pr(s_{1:t}^{^{0:i}},X_{1:t}^{0:i},Y_{1:t}^{0:i})} \over {Pr(s_{1:t}^{0:i - 1},X_{1:t}^{0:i - 1},Y_{1:t}^{0:i - 1})}}. \cr & Pr({X_t}^i|X_t^{0:i - 1},{s_t}^i){\rm{ }}Pr({s_t}^i|s_{_{t1:tn}}^i){\rm{ }}Pr({Y_t}^i|{X_t}^i). \cr} $

(2)

Then,by doing parameter elimination,the probability contains only s_tⁱ and X_tⁱ and can be derived

$\matrix{ {Pr\left( {{s_t}^i,{X_t}^i,Y_{1:t}^{0:i}} \right) = } \hfill \cr {Pr\left( {{Y_t}^i|{X_t}^i} \right){\rm{ }}Pr\left( {Y_{1:t - 1}^{0:i - 1}} \right)\left[ {\sum\limits_{{X_t}^{i - 1}} {Pr} \left( {{X_t}^{i - 1},Y_{1:t}^{0:i - 1}} \right){\rm{ }}Pr\left( {{X_t}^i|{X_t}^{i - 1},{s_t}^i} \right)} \right]\cdot} \hfill \cr {\left[ {\sum\limits_{s_{t - 1:t - n}^i} {Pr} \left( {s_{t - 1}^i,Y_{1:t}^{0:i - 1}} \right){\rm{ }}Pr\left( {{s_t}^i|s_{t - 1:t - n}^i} \right)} \right].} \hfill \cr } $

(3)

From (3) ,the posterior probability Pr(X_tⁱ|Y_1:t^0:i) and Pr(s_tⁱ|Y_1:t^0:i) can be derived

$\matrix{ {Pr\left( {{X_t}^i|Y_{1:t}^{0:i}} \right) = C\cdot{\rm{ }}Pr\left( {{Y_t}^i|{X_t}^i} \right)\cdot} \hfill \cr {\sum\limits_{{X_t}^{i - 1}} {Pr} \left( {{X_t}^{i - 1}|Y_{1:t}^{0:i - 1}} \right)\sum {s_t}^iPr\left( {{X_t}^i|{X_t}^{i - 1},{s_t}^i} \right)\cdot} \hfill \cr {\sum\limits_{s_{t - 1:t - n}^i} {Pr} \left( {s_{t - 1}^i,Y_{1:t}^{0:i - 1}} \right){\rm{ }}Pr\left( {{s_t}^i|s_{t - 1:t - n}^i} \right)} \hfill \cr } $

(4)

$\matrix{ {Pr\left( {{s_t}^i|Y_{1:t}^{0:i}} \right) = } \hfill \cr {C\cdot\sum\limits_{s_{t - 1:t - n}^i} {Pr} \left( {s_{t - 1}^i,Y_{1:t - 1}^{0:i}} \right){\rm{ }}Pr\left( {{s_t}^i|s_{t - 1:t - n}^i} \right)\cdot} \hfill \cr {\sum\limits_{{X_t}^i} {Pr} \left( {{Y_t}^i|{X_t}^i} \right)\sum\limits_{{X_t}^{i - 1}} {Pr} \left( {{X_t}^{i - 1}|Y_{1:t}^{0:i - 1}} \right){\rm{ }}Pr\left( {{X_t}^i|{X_t}^{i - 1},{s_t}^i} \right).} \hfill \cr } $

(5)

In (4) and (5) ,$C = {\rm{ }}{{Pr(Y_{1:t}^{0:i - 1}){\rm{ }}Pr(Y_{1:t - 1}^{0:i})} \over {Pr(Y_{1:t - 1}^{0:i - 1}){\rm{ }}Pr(Y_{1:t}^{0:i})}}$ can be considered as a constant. Recalling the preliminary iteration sequence to estimate X_tⁱ in (1) and considering (4) and (5) ,a modified iteration sequence is proposed to estimate every joint position in chain at time t:

${s^0}_{t - 1},{X_t}^0, \ldots ,{s^m}_{t - 1},{X_t}^m.$

(6)

1.3 Conditional probability distribution (CPD) definition

The calculations of Pr(X_tⁱ|Y_1:t^0:i) and Pr(s_tⁱ|Y_1:t^0:i) need three key elements: spatial relationship Pr(X_tⁱ|X_t^i-1,s_tⁱ)， temporal relationship Pr(s_tⁱ|s_t-1:t-nⁱ), and likelihood Pr(Y_tⁱ|X_tⁱ). For defining and training simplicity,all these three elements are assumed to follow the normal distribution. After definition,the parameters in these three elements can be learned before reconstruction by training previous motion data gained from the same golfer.

1.3.1 Spatial relationship

According to the human skeleton model,the joint positions are the only elements to be discussed. Therefore the states s_tⁱ and X_tⁱ are both joint positions. The difference between the two kinds of states is that s_tⁱ is the position of one joint in its parent JCS while X_tⁱ is the position in GCS. Apparently,X_tⁱ can be gained by integrating s_tⁱ and X_t^i-1， according to the chain structure. The spatial relationship between parent and child joints can be written as

$Pr({X_t}^i|{X_t}^{i - 1},{s_t}^i) = N({X_t}^{i - 1} + {s_t}^i,{Q_i}).$

(7)

In (7) ,Q_i is the process noise from previous training sets and their ground truths of one golfer.

1.3.2 Likelihood

The observation Y_tⁱ is generated from state X_tⁱ. Since they are all measured in GCS and only have measurement errors,the likelihood can be written as

$Pr({Y_t}^i|{X_t}^i) = N({X_t}^i,{R_i}).$

(8)

1.3.3 Temporal relationship

For each golfer,the motion of joints should be unique and repeatable if the swing is performed again by the same person. This assumption is taken into consideration to train different motion models of each joint for different golfers. The motion of joints is only considered in their parents’ JCS to eliminate the affection of their parent joints and acquire “pure” motion models. Each joint can acquire a motion model by training previous motion data. Following the normal distribution,the temporal relationship of one joint can be described as

$Pr({s_t}^i|{s_{t - 1:t - n}}^i) = N({A^i}_{t - 1}{[{s^i}_{t - 1}, \ldots ,{s^i}_{t - n}]^T},\sum {_{i - 1}^i} ).$

(9)

In (9) ,the matrices Aⁱ_t-1and σⁱ_t-1 contain the parameters which need to be trained to get from s_t-1:t-nⁱto s_tⁱ.

2 Experiments and discussion 2.1 Setup

The MAT-T system and the algorithms in Refs.^{[22, 26]} are used to evaluate the reconstruction performance of the proposed algorithm. The placement of the 6 cameras and the application environment are shown in Fig. 3,while the Kinect is put in front of the golfer at a distance of about 2.5m with a height of about 1m from the ground to make sure the whole swing motion can be captured. The joint position observations are provided by OpenNI skeleton functions^[28]. These functions can generate preliminary joint positions from the RGB-D information.

Five golfers (4 males and 1 female) were participated in our experiment. Each of them repeated 6 swings (including stance,backswing,downswing,and follow-through). During the experiment of one golfer,all the 6 swings are used as testing set by turn. While 1 swing is testing set,the other 5 swings are training sets to generate “prior knowledge” of the DBN model. This ensures that every swing can have a corresponding reconstruction result. The two kinds of devices (i.e. 6 cameras MAT-T system and Kinect) capture the swing synchronously. The sampling rate of the cameras from MAT-T system is 180Hz,while the Kinect is 30Hz.

To illustrate the feasibility of our algorithm,the joints and body segments which have the most significant movements or rotations were chosen. In our implementation,we chose the hand arms to evaluate the performance of the proposed algorithm. The shoulder width (SW) and arm lengths (left upper arm (LU),left lower arm (LL),right upper arm (RU),and right lower arm (RL)) were also taken into consideration.

The length of above 5 body parts can be acquired easily. By comparing with outputs of MAT-T system,we calculate the error ratio e between our outputs and MAT-T system outputs:

$e = {{L - \tilde L} \over L},$

(10)

In (10) ,L is the body part length from MAT-T system,while ${\tilde L}$ is the body part length from our model.

Similar to the body part length comparison,the joint position difference between two kinds of outputs is calculated. To evaluate the difference,as mentioned in Ref.^[26],the mean value of the sum of joint errors (msJE) and the mean value of msJE (mmsJE) are calculated,

${\rm{msJE}} = {{\sum\limits_{t = 1}^T {\sum\limits_{i = 1}^N {\left| {{X_t}^i - {{\tilde X}_t}^i} \right|} } } \over 2},$

(11)

${\rm{mmsJE}} = {{\sum\limits_{t = 1}^T {\sum\limits_{i = 1}^N {\left| {{X_t}^i - {{\tilde X}_t}^i} \right|} } } \over {T \times N}}.$

(12)

In (11) and (12) ,X_tⁱ is the position of joint i at time t from MAT-T,and ${{{\tilde X}_t}^i}$ is from our model. N is the number of joints used to represent golfer (in our experiment N=15) . T represents the swing period,which equals the frame number Kinect captured during the whole swing.

2.2 Order

number selection

Since the proposed model is multi-order DBN,a proper order number should be determined before the experiment. To evaluate the model performance,i.e.,to quantify the average accuracy on the whole 5 testing data,the msJE is calculated and shown in Fig. 4. The order number rises from 1 to 7,which causes the msJE to drop first and rise when the order number is above 5. For economizing training and calculating consumption,other order numbers are not considered. Apparently,the order number of our model is chosen to be 5.

2.3 Computation consumption

The proposed DBN model is constructed and trained using the BNT (Bayesian network toolbox) in Matlab (R2011b). The joint positions are modified offline with a speed of 10fps (CPU: i5-2400 3.1GHz).

2.4 Comparison with MAT-T

The hand position outputs of our algorithm,the MAT-T system,and the original Kinect are shown in Fig. 5. Because the joint moves in 3D space,the three components (x,y, and depth) of positions are shown.

In Fig. 5,the outputs of the former two kinds of results are only slightly different. In general,our algorithm’s performance is comparable with MAT-T in acquiring joints’ positions. However,the cost of a Kinect is much cheaper than any existing commercial OMocap,including the MAT-T system. The main difference between the former two kinds of outputs is in depth component. The difference is a bit notable in follow-through stage. Although the DBN model improves the original Kinect outputs,the original hands’ depth outputs are very unstable when severe occlusion and rapid movement occur. These outputs will definitely affect the performance of the model.

The mentioned 5 body parts are chosen in the comparison. The results of comparison are shown in Table 1. The 5 body parts all have mean values of the whole swing procedure.

Compared with the MAT-T system,our model do have certain errors in the outputs. Moreover,some body part’s errors are significant (larger than 10%). These errors do not always mean bad performance. In the whole swing procedure,the skins of shoulders move more severely than any other body segments,because the shoulder joints rotate most rapidly. Markers may change their positions relative to their corresponding body segments due to skin movement^[29]. If the shoulder markers change positions,the outputs of the MAT-T system will fluctuate and this may cause the difference. The shoulder widths (SWs) of five golfers are shown in Fig. 6. The results are gained by our model,the MAT-T system,and original Kinect output.

In normal condition,the shoulder width of each golfer should keep constant during swing,because the body segments are rigid. In Fig. 6,the shoulder width results of the MAT-T system are not constant. The main reason is that the shoulder markers have changed their positions due to severe skin movements. The original Kinect outputs vary more severely due to the occlusion of shoulder joints. With the help of the DBN model,the outputs of our algorithm show more robustness than the other two kinds of results. There are some differences between the outputs of our model and the MAT-T system,but the feasibility of our algorithm is unquestionable. With the development of better capture device (better than Kinect but not as expensive as the MAT-T),there is no doubt that the performance will be better.

2.5 Comparison with other algorithms

In order to show the superiority of our algorithm to the state of art Kinect based joint modification algorithms,the algorithm in Ref.^[22] and the RFR algorithm in Ref.^[26] are chosen. The reasons why these algorithms are chosen as our opponent are two. One is that though the rotation of golfer is limited,the test motions are golf swing and the other is that they provide comparable results with OMocap systems.

As mentioned above,the msJE was calculated. Because the numbers of the selected joints are different (15 in ours and 20 in the other two),a more precise value,the mean value of msJE (mmsJE),is also calculated. The mmsJE is the mean joint error of one joint during the whole swing. These two groups of values are shown in Table 2. The comparison shows that both the three algorithms can improve the Kinect performance but our algorithm performs much better.

By comparing the outputs of our model,the MAT-T system,and two other algorithms,We find that our algorithm works well with high robustness against severe joint occlusion and improves the Kinect performance. The outputs of our model are comparable with the MAT-T system,which illustrates the feasibility and effectiveness of the proposed golf swing reconstruction algorithm. The comparison between our algorithm and two other algorithms shows that the mmsJE of our outputs is much lower. Moreover,our training and testing sets are all real golf swings without any limitation.

3 Conclusion and future work

A golf swing reconstruction algorithm is proposed based on Kinect and a full-body DBN model. The experiments have shown that it can reconstruct the golf swing with good quality. Although there are slight differences compared with the MAT-T system,the accuracy of the proposed algorithm will increase with the development of hardware and rise of sampling rate.

Our future work will be to incorporate the club head into the model and analysis of kinematic parameters generated by the model. Reconstruction and analysis of other regular motion using Kinect with our algorithm is also under consideration.