Human Motion Prediction for Human-Robot Collaboration

Abstract

This paper compares three human motion prediction methods — GMM-based clustering, TransFusion, and Graph-Mixer — evaluating their effectiveness in human-robot collaboration. These predictions are valuable for improving robot motion planning by anticipating human movements and enhancing safety and efficiency in dynamic environments.

I Introduction

Predicting human motion in real-time is key to safe human-robot collaboration. Traditional approaches, like Gaussian Mixture Models (GMMs)[1], work in controlled scenarios but struggle with complex human behaviour. Deep learning (DL) methods, including RNNs, GCNs, and Transformers, address these limitations by improving prediction accuracy[2, 3], though the real-time application is challenging due to computational demands.

Traditional motion planning methods ensure safety but lack adaptability for dynamic interaction. Proactive approaches anticipate human actions, with methods like temporal PRM enhancing collaboration [4]. Our work compares clustering and deep learning for HMP, aiming to integrate predictions into a human-aware planning framework for improved safety and responsiveness.

II Methodology

We present a comparison of real-time approaches to human motion prediction. One approach is to use clustering techniques to group similar gestures. The human movements are segmented based on variations in the hands’ velocity vectors, which often indicate transitions between gestures in a repetitive industrial setting. Once segmented, gestures are clustered using a Gaussian Mixture Model (GMM) with Dynamic Time Warping (DTW) and Principal Component Analysis (PCA) for time-series comparison.

We need to create clusters that are ($T\times J\times 3$)-dimensional, where $T$ is the number of time samples and $J$ is the number of human joints considered $\times 3$ to consider the 3D cartesian positions. To reduce the dimensionality of the problem, hence increasing real-time performances, we used PCA to preprocess all data, bringing the problem to $T\times N$ dimensions, where $N$ is the number of components we decided to keep after PCA.

Fig. 1: Comparison of predictions using the clustering method and deep learning method for the prediction. The red trails are the ground-truths of the joint trajectories of the human. The green trails are the predictions using the clustering method. The blue trails are the predictions using the deep learning method.

DTW allows us to generalize a path to multiple trajectories with different profiles of speed of execution. This lowers the burden on the clustering model since the same cluster can represent many trajectories that follow the same geometric path.

We tried both random sampling and K-Means clustering to initialize the GMM clusters. The K-Means initialization reached a slightly lower prediction error, so we decided to implement it over the random initialization. After the initialization, an Expectation–maximization algorithm (EM) is used to converge to the final multivariate Gaussians’ parameters. The centroids of the closest clusters are then used for the motion prediction inference in real-time. The clusters are retrieved with their responsibilities and probability values of being the Gaussian that generates the current gesture.

Alongside clustering, we investigate DL approaches. Specifically, we employ a graph-based model (Graph-Mixer), which consists of an initial pose embedding module that processes the input pose sequences into higher-dimensional features using an adaptive spatial graph convolution. This is followed by the spatial-temporal graph-mixer, which incorporates both adaptive spatial and temporal graph convolutional networks to model human skeleton dynamics and temporal relationships across different frames. Finally, the prediction head generates future motion predictions from these spatial-temporal features, allowing for more accurate long-term human motion forecasting. Additionally, we explore TransFusion, a Transformer-based architecture combined with a diffusion process to predict 3D human motion. The model uses long skip connections between shallow and deep layers to enhance information flow and employs squeeze-and-excitation (SE) blocks for token recalibration. Inputs, including historical motion data and diffusion steps, are treated as tokens. The model operates in the frequency domain using a discrete cosine transform (DCT) to process motion data, reducing noise and dimensionality.

We integrate real-time human predictions into the robot’s planning process. The robot continuously adapts its motion based on predicted human movements, ensuring safe and smooth collaboration. This dynamic adjustment allows the robot to anticipate human actions and modify its path accordingly.

III Results and Discussion

This section presents the results and compares the three prediction methods: the Clustering-Based prediction method, the Adaptive Spatial-Temporal Graph-Mixer network, and the TransFusion network. We trained and tested our system using the HA4M dataset [5].

III-A Training of the Models

For the GMM approach, we decided to keep 10 components after the PCA, and create $100$ clusters. These values were chosen after an ablation study. The length of the segmented gestures is variable and handled by the DTW, but corresponds roughly to 2 seconds of human joint trajectories.We decided to use 6 frames of historical motion to predict 60 frames (2 seconds) of future motion for the deep learning (DL) approaches. The Graph-Mixer network was trained for 20 epochs, while the TransFusion network was trained for 500 epochs.

III-B Evaluation of Human Motion Prediction Models

A qualitative evaluation and comparison of the GMM and TransFusion predictions is shown in Figure 1. We evaluated our methods using the Mean Per Joint Position Error (MPJPE), a standard metric in 3D human pose prediction. This metric calculates the average Euclidean distance between the predicted and actual joint positions in 3D space. The MPJPE is calculated by computing the error for each joint, averaging across all joints, and then averaging across all frames in a sequence. Formally:

$$MPJPE=\frac{1}{N}\sum_{i=1}^{N}\lVert\hat{J}_{i}-J_{i}\rVert$$ (1)

where $N$ is the number of joints, $\hat{J}_{i}$ is the predicted 3D position of the i-th joint, $J_{i}$ is the ground truth 3D position of the i-th joint, and $\lVert\cdot\rVert$ denotes the Euclidean distance between the predicted and ground truth joint positions. In Table I we report the MPJPE, variance, the 5th percentile, and the 95th percentile results for the three models.

TABLE I: Method comparison

Method	MPJPE	Variance	5th Percentile	95th Percentile
Clustering	$71.60$	$2.14$	$7.13$	$182.25$
TransFusion	$100.81$	$5853.17$	$20.57$	$251.47$
Graph-Mixer	$111.27$	$10381.51$	$14.44$	$343.57$

Note: All values are in millimeters [mm]

III-C Prediction-aware motion planning

Our preliminary simulations indicate that integrating human motion prediction into robot planning using the t-PRM [4] leads to collision-free trajectories for the manipulator. Such trajectories improve safety and efficiency in performing an industrial task.

IV Conclusions

In this work, we validated and compared three different human motion prediction approaches: the Clustering-Based GMM prediction method, the Adaptive Spatial-Temporal Graph-Mixer network, and the TransFusion network. Our preliminary simulations validated the quality of the predictions of both traditional clustering and modern deep learning techniques, and demonstrated a good capability of avoiding human motions during robot operation.

For future work, we plan to test the framework experimentally, using the ZED2 camera and a Universal Robots UR5. We plan to extend our approach by incorporating probabilistic scenario-based planning techniques [6], to enhance our system’s ability to handle multiple predictions with associated probabilities, since our Clustering-Based prediction method outputs multiple predictions with their corresponding probability. This extension will enable more robust and adaptive motion planning by generating and optimizing trajectories based on probabilistic human motion scenarios.

Our results contribute to advancing human-robot interaction by improving the robot’s proactive collision avoidance capabilities.

V Acknowledgments

Co-funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them (EU - HE Magician – Grant Agreement 101120731).