Abstract: Objectives In the maritime domain, navigational safety remains a critical issue. Ship trajectory prediction can provide forward-looking decision support for potential collision avoidance. However, prediction results may exhibit significant deviations due to factors such as manual maneuvering variations or equipment anomalies. Therefore, it is necessary to introduce a trajectory anomaly detection mechanism to assist in achieving more reliable collision avoidance decisions. Methods In terms of trajectory data construction, normal trajectories and multiple types of abnormal trajectories were generated, and ship motion features were extracted to build an anomaly detection dataset. For model design, an anomaly detection model based on a dual-attention residual prediction-discrimination autoencoder architecture is proposed. Temporal attention and multi-head self-attention mechanisms are introduced in the encoding stage to enhance temporal feature representation, while a residual bottleneck module is employed to strengthen trajectory deviation features. Meanwhile, a collaborative detection framework integrating reconstruction, prediction, and discrimination branches is constructed to achieve anomaly identification through multi-dimensional information fusion. Results Experimental evaluations are conducted under multiple abnormal scenarios, including turning anomalies, acceleration anomalies, stopping anomalies, and translation anomalies. The model performance is assessed using metrics such as Accuracy, Precision, Recall, F1-score, and AUC. The results demonstrate that the proposed model achieves favorable detection performance across different anomaly types and exhibits advantages in both detection accuracy and stability compared with baseline models. Conclusions The experimental results verify the effectiveness of the proposed anomaly detection method in complex AIS trajectory scenarios. By integrating dual-attention feature modeling, residual feature representation, and a prediction-discrimination joint detection mechanism, the model can more accurately characterize variations in ship navigation behavior and effectively identify anomalous navigation activities, thereby providing technical support for maritime safety monitoring and maritime supervision.