Sequential Modeling in Marine Navigation

The maritime industry’s continuous commitment to sustainability has led to a dedicated exploration of methods to reduce vessel fuel consumption. This paper undertakes this challenge through a machine learning approach, leveraging a real-world dataset spanning two years of a passenger vessel in west coast Canada. Our focus centers on the creation of a time series forecasting model given the dynamic and static states, actions, and disturbances. This model is designed to predict dynamic states based on the actions provided, subsequently serving as an evaluative tool to assess the proficiency of the vessel’s operation under the captain’s guidance. Additionally, it lays the foundation for future optimization algorithms, providing valuable feedback on decision-making processes.

Preprocessing

In this stage, we tackled outlier management using the 1.5 IQR method, treating them as missing data points and employing various imputation techniques for isolated missing values within trips. We structured the raw data into trips, with a focus on active travel, exploring factors impacting FC, such as engine speed, RPM, draft, trim, cargo, wind, and wave effects.
However, some factors like draft, cargo, and waves required estimation due to missing data, and we devised an approach based on domain knowledge. We also developed a clustering method to assign operating modes (mode 1 for autopilot travel and mode 2 for docking regions) based on factors like speed, acceleration, and engine speed. Wind factors were incorporated through squared relative wind speed and categorized wind direction. Feature selection involved computing the Pearson correlation coefficient and leveraging domain knowledge for decision-making. Feature engineering introduced acceleration and displacement, and we normalized data values to a 0-1 range. Additionally, power transformations were applied to features with skewed distributions, aiming to align them more closely with Gaussian distributions.

Modeling

As shown in Fig. 1, the architectural framework consists of two components: a pretrained transformer model called informer [1], responsible for executing time series forecasting and feature fusion. It is followed by a Gated Recurrent Unit (GRU) module employed to predict the residual, thereby mitigating auto-regressive cumulative errors within the predicted outcomes. The determination of the sequence length is predicated on two crucial considerations. Firstly, it must be of sufficient length to facilitate accurate forecasting. Secondly, it should avoid excessive elongation, as it signifies the time interval to wait in each trip before making new predictions. Following trade-offs, a sequence length of 25, coupled with a prediction horizon of 5, was deemed the optimal configuration.

Environment

We have out-sourced a RL compatible dataset for offline RL setting with a gym environment that can be served as reality-based simulator. We constructed an offline dataset tailored for conventional reinforcement learning (RL) frameworks, comprising observations, next observations, actions, rewards, and termination indicators. The actions encompass heading, engine speed, and mode selections. For observations, we employ both current and next states, encompassing static factors such as hour of the trip, direction, dynamic factors including heading rate, resistance (torque/thrust), displacement, and previous outputs, as well as disturbance-related variables like time, weekday, current, season, weather, wind direction, wind force, and water depth. In terms of rewards, we have defined three distinct intermediate rewards. The first penalizes deviations from the top 1\% of the dataset, the second penalizes fuel consumption (FC), and the last comprises sparse rewards, with +1 awarded for on-time arrival at the docking area and a penalty of -0.1 for each minute beyond the schedule. The model outputs four variables: LAT, LON, SOG, and FC.

In the Gym environment, we implemented curriculum learning by defining three reward stages. The first stage focuses on emulating the decisions made by captains in the dataset. In the second stage, rewards are structured to encourage behavior similar to the top 1% of best-performing trips. The third stage emphasizes the minimization of FC while adhering to the designated time schedule. The reinforcement learning process concludes with the issuance of the “done” signal, triggered when the vessel successfully reaches its intended destination. Conversely, if the vessel fails to reach its destination within 25% more time steps than usual, the process times out, resulting in a negative reward.

Conclusion

As demonstrated in Table 1, we observe that the root mean square error (RMSE) and coefficient of determination ($R^2$) exhibit the best performance for all four quantities when employing the AR + GRU approach. The second-best results are associated with the standard deviation (Std.) of SOG and LAT, while FC and LON still exhibit the best performance across all approaches. In general, the AR approach outperforms the others, primarily due to allowing the transformer to learn and mitigate cumulative errors during training.

We outline a step-wise approach to develop a time-series model for a passenger vessel using real data. Additionally, we introduce an offline dataset and a simulator that both can serve as training tools for captains or an environment for machine learning systems. The forecasting model will be instrumental in assessing the optimization model’s effectiveness. The Gym environment undergoes three stages of training. Initially, it aims to replicate captain driving behaviors, then shifts to imitating top trips with the lowest FC, and finally strives to surpass captain performance by pursuing to minimize FC while adhering to the time schedule and other limitation. This multi-stage training approach is expected to accelerate the convergence of RL systems. For future direction, we consider leveraging RL to optimize the navigational best practice to maximize fuel efficiency.

References:

[1] Zhou, Haoyi, et al. “Informer: Beyond efficient transformer for long sequence time-series forecasting.” Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 12. 2021.

Links:

How to cite:

@inproceedings{Fan2024sequential,
  title={Sequential Modeling in Marine Navigation: Case Study on a Passenger Vessel},
  author={Fan, Yimeng and  Agand, Pedram  and Chen, Mo and Park, Edward J and Kennedy, Allison and Bae, Chanwoo},
  booktitle={Proceedings of the AAAI conference on artificial intelligence},
  volume={},
  number={},
  pages={},
  year={2024}
}