Domain shifts in industrial condition monitoring: a comparative analysis of automated machine learning models

Abstract

Selecting an appropriate model for industrial condition monitoring is challenging due to various factors. Typically, industrial datasets are small and lack statistical independence because experimental coverage of all possible operational variations is costly and sometimes practically impossible. Consequently, the resulting domain shifts pose a significant challenge. Although deep learning (DL) methods have frequently been regarded as the primary and optimal choice in many applications, they often lack major success factors in condition monitoring tasks. In this study, we benchmark the robustness of typical DL architectures against classical feature extraction and selection followed by classification (FESC) methods under domain shifts commonly encountered in industrial condition monitoring. Both DL and FESC methods are employed within an automated machine learning framework. We benchmarked these methods on seven publicly available datasets, and to simulate domain shifts, we employed leave-one-group-out validation on those datasets. Our experiments demonstrate high accuracy across all tested models for random K-fold cross-validation. However, the overall performance significantly decreases when faced with domain shifts, such as transferring the trained model from one machine to another. In four out of seven datasets, FESC methods showed better results in the presence of domain shifts. Furthermore, we also show that FESC methods are easier to interpret than DL methods. Finally, our results suggest that deep neural networks are not universally preferred over classical, low-capacity models for such tasks, as typically only a limited number of features from the input signal are needed.