Enabling Process Mining on Multimodal Robotic Data

Abstract

Robotic systems are increasingly deployed across diverse domains, performing multiple operations while continuously interacting with the environment and making decisions. This results in large volumes of multimodal data from various sources including sensor readings, video feeds, and communication data. In this domain, process mining holds the potential for extracting behavioral patterns, identifying anomalies, and evaluating performance metrics in robotic systems. However, its application remains largely unexplored due to the fine-grained, multimodal nature of robotic data. In this work, we investigate how robotic data should be processed to enable process mining applications. We explore activity recognition techniques to transition from robotic fine-grained data to high-level activities, applying Conditional Random Fields to sensor data and fine-tuning the Florence-2 model for video data. Furthermore, we outline key challenges and opportunities in preparing robotic data for process mining, laying the groundwork for future research on process mining-based analysis of robotic systems.

Download paper here

Cite as: F. Corradini, S. Pettinari, B. Re, L. Rossi, and M. Sampaolo. Enabling Process Mining on Multimodal Robotic Data. Advanced Information Systems Engineering Workshops. CAiSE: 257–-269, 2025

@inproceedings{corradiniPRRS25a,
  title={Enabling Process Mining on Multimodal Robotic Data},
  author={Corradini, Flavio and Pettinari, Sara and Re, Barbara and Rossi, Lorenzo and Sampaolo, Massimiliano},
  booktitle={Advanced Information Systems Engineering Workshops},
  year={2025},
  series={LNBIP},
  volume={556},
  pages={257–269},
  publisher={Springer}
}