Advanced Robotics Workshop

Multi-Modal Garment Sorting and Classification Combining Tactile and Visual Sensing

---

Abstract

Automated garment handling in textile recycling remains challenging due to the deformability of textiles, their high shape variability, frequent self occlusion, and the presence of foreign objects in cluttered heaps. This paper presents a Multi- Modal robotic sorting system that combines semantic visual perception with tactile grasp monitoring. The proposed approach integrates Visual Language Model (VLM) based garment classification, Convolutional Neural Network (CNN) based grasp prediction using RGB-D images, and capacitive tactile fingertips mounted on a parallel gripper to detect grasp success, object loss, and approximate weight during manipulation. The estimated weight serves as a plausibility measure for the visually predicted garment class and as a coarse indicator of garment size. To support safe execution, a Digital Twin implemented in MoveIt2 is used for motion planning and collision avoidance in a synchronized real and virtual environment. A classification accuracy of up to 87.89 % across six classes was achieved in an experimental robotic sorting scenario including 219 items. Furthermore, the tactile finger sensor is evaluated under wet conditions and in contact with wet textiles to assess robustness, showing reliable sensing behavior even in these challenging scenarios. Overall, the results demonstrate the potential of combining semantic vision and robust tactile sensing for dependable textile sorting in recycling applications.

How to Cite:

Ergun, S., Mitterer, T. & Zangl, H., (2026) “Multi-Modal Garment Sorting and Classification Combining Tactile and Visual Sensing”, Proceedings of the Austrian Symposium on AI, Robotics, and Vision 3(1), 205-214.