Development and evaluation of a computer vision system for assembly bolt pattern traceability and poka-yoke

Decisions made based on manufacturing data can help identify key problem areas in an assembly line and mitigate any defects from progressing through to the next step in the assembly process. But what if the products' "as manufactured" data was inaccurate or did not exist at all? This paper outlines the development and validation of a system for tracking and error-proofing the assembly of bolted joints that require specific torque patterns in an industrial environment. Using a machine vision system, our solution recognizes the type of fastener being used, traces the tool location relative to the mechanical fastener, and records the order in which the fasteners were torqued in. If an error is detected, the system does not allow the user to progress through the assembly process. We leverage open source machine learning algorithms to allow efficient object detection model training. Our solution was validated using a series of tests and evaluated using the STEP method. Results show that the proposed system provides an effective and intuitive human interface for cyber-physical systems and an approach toward the digitalization of human skills needed for product assembly and their utilization in manufacturing.