Novel Human Machine Interaction with Sticky Notes for Industrial Production

In this paper we present a 3D documentation system which utilizes new human machine interaction concepts on the example of virtual sticky notes. Using different tracking techniques the virtual notes can be attached to a physical object and are displayed on a tablet in an Augmented Reality way. The main intention is to strengthen the interplay between construction and production of industrial machines as the virtual notes are synchronized with a production lifecycle management system. I. PROBLEM DESCRIPTION An essential part of machine manufacturing is the interplay between construction and production. Often this connection leaks information in both ways: the construction team changes details in the last minute, while in production things are mounted in a different order or way as it was intended. Since in the end both sides have to synchronize their knowledge this results mainly in a mass of notes often stuck on the machine or even worse written on the machine itself. This industrial spotlight paper presents a novel development integrating latest technologies to manage position based notes digitally in an Augmented Reality based way. II. STATE-OF-THE-ART Although in media one can see photos of tablets showing Augmented Reality (AR) overlays of shop floors, industrial grade AR documentation systems are rare. To our knowledge the most similar system available to our proposition is Docufy [1]. It is an AR interface to a dedicated content management system and used to display technical content like manuals in a read-only way. The mapping of the data to 3D is managed via a manual registration step of interest points originating from a priori known CAD data. When the tablet is moving, the interest points are tracked and the 3D data adapts to the new viewpoint. III. PROPOSED SYSTEM In this paper we propose a system which enables a user to view, edit and add virtual sticky notes to a machine during assembling by using Augmented Reality techniques. The notes are position dependent and synchronized with the 3D database of a production lifecycle management (PLM) system. Since the construction team mainly works with the PLM system, they immediately have access to the information and may modify existing or add new notes directly. 1PROFACTOR GmbH, 4407 Steyr-Gleink, Im Stadtgut A2, Austria {Forename.Surname}@profactor.at A. Tracking System For the tracking we pursue a multi-modality strategy, which utilizes a combination of • a commercially available infrastructure-based tracking system, • fiducial Augmented Reality markers, • and a visual real-time tracking system. The infrastructure-based tracking system is the main source of 6 degree-of-freedom (DOF) data for the system. See Figure 1 for a tablet enhanced with a tracking system receiver. The two major extensions to the state-of-the-art Fig. 1. Tablet with attached tracking system receiver on the left side. This enables a 6 degree-of-freedom positioning in space. concern the tracking part as well as the way a user can interact with the system. The initial registration is done with an AR marker system based on the work of Garrido-Jurado et al. [2]. Fiducial markers are preferred to feature based approaches as the presence of stable features on industrial objects cannot be guaranteed. After that the tracking data is transformed to the coordinate system of the machine’s CAD model which is provided by the PLM software. This allows the attachment of notes to positions in the CAD data. A special approach is required for movable parts of the machine, like panels. When mounted to the machine, their relative positions to the base CAD data can be determined using attached AR markers. An additional feature was built in to handle unmounted sub-assemblies. They can be annotated like any other machine parts, however they have to be pre-identified in a manual step. This is the input of a real-time tracker which is an extension of Akkaladevi et al. [3]. When a user likes to add a new note to the machine he/she has two possibilities to define the position: either the desired position is touched with the measuring tip mounted on the Proceedings of the OAGM&ARW Joint Workshop 2017 DOI: 10.3217/978-3-85125-524-9-21