While artificial intelligence (AI) models have improved at understanding large-scale data, understanding AI models themselves at any scale is difficult. For example, even two models that implement the same network architecture may differ in frameworks, datasets, or even domains. Furthermore, attempting to use either model often requires much manual effort to understand it. As software engineering and AI development share many of the same languages and tools, techniques in mining software repositories should enable more scalable insights into AI models and AI development. However, much of the relevant metadata around models are not easily extractable. This paper (an extension of our MSR 2020 paper) presents a library called AIMMX for AI Model Metadata eXtraction from software repositories into enhanced metadata that conforms to a flexible metadata schema. We evaluated AIMMX against 7,998 open-source models from three sources: model zoos, arXiv AI papers, and state-of-the-art AI papers. We also explored how AIMMX can enable studies and tools to advance engineering support for AI development. As preliminary examples, we present an exploratory analysis for data and method reproducibility over the models in the evaluation dataset and a catalog tool for discovering and managing models. We also demonstrate the flexibility of extracted metadata by using the evaluation dataset in an existing natural language processing (NLP) analysis platform to identify trends in the dataset. Overall, we hope AIMMX fosters research towards better AI development.

Background: Assistive automatic seizure detection can empower human annotators to shorten patient monitoring data review times. We present a proof-of-concept for a seizure detection system that is sensitive, automated, patient-specific, and tunable to maximise sensitivity while minimizing human annotation times. The system uses custom data preparation methods, deep learning analytics and electroencephalography (EEG) data. Methods: Scalp EEG data of 365 patients containing 171,745 s ictal and 2,185,864 s interictal samples obtained from clinical monitoring systems were analysed as part of a crowdsourced artificial intelligence (AI) challenge. Participants were tasked to develop an ictal/interictal classifier with high sensitivity and low false alarm rates. We built a challenge platform that prevented participants from downloading or directly accessing the data while allowing crowdsourced model development. Findings: The automatic detection system achieved tunable sensitivities between 75.00% and 91.60% allowing a reduction in the amount of raw EEG data to be reviewed by a human annotator by factors between 142x, and 22x respectively. The algorithm enables instantaneous reviewer-managed optimization of the balance between sensitivity and the amount of raw EEG data to be reviewed. Interpretation: This study demonstrates the utility of deep learning for patient-specific seizure detection in EEG data. Furthermore, deep learning in combination with a human reviewer can provide the basis for an assistive data labelling system lowering the time of manual review while maintaining human expert annotation performance. Funding: IBM employed all IBM Research authors. Temple University employed all Temple University authors. The Icahn School of Medicine at Mount Sinai employed Eren Ahsen. The corresponding authors Stefan Harrer and Gustavo Stolovitzky declare that they had full access to all the data in the study and that they had final responsibility for the decision to submit for publication.

ABSTRACTDespite all of the power that machine learning and artificial intelligence (AI) models bring to applications, much of AI development is currently a fairly ad hoc process. Software engineering and AI development share many of the same languages and tools, but AI development as an engineering practice is still in early stages. Mining software repositories of AI models enables insight into the current state of AI development. However, much of the relevant metadata around models are not easily extractable directly from repositories and require deduction or domain knowledge. This paper presents a library called AIMMX that enables simplified AI Model Metadata eXtraction from software repositories. The extractors have five modules for extracting AI model-specific metadata: model name, associated datasets, references, AI frameworks used, and model domain. We evaluated AIMMX against 7,998 open-source models from three sources: model zoos, arXiv AI papers, and state-of-the-art AI papers. Our platform extracted metadata with 87% precision and 83% recall. As preliminary examples of how AI model metadata extraction enables studies and tools to advance engineering support for AI development, this paper presents an exploratory analysis for data and method reproducibility over the models in the evaluation dataset and a catalog tool for discovering and managing models. Our analysis suggests that while data reproducibility may be relatively poor with 42% of models in our sample citing their datasets, method reproducibility is more common at 72% of models in our sample, particularly state-of-the-art models. Our collected models are searchable in a catalog that uses existing metadata to enable advanced discovery features for efficiently finding models.

Background Assistive automatic seizure detection can empower human annotators to shorten patient monitoring data review times. We present a proof-of-concept for Findings The automatic detection system achieved tunable sensitivities between 75.00% and 91.60% allowing to reduce the amount of raw EEG data to be reviewed by a human annotator by factors between 142x, and 22x respectively. The algorithm enables instantaneous reviewer-managed optimisation of the balance between sensitivity and the amount of raw EEG data to be reviewed. Interpretation This study demonstrates the utility of deep learning for patient-specific seizure detection in EEG data. Furthermore, deep learning in combination with a human reviewer can provide the basis for an assistive data labelling system lowering the time of manual review while maintaining human expert annotation performance. the data in the study that they final for the to for publication.

Runway is a cloud-native tool for managing machine learning experiments and their associated models. The iterative nature of developing models results in a large number of experiments and models that are often managed in an ad hoc manner. Runway is a workflow and framework independent tool that centrally manages and maintains metadata and links to artifacts needed to reproduce models and experiments. Runway provides a web dashboard with multiple levels of visualizations to evaluate performance and enable side-by-side comparisons of models and experiments.

The importance of understanding end user needs and involving them in the software development process is well known in software engineering. Agile Software Development methodologies have incorporated user feedback in different ways. User stories should represent the needs of a user, but often express the views of the Product Owner or the software development team. Several works have investigated integrating User Centered Design into Agile Software Development to satisfy end user needs. This work proposes a different approach focused on satisfying end user needs employing Design Thinking iterative software development. This methodology was applied in five real software development projects which have been analyzed as part of this work.

Multi-party Conversational Systems are systems with natural language interaction between one or more people or systems. From the moment that an utterance is sent to a group, to the moment that it is replied in the group by a member, several activities must be done by the system: utterance understanding, information search, reasoning, among others. In this paper we present the challenges of designing and building multi-party conversational systems, the state of the art, our proposed hybrid architecture using both rules and machine learning and some insights after implementing and evaluating one on the finance domain.

In this work we aim to investigate the way people behave and think about transactions in everyday life. In industrial lab research, it is not always possible to conduct full-scale ethnographic studies to understand people's everyday practices in support of the development of new technologies or to understand the ways in which technology may improve people's life. We apply a less-time consuming design approach to gather preliminary ideas in preparation for more in-depth field studies. We describe four design-inspired studies to gain insights about user thoughts and behaviors while engaged in financial transactions. We also provide important design considerations for developing mobile payment systems using social networks.

Agile Software Development (ASD) has been on mainstream through methodologies such as XP and Scrum enabling them to be applied in the development of complex and reliable software systems. This paper is the end result of the Master's dissertation of the main author, and proposes a solution to guide the development of complex systems based on components by adding exceptional behavior modeling practices to Scrum, resulting in the Scrum+CE method (Scrum with Exceptional Behavior). In order to evaluate the proposed method, a synthetic controlled experiment was conducted with three groups. We compared the efficiency of the new process in relation to plain Scrum and the results were the production of a better quality software but with less features implemented during the same amount of time.

In this paper we describe an user evaluation that aimed to understand how a group of endusers interpret a visual analytics tool in the context of service delivery. It is common for service factories to have an organization devoted to handle incidents. Many incident management systems have strict controls on how fast incidents should be handled, often subjected to penalties when targets are not met.We call Time-Bounded Incident Management (TBIM) those systems, which require clearly defined incident resolution times. In our project, research scientists proposed a method and a visual representation named Workload Profile Chart (WPC) that had as primary goal to understand the area of incident management in a service delivery department. The objective of this visual representation is to help characterizing the performance of TBIM systems and diagnosing major issues such as resource and skill allocation problems, abnormal behavior, and incident characteristics. Researchers wanted to understand if end-users, the quality analysts (QAs), would comprehend the charts and would be able to use them to identify problems and propose effective improvement actions related to TBIM activities. The study was conducted with ten QAs of a service delivery department of a IT company based in Brazil. The data was analyzed using descriptive statistical and qualitative methods. As a result, participants were mainly guided by the axes titles and chart legends to interpret the visualizations, and not always understood what kind of data the chart was displaying. Those results served as insights of how QAs think when analyzing TBIM information in a service delivery department and what improvements in the visual representation tool may be proposed to facilitate their activity. At last we identified evidences of how to design better visual analytics tools based on participant's perceptions and interpretations of color differences and verbal information in chart labels and legend.

The management of IT services is increasingly driven by the necessity of cost reduction and quality improvement. As a result, new organizational models have been created and implemented aiming at achieving economies of scale, while improving customer satisfaction and experience has been one of the main focuses of recent research in Service Science. In this context, translating business elements and necessities of customers into IT components of service operations is a challenging and key task that can affect the long-term relationship between customers and IT provider. Further, market fluctuations and peaks that affect the business of customers of IT outsourcing shall, carefully, be taken into account by support teams in charge of managing customers' infrastructures. Thus, to incorporate it to the dynamics of IT support activities, service delivery teams of IT providers must develop abilities to understand and mechanisms to retain and use the knowledge of the core business of each customer. This paper addresses relevant aspects of service delivery affected by the peaks that impose variations on demands and levels of service expected by customers of IT outsourcing. Also, it proposes some insights to transform customer (business driven) needs into elements already existing inside IT support organizations and argue that this transformation may reduce substantially the overhead of translating customer business to support operations. In order to corroborate the argument and instantiate the discussion, incident data collected by an IT service delivery provider are presented and analyzed graphically in details.

Agile Software Development (ASD) has been on mainstream through methodologies such as Extreme Programming (XP) and Scrum in the last decade enabling them to be applied in the development of computer systems of various size, technical and domain complexity and degress of reliability.This fact highlights the need for software development processes that are accurate and have an adequate amount of modeling and documentation, especially regarding the architectural design, aiming to increase the quality of the end result.The reliability can be achieved by adding elements of exception handling at early stages of development and through components reuse.Exception handling has been a widely used technique in detecting and fixing errors in software systems.The MDCE+ is a method that assists exceptional behavior modeling at components based systems, which is architecture-centric what improves the definition and flow analysis of exceptions between system components.This paper proposes a solution to guide the development of reliable systems based on components by adding MDCE+ practices to Scrum, resulting in the Scrum+CE method (Scrum with Exceptional Behavior).This process exposes the exceptional requirements, at the User Stories level, documents acceptance tests with more details, requires the creation of a high-level architecture artifact and adds a new role of Architecture Owner.In order to evaluate this proposed method, a controlled experiment was conducted with three teams, who developed a system with reliability requirements using Scrum and Scrum+CE.We collected metrics to compare the efficiency of the new process and the result was the production of software with better quality but with less features using Scrum+CE.

This chapter describes and validates the applicability of a visual analytics tool, named Workload Profile Analytics (WPA), in the context of incident management (IM) inside a large IT services provider. The WPA tool implements a method to evaluate the performance and quality of IM systems that uses temporal information from incident records (tickets). The method is based on a graphical representation, called Workload Profile Chart (WPC), which consists of a log-log chart obtained by computing and plotting the spreading of ticket data. Then, using the chart, a visual inspection is done to examine the concentration of tickets over the various areas of interest of the WPC and a diagnosis of potential issues is suggested. To illustrate how powerful WPA can be, two real world scenarios are depicted and used to validate the use of WPA in a large service delivery organization. Finally, some discussions about further improvements on the tool and research directions are done.