This paper explores how to develop machine learning regression models that are more explainable and transparent for the end-user. Explainable regression models can be created by rank-ordering the features of the regression model that contribute most to predictive accuracy. In addition, fitting graphs can be generated that show how the addition of each feature in a regression model incrementally improves predictive accuracy. These information graphics are especially useful in understanding the tradeoffs involved in selecting a model that considers both model complexity and model performance. These methods are illustrated with two examples: a multiple regression model using a straightforward application of machine learning regression; and a more complex polynomial regression model that captures higher-order terms and interactions among all variables in the model.

Abstract In recent years, there have been several calls by practitioners of machine learning to provide more guidelines on how to use its methods and techniques. For example, the current literature on resampling methods is confusing and sometimes contradictory; worse, there are sometimes no practical guidelines offered at all. To address this shortcoming, a simulation study was conducted that evaluated ridge regression models fitted on five real-world datasets. The study compared the performance of four resampling methods, namely, Monte Carlo resampling, bootstrap, k-fold cross-validation, and repeated k-fold cross-validation. The goal was to find the best-fitting λ (regularization) parameter that would minimize mean squared error, by using nine variations of these resampling methods. For each of the nine resampling variations, 1,000 runs were performed to see how often a good fit, average fit, and poor fit λ value would be chosen. The resampling method that chose good fit values the greatest number of times was deemed the best method. Based on the results of the investigation, three general recommendations are made: (1) repeated k-fold cross-validation is the best method to select as a general-purpose resampling method; (2) k = 10 folds is a good choice in k-fold cross-validation; (3) Monte Carlo and bootstrap are underperformers, so they are not recommended as general-purpose resampling methods. At the same time, no resampling method was found to be uniformly better than the others.

This article investigates resampling methods used to evaluate the performance of machine learning classification algorithms. It compares four key resampling methods: 1) Monte Carlo resampling, 2) the Bootstrap Method, 3) k-fold Cross Validation, and 4) Repeated k-fold Cross Validation. Two classification algorithms, Support Vector Machines and Random Forests, applied to three datasets, are used in this article. Nine variations of the four resampling methods are used to tune parameters on the two classification algorithms on each of the three datasets. Performance is defined by how well the resampling method chooses a parameter value that fits the data well. A main finding is that Repeated k-fold Cross Validation, overall, outperforms the other resampling methods in selecting the best-fit parameter value across the three different datasets.

This paper will look at what types of information graphics and visualizations can support supervised Machine Learning tasks: in essence, how to support the problem of model validation and model overfitting. In particular, I look, graphically, at model performance as a function of model complexity. With an appropriate information graphic, we can visualize at what point the model becomes too complex and starts to deteriorate in performance because of model overfitting. I will look at two actual case studies-the first, a regression task using polynomial regression and the second, a classification problem using neural networks. I create information graphics, in particular fitting graphs, to support the end-user in visualizing which model is the best choice.

We investigated the user interface features of seven different types of crowdsourcing applications, which were classified along three dimensions of task type: (1) Task Structure: does the task have a well-defined solution? (2) Task Interdependence: can the task be solved by an individual or does it require a community of problem solvers? (3) Task Commitment: what level of resources is required to perform the task? Our initial investigation revealed a number of differences in the seven categories including: site searchability, online credentialing, community building features, gamification, mobility, and the use of wiki software.

Although a great many different crowdsourcing approaches are available to those seeking to accomplish individual or organizational tasks, little research attention has yet been given to characterizing how those approaches might be based on task characteristics. To that end, we conducted an extensive review of the crowdsourcing landscape, including a look at what types of taxonomies are currently available. Our review found that no taxonomy explored the multidimensional nature of task complexity. This paper develops a taxonomy whose specific intent is the classification of approaches in terms of the types of tasks for which they are best suited. To develop this task-based taxonomy, we followed an iterative approach that considered over 100 well-known examples of crowdsourcing. The taxonomy considers three dimensions of task complexity: (a) task structure – is the task well-defined, or does it require a more open-ended solution; (2) task interdependence – can the task be solved by an individual, or does it require a community of problem solvers; and (3) task commitment – what level of commitment is expected from crowd members? Based on this taxonomy, we identify seven categories of crowdsourcing and discuss prototypical examples of each approach. Furnished with such an understanding, one should be able to determine which crowdsourcing approach is most suitable for a particular task situation.

We investigate characteristics of the technology platform for different types of crowdsourcing initatives, as characterized by their task type—specifically we classify crowdsourcing applications by task structure, task interdependence, and task commitment. The method employed is to examine best practices of well-known crowdsourcing applications, investigating their user interface features, and characteristics that make them successful examples of crowdsourcing. Among the best practices uncovered were the following: easy searching for information; adaptive user interfaces that learned from the crowd; easy-to-use mobile interfaces; the ability to vote ideas up or down; credentialing; and creating sticky user interfaces that engaged the user. Finally, we consider issues for further study and investigation.

I describe a Venn diagramming technique used to perform syllogistic reasoning on categorical statements. The notation uses overlapping circles to represent relationships among two or three sets, shadings to represent emptiness, and x sequences to represent nonemptiness. These notations allow one to easily visualize logic problems. I then discuss rules of manipulation that can be used to transform one Venn diagram into another valid Venn diagram. These rules provide us with a formal procedure for performing syllogistic reasoning that is to say, they provide us with an algorithm for proving or disproving the validity of a syllogism. I extend the Venn diagramming algorithm for syllogistic reasoning to allow for more than three sets of information at a time. This technique makes use of tables, which is also very intuitive and highly visual. The tabular technique described is capable of processing a much larger variety of logic statements. (C) 2013 Wiley Periodicals, Inc.

I report on my findings on diagrammatic user interfaces. First, I discuss the essence of diagramming, and present a framework for classifying a great variety of diagramming notations commonplace today. Despite a tremendous variety of usage, in terms of notations and applications, I classify diagrams into six themes: (1) topology, (2) sequence and flow, (3) hierarchy: classification, (4) hierarchy: composition, (5) association, and (6) causality. The framework is largely drawn from my recent book, "Diagrammatic Reasoning in AI" (Wiley, 2010), on which much of this discussion is based.Second, I present a framework for diagrammatic user interfaces, in which a diagram is intended to be used in a more dynamic way, and hence, can serve as a graphical user interface. I classify such diagrammatic user interfaces along two dimensions: 1) whether the structure is static or dynamic (i.e., can be actively constructed by the end-user) and 2) whether information on the diagram is static or dynamic (i.e., allows for information propagation). While most of the diagrams in use today are static representations, I explore some of these more dynamic and interesting uses, thereby showing how a diagram can transform a rigid black box user interface into one that is transparent, flexible, and easy-to-use.

Pioneering work shows how using Diagrams facilitates the design of better AI systems The publication of Diagrammatic Reasoning in AI marks an important milestone for anyone seeking to design graphical user interfaces to support decision-making and problem-solving tasks. The author expertly demonstrates how diagrammatic representations can simplify our interaction with increasingly complex information technologies and computer-based information systems. In particular, the book emphasizes how diagrammatic user interfaces can help us better understand and visualize artificial intelligence (AI) systems. It examines how diagrammatic reasoning enhances various AI programming strategies used to emulate human thinking and problem-solving, including: Expert systems Model-based reasoning Inexact reasoning such as certainty factors and Bayesian networks Logic reasoning A key part of the book is its extensive development of applications and graphical illustrations, drawing on such fields as the physical sciences, macroeconomics, finance, business logistics management, and medicine. Despite such tremendous diversity of usage, in terms of applications and diagramming notations, the book classifies and organizes diagrams around six major themes: system topology; sequence and flow; hierarchy and classification; association; cause and effect; and logic reasoning. Readers will benefit from the author's discussion of how diagrams can be more than just a static picture or representation and how diagrams can be a central part of an intelligent user interface, meant to be manipulated and modified, and in some cases, utilized to infer solutions to difficult problems. This book is ideal for many different types of readers: practitioners and researchers in AI and human-computer interaction; business and computing professionals; graphic designers and designers of graphical user interfaces; and just about anyone interested in understanding the power of diagrams. By discovering the many different types of diagrams and their applications in AI, all readers will gain a deeper appreciation of diagrammatic reasoning.

We investigated the risk factors of outsourced software development. Our first objective was to create empirically generated lists of risk factors for both domestically- and offshore-outsourced projects. Our second objective was to compare these two contexts: how do the risk factors change and which ones are most important in each. To address these objectives, we conducted two Delphi surveys to identify the important risk factors from a client perspective, in domestic and offshore settings. We qualitatively compared the results of the surveys to identify similarities and differences across their risk profiles. We identified three types of risks: those that appeared in both contexts; those that appeared in both but were exacerbated in the offshore context; and those that were unique to the offshore context. Our findings suggested that traditional project management risks were important in both contexts; however, the offshore context seemed to be more vulnerable to some traditional risks as well as factors that were unique to it.

This chapter contains sections titled: System Topology Sequence and Flow Hierarchy and Classification Association Causality Decision Diagrams Discussion Endnotes

Even the best project management skills will not guarantee success in the complex world of offshore outsourcing.

This research empirically investigates the design choices that can be made to facilitate problem solving when intelligent systems fail. One way is to provide deep explanations, which are explanations that justify system actions. Another way is to manipulate system restrictiveness of the user interface. An experiment was conducted to investigate the effectiveness of deep explanation support, as well as manipulations of system restrictiveness. Results suggest that the less restrictive system was more effective for problem‐solving situations where system failure occurred. In addition, deep explanations were found to be somewhat helpful in system understanding, and this, in turn, led to improved problem‐solving performance.

This chapter provides a framework for understanding the explanatory power of intelligent systems. It looks at content-based enhancements, drawn primarily from the expert systems literature, interface-based enhancements, and the appropriate selection of an advisory strategy. Such enhancements contribute to explanatory power by increasing system transparency and flexibility, and lead to outcomes such as better decision-making and problem-solving performance. Three illustrative examples demonstrate each type of enhancement of explanatory power. In the first case, a graphical hierarchy of an expert-system knowledge base is illustrated. In the second case, the use of restrictive vs. nonrestrictive advisory strategies is discussed. Finally, deep explanations that provide a deeper understanding of a domain of expertise are described.

This article describes an intelligent system called LogNet that provides advice on how to design business logistics networks. It implements its capabilities by utilizing model-based reasoning techniques. In addition, it utilizes heuristic-based searching to guide an end-user toward more effective network designs. The network design problem studied in this article addresses the warehouse facility location problem: how many warehouses are needed in a network and to which customer markets should they be assigned? LogNet enables end-users to incrementally create and test a logistics network design by using a graphical user interface. LogNet employs model-based reasoning procedures that analyze the structure of the current network design in order to offer recommendations on how to consolidate or decentralize a network. The overall goal of LogNet is to provide a flexible user interface that is capable of supporting this design task.

This paper presents a framework to plan organizational incentives for aligning usage behavior with organizational objectives across various IT adoption stages. Our framework is motivated by the introduction of incentive alignment as another dimension in Information Systems design (Ba et al., 2001) and is based on technology diffusion models presented in the literature. We focus in particular on the usage of technology that is discretionary in nature. In addition, the user group is limited to internal users of the technology at the operational level. The framework integrates key issues on motivation and incentives in order to understand how organizations can induce desired user behaviors congruent with the goals of the organization. We conclude with the use of an illustrative example to show how the framework can be used in an academic environment. This framework will help researchers and practitioners understand how to better manage and align the incentive structures for internal users across IT adoption stages.