This paper deals with the evaluation of the recommendation functionality inside a connected consumer electronics product in prototype stage. This evaluation is supported by a framework to access and analyze data about product usage and user experience. The strengths of this framework lie in the collection of both objective data (i.e., “What is the user doing with the product?”) and subjective data (i.e., “How is the user experiencing the product?”), which are linked together and analyzed in a combined way. The analysis of objective data provides insights into how the system is actually used in the field. Combined with the subjective data, personal opinions and evaluative judgments on the product quality can be then related to actual user behavior. In order to collect these data in a most natural context, remote data collection allows for extensive user testing within habitual environments. We have applied our framework to the case of an interactive TV recommender system application to illustrate that the user experience of recommender systems can be evaluated in real-life usage scenarios.

The overall aim of this paper is to demonstrate how to obtain reliable information from the user generated product reviews on the Internet. Most of these reviews only contain information about products’ soft failures, which result in the rejection of innovative consumer electronics products that are not due to hardware or software, but due to broken expectations of users. However, the reliability of this essentially subjective information is difficult to evaluate. In order to evaluate the reliability of the user generated product reviews, we explore a method to first classify them using the Disconfirmed Expectations Ontology (DEO) that is developed to analyze soft reliability issues, and then further to explore their information content by means of user tests. We apply this method in a case study using a consumer electronic product from a multi-national company.

Understanding customers-in-context for actual product realization processes (PRPs) has become a pressing need since a large and rapidly increasing share of complaints in the field cannot be attributed to violation of products' technical specifications. While addressing this problem requires a multidisciplinary approach, more studies in the engineering design domain have of late been proposed on engineering contextual and emotional values in product design. However, it is not yet clear how these findings can be utilized within large-scale operational PRPs. Accordingly, in this paper, we propose an operational method empowering the stakeholders in collaborative PRPs with core decision templates, which provide (i) relevant information on customers-in-context, and (ii) corresponding guidelines to improve underlying processes. The content of these templates builds on the results of user feedback analysis with the subjective-feedback ontology from Soft Reliability, and their structure is based on the compromise Decision-Support Problem templates. Partial application of our method is demonstrated through two industrial cases. We envision that our method can help to evaluate and foresee the impact of new technology as it gets incorporated into the specific ecology of values and activities of its users.

Especially in the past few years, there has been an increase in the rejection rate of interactive consumer electronics products in the field, not due to broken hardware or software, but due to 'broken expectations' of users. However, operational methods to capture triggering contextual reasons are not functional in the industry. In addressing this gap, we propose systematic analysis of qualitative user feedback data resources from the field by utilizing our Disconfirmed Expectations Ontology (DEO). DEO provides for an efficient means to elicit relevant -but currently unrecognizable- feedback from the field to communicate that to the respective units in a product development process. We further demonstrate the utilization of DEO on a rich qualitative data set regarding the Apple iPhone".

A recent trend in technological innovation is towards the development of increasingly multifunctional and complex products to be used within rich socio‐cultural contexts such as the high‐end office, the digital home, and professional or personal healthcare. One important consequence of the development of strongly innovative products is a growing market uncertainty regarding ‘if’, ‘how’, and ‘when’ users can and will adopt such products. Often, it is not even clear to what extent these products are understood and interacted with in the intended manner. The mentioned problems have already become an evident concern in the field, where there is a significant rise in the numbers of seemingly sound products being complained about, signaling a lack of soft reliability. In this paper, we position soft reliability as a growing and critical industrial problem, whose solution requires new academic expertise from various disciplines. We illustrate potential root causes for soft reliability problems, such as discrepancy between the perceptions of users and designers. We discuss the necessary approach to effectively capture subjective feedback data from actual users, e.g. when they contact call centers. Furthermore, we present a novel observation and analysis approach that enables insight into actual product usage, and outline opportunities for combining such objective data with the subjective feedback provided by users. Copyright © 2008 John Wiley & Sons, Ltd.

A recent trend in technological innovation is towards the developmentof increasingly multifunctional and complex products to be used within rich socio-cultural contexts such asthe high-endoffice,the digitalhome,andprofessionalor personalhealthcare. One important consequence of the development of strongly innovative products is a growing market uncertainty regarding ‘if’, ‘how’, and ‘when’ users can and will adopt such products. Often, it is not even clear to what extent these products are understood and interacted with in the intended manner. The mentioned problems have already become an evident concern in the field, where there is a significant rise in the numbers of seemingly sound products being complained about, signaling a lack of soft reliability. In this paper, we position soft reliability as a growing and critical industrial problem, whose solution requires new academic expertise from various disciplines. We illustrate potential root causes for soft reliability problems, such as discrepancy between the perceptions of users and designers. We discuss the necessary approach to effectively capture subjective feedback data from actual users, e.g. when they contact call centers. Furthermore, we present a novel observation and analysis approach that enables insight into actual product usage, and outline opportunities for combining such objective data with the subjective feedback provided by users. Copyright © 2008 John Wiley & Sons, Ltd.

Currently, despite the explicit industrial consideration to improve the appeal and usability of technically sound electronics products, users increasingly seem to have dissatisfactory experiences in interacting with them. These unforeseen experiences (attributable to specifications omissions, usability/learnability problems, or specific usage context) lead to a large and increasing share of unknown field complaints. To correct and prevent such complaints or user reports, we promote effective exploitation of call centers: Valuable usage data is retrievable from the field by adopting a user-centered failure classification model that we developed. We also report on the supporting results of a test from applying our model to a set of call center data.

With the emergence of highly interactive products in the domestic space, consumer electronics brands are facing an increasing challenge in predicting the way their products are being used and experienced. Unforeseen experiences relating to functional, emotional and social/contextual aspects of product use lead to a large and increasing share of field complaints that cannot be attributed to a violation of products' specifications. In a project called Soft Reliability, we are trying to develop a product evaluation ecology that enables the anticipation of product use by gathering behavioral and attitudinal data early in the product development process, through longitudinal field studies with working prototypes. This paper introduces a novel framework for behavioral and attitudinal data collection and analysis. The framework enables instrumentation that relies on the event-based experience sampling method, and as such deploys an analysis methodology that includes process mining techniques for the analysis of usage patterns, multivariate techniques for the analysis of longitudinal attitudinal data, and product quality analysis techniques for the analysis of combined information. We illustrate the value and applicability of this framework in practice, through the findings of an ongoing project concerning the conceptualization of an innovative Internet on TV product, which is being conducted in collaboration with Philips, a multi-national consumer electronics company.

In the past, quality and reliability measures of products were mainly for technical component performances. However, today's global market conditions led to the realization that the dominating indicator of product qualityand reliability is often customer satisfaction. This is justified by the industry where significantly rising numbers of products are being returned or being sought redress, which in fact technically function well according to specifications. Continuous influx of new technology, whilst creating huge opportunities fornew product-market combinations, results in high levels ofuncertainty about if/how/when users will adopt a product. In orderto address such uncertainty, the complete end-userview and reasons for unexpected and dissatisfactoryinteractions ofusers with products should be well understood. Accordingly, this study proposes a user-centered failure classification model, to methodically analyze user-centered failure mechanisms. An exploratory analysis offield data provides encouraging preliminary findings about the applicability of this model in reality.

This paper proposes a conceptual framework to distinguish between different classes of reliability problems encountered in strongly innovative products. Next to the conventional (hardware and software) problems, new classes of failures have emerged with a wide range of often strongly related definitions, such as: soft failures, "No Fault Found" failures, "Fault Not Found" failures, "Cause Not Found" failures, nuisance failures. The fact that these new classes of failures do not have precise and orthogonal definitions, leads to difficulties in failure identification and classification. A list of dimensions is proposed to identify and classify failures in an unambiguous manner. Contribution of this research is two-fold: From the academic point of view, it encourages precise reasoning about the emerging failure classes in the general context of reliability problems, as well as forming grounds for consistent use of terminology within the community. From the industrial point of view, it potentially provides more accurate and easier detection of failures, hence facilitating more effective and efficient ways to handle them