This paper presents a trace-driven experimentation and analytics framework that allows researchers and engineers to devise and evaluate operational strategies for large-scale AI workflow systems. Analytics data from a production-grade AI platform developed at IBM are used to build a comprehensive system and simulation model. Synthetic traces are made available for ad-hoc exploration as well as statistical analysis of experiments to test and examine pipeline scheduling, cluster resource allocation, and similar operational mechanisms.
Consumer-to-consumer (C2C) e-Commerce is a large and growing industry with millions of monthly active users. In this paper, we propose auto content moderation for C2C e-Commerce to moderate items using Machine Learning (ML). We will also discuss practical knowledge gained from our auto content moderation system. The system has been deployed to production at Mercari since late 2017 and has significantly reduced the operation cost in detecting items violating our policies. This system has increased coverage by 554:8 % over a rule-based approach.
SaaS Enterprise workflow companies, such as Salesforce and ServiceNow, facilitate AI adoption by making it easy for customers to train AI models on top of workflow data, once they know the problem they want to solve and how to formulate it. However, as we experience over and over, it is very hard for customers to have this kind of knowledge for their processes, as it requires an awareness of the business and operational side of the process, as well as of what AI could do on each with the specific data available. The challenge we address is how to take customers to that stage, and in this paper we focus on a specific aspect of such challenge: the identification of which "useful inferences" AI could make and which process attributes can (or cannot) be leveraged as predictors, based on the data available for that customer. This is one of the steps we see as central to improve what today is a very limited adoption of AI in enterprises, even in the most "digitized" organization [8,9]. For simplicity in the following we assume that business process data is stored in a DB table, and define the problem semi-formally as follows: Given a table T with a set of fields f ∈ F , and given a point e ∈ E in the process where inference is desired, identify i) the set of fields LF ⊆ F on which AI could make "useful predictions" (potential labels) and, for each such field l ∈ LF , ii) which sets of fields FSl ⊆ P (F) are "usable predictors" (potential featuresets)1. This formulation, besides being simple and fairly general (many problems can be mapped to it) corresponds to the question our customers typically seek answer to.
Our data scientists are adept in using machine learning algorithms and building model out of it, and they are at ease with their local machine to do them. But, when it comes to building the same model from the platform, they find it slightly challenging and need assistance from the platform team. Based on the survey results, the major challenge was platform complexity, but it is hard to deduce actionable items or accurate details to make the system simple. The complexity feedback was very generic, so we decided to break it down into two logical challenges: Education & Training and Simplicity-of-Platform. We have developed a system to find these two challenges in our platform, which we call an Analyzer. In this paper, we explain how it was built and it’s impact on the evolution of our machine learning platform. Our work aims to address these challenges and provide guidelines on how to empower machine learning platform team to know the data scientist’s bottleneck in building model.
This paper presents how VMware addressed the following challenges in operationalizing our ML-based performance diagnostics solution in enterprise hybrid-cloud environments: data governance, model serving and deployment, dealing with system performance drifts, selecting model features, centralized model training pipeline, setting the appropriate alarm threshold, and explainability. We also share the lessons and experiences we learned over the past four years in deploying ML operations at scale for enterprise customers.
