Classical scheduling abstractions such as deadlines and priorities do not readily capture the complex timing semantics found in many real-time cyber-physical systems. Time utility functions provide a necessarily richer description of timing semantics, but designing utility-aware scheduling policies using them is an open research problem. In particular, optimal utility accrual scheduling design is needed for real-time cyber-physical domains. In this paper we design optimal utility accrual scheduling policies for cyber-physical systems with periodic, non-preemptable tasks that run with stochastic duration. These policies are derived by solving a Markov Decision Process formulation of the scheduling problem. We use this formulation to demonstrate that our technique improves on existing heuristic utility accrual scheduling policies. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 Optimal Time Utility Based Scheduling Policy Design for Cyber-Physical Systems Terry Tidwell, Robert Glaubius, Christopher D. Gill and William D. Smart Department of Computer Science and Engineering Washington University in St. Louis Email: {ttidwell,rlg1,cdgill,wds}@cse.wustl.edu Abstract—Classical scheduling abstractions such as deadlines and priorities do not readily capture the complex timing semantics found in many real-time cyber-physical systems. Time utility functions provide a necessarily richer description of timing semantics, but designing utility-aware scheduling policies using them is an open research problem. In particular, optimal utility accrual scheduling design is needed for real-time cyber-physical domains. In this paper we design optimal utility accrual scheduling policies for cyber-physical systems with periodic, non-preemptable tasks that run with stochastic duration. These policies are derived by solving a Markov Decision Process formulation of the scheduling problem. We use this formulation to demonstrate that our technique improves on existing heuristic utility accrual scheduling policies.Classical scheduling abstractions such as deadlines and priorities do not readily capture the complex timing semantics found in many real-time cyber-physical systems. Time utility functions provide a necessarily richer description of timing semantics, but designing utility-aware scheduling policies using them is an open research problem. In particular, optimal utility accrual scheduling design is needed for real-time cyber-physical domains. In this paper we design optimal utility accrual scheduling policies for cyber-physical systems with periodic, non-preemptable tasks that run with stochastic duration. These policies are derived by solving a Markov Decision Process formulation of the scheduling problem. We use this formulation to demonstrate that our technique improves on existing heuristic utility accrual scheduling policies.

Reinforcement learning (RL) has shown itself to be an effective paradigm for solving optimal control problems with a finite number of states. Generalizing RL techniques to problems with a continuous state space has proven a difficult task. We present an approach to modeling the RL value function using a manifold representation. By explicitly modeling the topology of the value function domain, traditional problems with discontinuities and resolution can be addressed without resorting to complex function approximators. We describe how manifold techniques can be applied to value-function approximation, and present methods for constructing manifold representations in both batch and online settings. We present empirical results demonstrating the effectiveness of our approach.

We describe a complete system for quantitatively measuring the optical distortion in aircraft windshields and automatically classifying that distortion as acceptable or not. The system comprises two parts: The first uses digital imaging of a known grid pattern through the windshield of interest to create a distortion map of that windshield; the second uses the distortion maps created for 100 windshields (some acceptable and some not) and automatically learns a decision-tree-based classifier. We show results that highlight the robustness of this system, including a demonstration that the distortion map for a windshield is consistently measured over 3 years, and cross-validation study shows that we can effectively classify windshields based on the distortion maps.

Classical scheduling abstractions such as deadlines and priorities do not readily capture the complex timing semantics found in many real-time cyber-physical systems. Time utility functions provide a necessarily richer description of timing semantics, but designing utility-aware scheduling policies using them is an open research problem. In particular, optimal utility accrual scheduling design is needed for real-time cyber-physical domains. In this paper we design optimal utility accrual scheduling policies for cyber-physical systems with periodic, non-preemptable tasks that run with stochastic duration. These policies are derived by solving a Markov Decision Process formulation of the scheduling problem. We use this formulation to demonstrate that our technique improves on existing heuristic utility accrual scheduling policies. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 Optimal Time Utility Based Scheduling Policy Design for Cyber-Physical Systems Terry Tidwell, Robert Glaubius, Christopher D. Gill and William D. Smart Department of Computer Science and Engineering Washington University in St. Louis Email: {ttidwell,rlg1,cdgill,wds}@cse.wustl.edu Abstract—Classical scheduling abstractions such as deadlines and priorities do not readily capture the complex timing semantics found in many real-time cyber-physical systems. Time utility functions provide a necessarily richer description of timing semantics, but designing utility-aware scheduling policies using them is an open research problem. In particular, optimal utility accrual scheduling design is needed for real-time cyber-physical domains. In this paper we design optimal utility accrual scheduling policies for cyber-physical systems with periodic, non-preemptable tasks that run with stochastic duration. These policies are derived by solving a Markov Decision Process formulation of the scheduling problem. We use this formulation to demonstrate that our technique improves on existing heuristic utility accrual scheduling policies.Classical scheduling abstractions such as deadlines and priorities do not readily capture the complex timing semantics found in many real-time cyber-physical systems. Time utility functions provide a necessarily richer description of timing semantics, but designing utility-aware scheduling policies using them is an open research problem. In particular, optimal utility accrual scheduling design is needed for real-time cyber-physical domains. In this paper we design optimal utility accrual scheduling policies for cyber-physical systems with periodic, non-preemptable tasks that run with stochastic duration. These policies are derived by solving a Markov Decision Process formulation of the scheduling problem. We use this formulation to demonstrate that our technique improves on existing heuristic utility accrual scheduling policies.

Classical scheduling abstractions such as deadlines and priorities do not readily capture the complex timing semantics found in many real-time cyber-physical systems. Time utility functions provide a necessarily richer description of timing semantics, but designing utility-aware scheduling policies using them is an open research problem. In particular, scheduling design that optimizes expected utility accrual is needed for real-time cyber-physical domains. In this paper we design scheduling policies that optimize expected utility accrual for cyber-physical systems with periodic, non-preemptable tasks that run with stochastic duration. These policies are derived by solving a Markov Decision Process formulation of the scheduling problem. We use this formulation to demonstrate that our technique improves on existing heuristic utility accrual scheduling policies.

Open soft real-time systems, such as mobile robots, must respond adaptively to varying operating conditions, while balancing the need to perform multiple mission specific tasks against the requirement that those tasks complete in a timely manner.Setting and enforcing a utilization target for shared resources is a key mechanism for achieving this behavior.However, because of the uncertainty and non-preempt ability of some tasks, key assumptions of classical scheduling approaches do not hold. In previous work we presented foundational methods for generating task scheduling policies to enforce proportional resource utilization for open soft real-time systems with these properties. However, these methods scale exponentially in the number of tasks, limiting their practical applicability.In this paper, we present a novel parameterized scheduling policy that scales our technique to a much wider range of systems. These policies can represent geometric features of the scheduling policies produced by our earlier methods, but only require a number of parameters that is quadratic in the number of tasks. We provide empirical evidence that the best of these policies are competitive with exact solution methods in small problems, and significantly outperform heuristic methods in larger ones.

Cyber-physical systems, such as mobile robots, must respond adaptively to dynamic operating conditions. Effective operation of these systems requires that sensing and actuation tasks are performed in a timely manner. Additionally, execution of mission specific tasks such as imaging a room must be balanced against the need to perform more general tasks such as obstacle avoidance. This problem has been addressed by maintaining relative utilization of shared resources among tasks near a user-specified target level. Producing optimal scheduling strategies requires complete prior knowledge of task behavior, which is unlikely to be available in practice. Instead, suitable scheduling strategies must be learned online through interaction with the system. We consider the sample complexity of reinforcement learning in this domain, and demonstrate that while the problem state space is countably infinite, we may leverage the problem's structure to guarantee efficient learning.

Scheduling the execution of multiple concurrent tasks on shared resources such as CPUs and network links is essential to ensuring the reliable operation of many autonomic systems. Well-known techniques such as rate-monotonic scheduling can offer rigorous timing and preemption guarantees, but only under assumptions (i.e. a fixed set of tasks with well-known execution times and invocation rates) that do not hold in many autonomic systems. New hierarchical scheduling techniques are better suited to enforce the more flexible execution constraints and enforcement mechanisms that are required for autonomic systems, but a rigorous and efficient foundation for verifying and enforcing concurrency and timing guarantees is still needed for these approaches. This paper summarises our previous work on addressing these challenges, on Markov decision process-based scheduling policy design and on wrapping repeated structure of the scheduling state spaces involved into a more efficient model, and presents a new algorithm called expanding state policy iteration (ESPI), that allows us to compute the optimal policy for a wrapped state model.

Open soft real-time systems, such as mobile robots, experience unpredictable interactions with their environments and yet must respond both adaptively and with reasonable temporal predictability. Because of the uncertainty inherent in such interactions, many of the assumptions of the real-time scheduling techniques traditionally used to ensure predictable timing of system actions do not hold in those environments. In previous work we have developed novel techniques for scheduling policy design where up-front knowledge of execution time distributions can be used to produce both compact representations of resource utilization state spaces and efficient optimal scheduling policies over those state spaces. This paper makes two main contributions beyond our previous work, to the state of the art in scheduling open soft real-time systems: (1) it shows how to relax the assumption that the entire distribution of execution times is known up front, to allow online learning of an execution time distribution during system run-time; and (2) it shows how to relax the assumption that the execution time of a system action can be characterized by a single distribution, to accommodate different execution time distributions for an action being taken in one of multiple modes. Each of these contributions allows a wider range of system actions to be scheduled adaptively and with Notes: On-line version of paper submitted to RTSS 2009, with full proof in Appendix A. Type of Report: Other Department of Computer Science & Engineering Washington University in St. Louis Campus Box 1045 St. Louis, MO 63130 ph: (314) 935-6160 Scheduling Design with Unknown Execution Time Distributions or Modes Robert Glaubius, Terry Tidwell, Christopher Gill, and William D. Smart {rlg1,ttidwell, cdgill, wds}@cse.wustl.edu Department of Computer Science and Engineering Washington University, St. Louis Abstract— Open soft real-time systems, such as mobile robots, experience unpredictable interactions with their environments and yet must respond both adaptively and with reasonable temporal predictability. Because of the uncertainty inherent in such interactions, many of the assumptions of the real-time scheduling techniques traditionally used to ensure predictable timing of system actions do not hold in those environments. In previous work we have developed novel techniques for scheduling policy design where up-front knowledge of execution time distributions can be used to produce both compact representations of resource utilization state spaces and efficient optimal scheduling policies over those state spaces. This paper makes two main contributions beyond our previous work, to the state of the art in scheduling open soft realtime systems: (1) it shows how to relax the assumption that the entire distribution of execution times is known up front, to allow online learning of an execution time distribution during system run-time; and (2) it shows how to relax the assumption that the execution time of a system action can be characterized by a single distribution, to accommodate different execution time distributions for an action being taken in one of multiple modes. Each of these contributions allows a wider range of system actions to be scheduled adaptively and with temporal predictability, which increases the applicability of our approach to even more general classes of open soft real-time systems. Open soft real-time systems, such as mobile robots, experience unpredictable interactions with their environments and yet must respond both adaptively and with reasonable temporal predictability. Because of the uncertainty inherent in such interactions, many of the assumptions of the real-time scheduling techniques traditionally used to ensure predictable timing of system actions do not hold in those environments. In previous work we have developed novel techniques for scheduling policy design where up-front knowledge of execution time distributions can be used to produce both compact representations of resource utilization state spaces and efficient optimal scheduling policies over those state spaces. This paper makes two main contributions beyond our previous work, to the state of the art in scheduling open soft realtime systems: (1) it shows how to relax the assumption that the entire distribution of execution times is known up front, to allow online learning of an execution time distribution during system run-time; and (2) it shows how to relax the assumption that the execution time of a system action can be characterized by a single distribution, to accommodate different execution time distributions for an action being taken in one of multiple modes. Each of these contributions allows a wider range of system actions to be scheduled adaptively and with temporal predictability, which increases the applicability of our approach to even more general classes of open soft real-time systems.

Scheduling policies for open soft real-time systems must be able to balance the competing concerns of meeting their objectives under exceptional conditions while achieving good performance in the average case. Balancing these concerns requires modeling strategies that represent the range of possible task behaviors, and solution techniques that are capable of effectively managing uncertainty in order to discover scheduling policies that are effective across the range of system modes. We develop methods for solving a particular class of task scheduling problems in an open soft real-time setting involving repeating, non-preemptable tasks that contend for a single shared resource. We enforce timeliness by optimizing performance with respect to the proportional progress of tasks in the system. We model this scheduling problem as an infinite-state Markov decision process, and provide guarantees regarding the existence of optimal solutions to this problem. We derive several methods for approximating optimal scheduling policies and provide theoretical justification and empirical evidence that these solutions are good approximations to the optimal solution. We consider cases in which task models are known, and adapt reinforcement learning methods to learn task models when they are not available.

Open soft real-time systems, such as mobile robots, experience unpredictable interactions with their environments and yet must respond both adaptively and with reasonable temporal predictability. New scheduling approaches are needed to address the demands of such systems, in which many of the assumptions made by traditional real-time scheduling theory do not hold. In previous work we established foundations for a scheduling policy design and verification approach for open soft real-time systems, that can use different decision models, e.g., a Markov decision process (MDP), to capture the nuances of their scheduling semantics.However, several important refinements to the preliminary techniques developed in that work are needed to make the approach applicable in practice. This paper makes three main contributions to the state of the art in scheduling open soft real-time systems: (1) it defines a novel representation of the scheduling state space that is both more compact and more expressive than the model defined in our previous work; (2) it exploits regular structure of that representation to allow efficient verification of properties involving both discrete and continuous system state variables under specific scheduling policies; and (3) it removes the unnecessary use of a time horizon in our previous approach, thus allowing the more precise specification and enforcement of a wider range of scheduling policies for open soft real-time systems.

Scheduling the execution of multiple concurrent tasks on shared resources such as CPUs and network links is essential to ensuring the reliable operation of many autonomic systems. Well known techniques such as rate-monotonic scheduling can offer rigorous timing and preemption guarantees, but only under assumptions (i.e., a fixed set of tasks with well-known execution times and invocation rates) that do not hold in many autonomic systems. New hierarchical scheduling techniques are better suited to enforce the more flexible execution constraints and enforcement mechanisms that are required for autonomic systems, but a rigorous foundation for verifying and enforcing concurrency and timing guarantees is still needed for these approaches. The primary contributions of this paper are: (1) a scheduling policy design technique that can use different decision models across a wide range of systems models, and an example of how a specific (Markov Decision Process) decision model can be applied to a basic multi-threaded system model; (2) novel model checking techniques that can evaluate the behavior of the system model when it is placed under the control of the resulting scheduling policy; and (3) an evaluation of those scheduling policy design and model checking techniques for a simple but representative example of the kinds of execution scenarios that can arise in autonomic systems.

Reinforcement Learning (RL) has proven to be a useful set of techniques for planning under un- certainty in robot systems. Effective RL algorithms for this domain need to be able to deal with large, continuous state spaces, and must make efficient use of experience. In this paper, we present methods to better leverage observed experience by reusing experience across parts of the problem state space that are known to be similar. We present experimental results in a navigational, goal-based domain. We develop an approach to identifying portions of the world that appear similar based on observed transition samples.

Introduction Reinforcement learning (RL) has been shown to be an effective paradigm for learning control policies for problems with discrete state spaces. For problems with continuous multi-dimensional state spaces, the results are less compelling. When these state spaces can be effectively discretized, traditional techniques can be applied. However, many interesting problems must be discretized into an infeasibly large number of states. In these cases, other techniques must be used. Value-function approximation (VFA) addresses some of the problems of traditional RL algorithms, including that of dealing with continuous state spaces. Although it has been successful in a number of specific applications, it has been shown to fail in the general case, and has been known to fail even on simple problems (Boyan & Moore 1995). We propose a novel approach to value-function approximation based on ideas from topology. We identify a key failing of current techniques, and show how our approach avoids this problem by constructing an explicit model of the state space topology. We begin with a brief description of the problems of current valuefunction approximation techniques. We then motivate our proposed approach, outline its mathematical foundations, and provide results from initial experiments.

Constraint satisfaction problems (CSPs) frequently emerge in practical situations. Unfortunately these problems are likely to be computationally intractable. In practice, it is necessary to implement strategies that reduce the amount of computational effort expended in finding solutions to CSPs.

We discuss the modeling and reformulation of a resource allocationproblem, the assignment of Graduate Teaching Assistants tocourses in the University of Nebraska-Lincoln Computer Science Department.

We explore fundamental issues of the modeling, implementation, and processing of non-binary constraints. General techniques for reformulating non-binary constraints (i.e., hidden variable, dual graph [1]) are not practical for high-arity constraints [4]. In our study, we motivate our need to express practical requirements as non-binary constraints, then we explore reformulation methods to deal with them since the conventional methods [1] become impractical. Our work builds on the work of Gent et al. [2], while we motivate and anchor our investigations in the practical context of a real-world application. This is the assignment of graduate teaching assistants to courses in our department. This task is a critical responsibility that our department’s administration has to drudge through every semester. The idea for this particular application is borrowed from Rina Dechter, at the UC Irvine. We model this application using 4 types of unary constraints, one type of binary constraint, and 3 types of non-binary constraints. Since in our application problems are over-constrained, a satisfactory assignment is one that maximizes the number of courses covered. For this purpose, we adopt a new consistency checking mechanism that allows variables to be assigned a null value during search. For two assignments that cover the same number of courses, we further discriminate between them by choosing the one of highest quality, obtained by a combination of the value of the preferences in each assignment. We experiment with two different criteria to maximize preferences. We establish that two of our non-binary constraints are decomposable into equivalent networks of binary constraints, which significantly improves the performance of search [3]. Our efforts have resulted in a prototype system under field-test since August 2001. This system has effectively reduced the number of conflicts, thus yielding a commensurate increase in course quality. It has also decreased the amount of time and effort spent on making the assignment and gained the approval and satisfaction of our staff, faculty and student body.

In recent years, scheduling problems have been advantageously modeled as Constraint Satisfaction Problems (CSPs) and effectively solved by constraint propagation and processing techniques. In this document, we discuss one such application: the assignment of graduate teaching assistants (GTAs) to courses as instructors or grading assistants in a university setting. We analyze and model this problem in the real-world setting of the Department of Computer Science and Engineering of the University of Nebraska-Lincoln. This problem has traditionally been over-constrained. The results of our investigations are as follows. We enrich the definition of the problem by including students’ own preferences to be assigned to courses. We propose, implement, and evaluate a representational model of the entities and constraints that constitute this problem, and explain how the current model has emerged from, and improves upon, a series of modeling attempts. We also propose, implement, and test various optimization criteria that model ’satisfactory’ solutions. Finally, we describe, implement and test various computational mechanisms for solving this problem such as ordering heuristics, constraint propagation mechanisms, and search mechanisms.

We describe a complete system for quantita- tively measuring the optical distortion in aircraft wind- shields and automatically classifying that distortion as acceptable or not. The system comprises two parts: The first uses digital imaging of a known grid pattern through the windshield of interest to create a distortion map of that windshield; the second uses the distortion maps cre- ated for 100 windshields (some acceptable and some not) and automatically learns a decision-tree-based classifier. We show results that highlight the robustness of this system, including a demonstration that the distortion map for a windshield is consistently measured over three years, and cross-validation study shows that we can effec- tively classify windshields based on the distortion maps.

Scheduling the execution of multiple concurrent tasks on shared re- sources such as CPUs and network links is essential to ensuring the reliable and correct operation of real-time systems. For closed hard real-time systems in which task sets and the dependences among them are known a priori, existing real- time scheduling techniques can offer rigorous timing and preemption guarantees. However, for open soft-real-time systems in which task sets and dependences may vary or may not be known a priori and for which we would still like assurance of real-time behavior, new scheduling techniques are needed. Our recent work has shown that modeling non-preemptive resource sharing be- tween threads as a Markov Decision Process (MDP) produces (1) an analyzable utilization state space, and (2) a representation of a scheduling decision policy based on the MDP, even when task execution times are loosened from exact val- ues to known distributions within which the execution times may vary. However, if dependences among tasks, or the distributions of their execution times are not known, then how to obtain the appropriate MDP remains an open problem. In this paper, we posit that this problem can be addressed by applying focused reinforcement learning techniques. In doing so, our goal is to overcome a lack of knowledge about system tasks by observing their states (e.g., task resource utilizations) and their actions (e.g., which tasks are scheduled), and comparing the transitions among states under different actions to obtain models of system behavior through which to analyze and enforce desired system properties.