The expansion of residential demand side management programs and increased deployment of controllable loads require accurate appliance-level load modeling and forecasting. This paper proposes a conditional hidden semi-Markov model to describe the probabilistic nature of residential appliance demand. Model parameters are estimated directly from power consumption data using scalable statistical learning methods. We also propose an algorithm for short-term load forecasting as a key application for appliance-level load models. Case studies performed using granular sub-metered power measurements from various types of appliances demonstrate the effectiveness of the proposed load model for short-term prediction.

The increasing penetration of renewable resources has changed the characteristics of power system and market operations, from one relying primarily on deterministic and static planning to one involving highly stochastic and dynamic operations. In such new operation regimes, the ability of adapting changing environments and managing risks arising from complex scenarios of contingencies is essential. To this end, an operation tool that provides probabilistic forecasting that characterizes the underlying probability distribution of variables of interest can be extremely valuable. A fundamental challenge in probabilistic forecasting for system and market operations is the scalability. As the size of system and the complexity of stochasticity increase, standard techniques based on direct Monte Carlo and machine learning techniques become intractable. This chapter outlines an alternative approach based on an online learning to overcome barriers of computation complexity.

A new stochastic control methodology is introduced for distributed control, motivated by the goal of creating virtual energy storage from flexible electric loads, i.e. Demand Dispatch. In recent work, the authors have introduced Kullback- Leibler-Quadratic (KLQ) optimal control as a stochastic control methodology for Markovian models. This paper develops KLQ theory and demonstrates its applicability to demand dispatch. In one formulation of the design, the grid balancing authority simply broadcasts the desired tracking signal, and the hetero-geneous population of loads ramps power consumption up and down to accurately track the signal. Analysis of the Lagrangian dual of the KLQ optimization problem leads to a menu of solution options, and expressions of the gradient and Hessian suitable for Monte-Carlo-based optimization. Numerical results illustrate these theoretical results.

The problem of disaggregating the household electricity demand into appliance-level consumption is considered. A deep neural network, based on the long short term memory model, is developed to jointly predict all appliances by leveraging information of neighboring homes. The proposed technique is evaluated on a real-world data set with over 300 homes and more than 20 appliances. Numerical results show significant improvement of the proposed technique over the baseline without joint training and using the wisdom of neighbors.

A generalization of the coordinated transaction scheduling (CTS)---the state-of-the-art interchange scheduling---is proposed. Referred to as generalized coordinated transaction scheduling (GCTS), the proposed approach addresses major seams issues of CTS: the ad hoc use of proxy buses, the presence of loop flow as a result of proxy bus approximation, and difficulties in dealing with multiple interfaces. By allowing market participants to submit bids across market boundaries, GCTS also generalizes the joint economic dispatch that achieves seamless interchange without market participants. It is shown that GCTS asymptotically achieves seamless interface under certain conditions. GCTS is also shown to be revenue adequate in that each regional market has a non-negative net revenue that is equal to its congestion rent. Numerical examples are presented to illustrate the quantitative improvement of the proposed approach.

The problem of multi-area interchange scheduling under system uncertainty is considered in this paper. A new scheduling technique is proposed for a multi-proxy bus system based on stochastic optimization that captures uncertainty in renewable generation and stochastic load. In particular, the proposed algorithm iteratively optimizes interface flows using multidimensional demand and supply functions. Optimality and convergence are guaranteed for both synchronous and asynchronous scheduling under nominal assumptions.

The short-term forecasting of real-time locational marginal price (LMP) and network congestion is considered from a system operator perspective. A new probabilistic forecasting technique is proposed based on a multiparametric programming formulation that partitions the uncertainty parameter space into critical regions from which the conditional probability distribution of the real-time LMP/congestion is obtained. The proposed method incorporates load/generation forecast, time varying operation constraints, and contingency models. By shifting the computation associated with multiparametric programs offline, the online computational cost is significantly reduced. An online simulation technique by generating critical regions dynamically is also proposed, which results in several orders of magnitude improvement in the computational cost over standard Monte Carlo methods.

The problem of incorporating interface bids by market participants in the coordinated economic dispatch for multi-area power systems is considered. Interface bids facilitate power flow between different areas whereas different system operators do not trade directly. The model of interface bids in coordinated economic dispatch is presented. Interface bids are cleared prior to real time markets, with the objective to minimize the sum of costs of internal and interface bids, and settled together with internal bids by real time locational marginal prices. Properties of individual optimality and revenue adequacy are established. Numerical experiments show the advantages of the proposed method over existing benchmarks.

The problem of probabilistic forecasting and online simulation of real-time electricity market with stochastic generation and demand is considered. By exploiting the parametric structure of the direct current optimal power flow, a new technique based on online dictionary learning (ODL) is proposed. The ODL approach incorporates real-time measurements and historical traces to produce forecasts of joint and marginal probability distributions of future locational marginal prices, power flows, and dispatch levels, conditional on the system state at the time of forecasting. Compared with standard Monte Carlo simulation techniques, the ODL approach offers several orders of magnitude improvement in computation time, making it feasible for online forecasting of market operations. Numerical simulations on large and moderate size power systems illustrate its performance and complexity features and its potential as a tool for system operators.

The problem of inter-regional interchange scheduling in the presence of stochastic generation and load is considered. An interchange scheduling technique based on a two-stage stochastic minimization of expected operating cost is proposed. Because directly solving the stochastic optimization is intractable, an equivalent problem that maximizes the expected social welfare is formulated. The proposed technique leverages the operator's capability of forecasting locational marginal prices and obtains the optimal interchange schedule without iterations among operators. Several extensions of the proposed technique are also discussed.

Smart metering and submetering technologies make energy data available at the granularity of individual appliance. Based on a real-world data set, we characterize energy consumption of individual appliances, and quantify the flexibility for demand response as realizable increase and decrease of energy consumption. Results show significant flexibility potential in residential appliances and substantial cost savings for customers under time-of-use pricing.

The problem of incorporating interface bids into the coordinated economic dispatch for multi-area power systems is considered. Interface bids are submitted by external market participants. They facilitate interregional power transactions and preserve the financial neutrality of independent system operators. A model of interface bids is presented and its clearing process and settlement mechanism are proposed. For each independent system operator, its revenue adequacy is established and the local performance in its operating area is analyzed. Simulation results demonstrate the effectiveness of the proposed method and verify the established properties.

The problem of inter-regional interchange scheduling using a multiple proxy bus representation is considered. A new scheduling technique is proposed for the multi-proxy bus system based on a stochastic optimization that captures uncertainty in renewable generation and stochastic load. In particular, the proposed algorithm iteratively optimizes the interchange across multiple proxy buses using a vectorized notion of demand and supply functions. The proposed technique leverages the operator's capability of forecasting locational marginal prices (LMPs) and obtains the optimal interchange schedule directly without iterations between operators.

The problem of real-time interchange scheduling between two independently operated regions is considered. An optimal scheduling scheme is proposed by maximizing the expected economic surplus based on Coordinated Transaction Scheduling (CTS) mechanism. The proposed technique incorporates probabilistic forecasts of renewable generation to optimize the interchange schedule using a parametric programming formulation, from which statistical real-time generation supply offer curves are constructed.

The problem of short-term probabilistic forecast of real-time locational marginal price (LMP) is considered. A new forecast technique is proposed based on a multiparametric programming formulation that partitions the uncertainty parameter space into critical regions from which the conditional probability mass function of the real-time LMP is estimated using Monte Carlo techniques. The proposed methodology incorporates uncertainty models such as load and stochastic generation forecasts and system contingency models. With the use of offline computation of multiparametric linear programming, online computation cost is significantly reduced.

As a state-of-the-art programming model for big data analytics, MapReduce is well suited for parallel processing of large data sets in opportunistic environments. Existing research on MapReduce in opportunistic environment has focused on improving single job performance; the issue of fairness that is critical in the more dominant scenario of multiple concurrent jobs remains unexplored. We address this problem by proposing an opportunistic fair scheduling algorithm, which extends the broadly adopted Fair Scheduler to an environment where nodes are intermittently available with possibly different availability patterns. The proposed scheduler maintains statistics specific to the opportunistic environment, e.g., node availability rates and pairwise availability correlations, and utilizes this information in scheduling decisions to improve fairness. Using a Hadoop-based implementation, we compare our scheduler with the current Hadoop Fair Scheduler on representative benchmarks. Our experiments verify that our scheduler can significantly reduce the variability in job completion times.

The problem of forecasting the real-time locational marginal price (LMP) by a system operator is considered. A new probabilistic forecasting framework is developed based on a time in-homogeneous Markov chain representation of the realtime LMP calculation. By incorporating real-time measurements and forecasts, the proposed forecasting algorithm generates the posterior probability distribution of future locational marginal prices with forecast horizons of 6-8 hours. Such a short-term forecast provides actionable information for market participants and system operators. A Monte Carlo technique is used to estimate the posterior transition probabilities of the Markov chain, and the real-time LMP forecast is computed by the product of the estimated transition matrices. The proposed forecasting algorithm is tested on the PJM 5-bus system. Simulations show marked improvements over benchmark techniques.