Trading in the New Zealand wholesale electricity market (NZEM) pool began on 1 October 1996. The New Zealand pool market was the first one to use two elements of what has since become the North American standard market design, with locational marginal pricing and ancillary service cooptimization, but it was implemented in a physical and regulatory environment that differs markedly from that in North America. This article reflects on 20 years of experience with New Zealand's 1996 market design, assesses its points of difference from other jurisdictions, and speculates on some possible future directions.

Nitrate discharges from diffuse agricultural sources significantly contribute to groundwater and surface water pollution. Tradable permit programs have been proposed as a means of controlling nitrate emissions efficiently, but trading is complicated by the dispersed and delayed effects of the diffuse pollution. Hence, markets in nitrate discharge permits should be carefully designed to account for the underlying spatial and temporal interactions. Nitrate permit markets can be designed similar to the modern electricity markets which use LPs to find the equilibrium prices because the two trading problems have close analogy. In this paper, we propose alternative LP models to find efficient permit prices for year-ahead markets. The model structure varies depending on the catchment hydro-geology and long-term goals of the community. We show how the market price structures are driven by the constraint structure under different environmental conditions. We discuss the physical and economic conditions required to assure consistent prices, the modeling of essential and optional constraints in an LP, and the problem of balancing resource allocation over time among delayed-response discharge units. We then extend the LP model to balance resource allocation over time and to improve the market performance.

Constructive dual dynamic programming (CDDP) can be used to optimise a multi-nodal water system, or to clear a multi-nodal water market in the presence of competing consumptive demands, such as irrigation for farming, and non-consumptive demands, such as hydro-power generation. In this setting, CDDP is used in a two-stage process. First a deterministic multi-node version of the algorithm is used to construct a series of demand curves for release (ݎ ) under various catchment inflow scenarios. This is termed the 'intra- period' problem. Then a stochastic version of the algorithm uses the constructed ݎ 's to optimise the system, or clear the market over multiple periods, under uncertainty. This paper outlines the intra-period multi-node CDDP algorithm, and shows how to adapt this to address uses with water-mixing requirements, such as water returned to the river after being used for cooling a thermal power plant. All cost and benefit functions at the nodes are converted to net demand functions, i.e., marginal net benefit as a function of water supplied. For this, the nodal marginal value (or bid) information is re-cast to form net demand curves, and the arcs re-oriented towards the reservoir. Then the algorithm constructs the intra-period demand curve for release (ݎ ) by sequentially forming marginal water value curves at each node, passing these curves towards the reservoir. Arc flow bounds may limit the opportunities for using water at the nodes. Consumptive users extract water from the system, so each unit of water flow can only be used for a single consumptive use. A non-consumptive user transfers water from one node to another, extracting some benefit (e.g. from hydropower generation), or incurring some cost (e.g. for a pump). Costs can be associated with arc flow bounds and distributary demands to represent in-stream and environmental reserve flows enforced using penalty costs. High temperature return flows from thermal power plant cooling can affect the downstream ecosystem. As a result an additional flow past the thermal station is needed to control the temperature. The mixing flow required can be modelled as a fixed ratio of heated to unheated water. The paper explains how to extend a CDDP model to incorporate flow mixing externalities into the model, using multiple parallel arcs. The model assumes a single long-term storage reservoir embedded in a catchment with a "tree-like" topology. Consumptive demands occur at the nodes. Non-consumptive demands appear on arcs. Cooling and temperature control flow demands are represented as parallel arcs assuming that an upstream flow unit can be diverted to any arc of our choice, but other types of flow splitting are discussed. The deterministic CDDP algorithm then forms a net conditional ݎ for each node that implicitly trades off the marginal value of all uses, to maximise total value, or clear the intra-period market. That ݎ can then be incorporated into a stochastic CDDP algorithm to construct (inter-period) marginal storage values for the reservoir (and implicit release/allocation schedules for the entire catchment) over the entire planning horizon.

Society faces high costs due to environmental degradation from sediment discharge. Development can increase stream peak flows and sediment discharge. Excess sediment discharges threaten natural habitat, recreational places, rural areas, and commercial ports. In New Zealand, the 2004 North Island floods were estimated to cost about NZ$300 million. Annual soil erosion and sedimentation in NZ cost about NZ$127 million. As a potential solution to minimizing environmental impacts and social costs, we developed a smart market for sediment discharge. This approach would encourage individuals to internalize the environmental and social cost of sediment. The smart market system uses a hydrological simulator and a linear program that would allow an auctioneer to manage the third-party effects of trades, which are not possible with an ordinary auction. The system would give better price signals related to the sensitivity of key environmental features by location. Importantly, the proposed system would reduce transaction costs, because users do not need to search for trading partners, bargaining is simpler, price history information can be made available, and the auction manager ensures market discipline.

As one of the first Full Nodal Pricing (FNP) electricity markets, New Zealand was also one of the first places where FTR concepts were developed and considered for implementation, actually as early as 1989. Ironically, though, it is only now, after more than two decades of discussion, that a limited FTR market seems likely to be actually implemented. This long delay may be partly attributed to failures in the regulatory process, but it also reflects the special circumstances facing the small hydro-dominated New Zealand market, in which a relatively small group of vertically integrated participants compete over a fairly sparse network, in which losses and reserve support requirements play a more important role than line transfer limits, per se. Thus there has been considerable debate over whether classical FTR concepts are really suitable. We discuss several variant proposals, one of which is moving toward implementation by 2012.

Many electricity markets are now cleared using Linear Programming (LP) formulations that simultaneously determine an optimal dispatch and corresponding nodal prices, for each market dispatch interval. Although natural gas markets have traditionally operated in a very different fashion, the same basic concept can be applied. Since 1999, the Australian state of Victoria has operated a gas market based on an LP approximation to the underlying gas flow optimization problem. Here we discuss market design issues, using a formulation derived from the key gas flow equations. Dual variables on key constraints imply prices which vary by location, as for electricity markets, but also by time. But gas is both delayed and stored within the transportation system itself. This raises a number of operational, pricing, and hedging issues which could be ignored in the case of electricity, but become important when operating this kind of market for gas, or other commodities, such as water, in a supply network where there are delays and storage.

Since 1999, the Australian state of Victoria has operated a natural gas spot market to both determine daily prices for natural gas and develop an optimal schedule for the market based on an LP (Linear Programming) approximation to the underlying inter-temporal nonlinear aspects of the gas flow optimization problem. This market employs a dispatch optimization model and a related market clearing model. Here we present the model employed for both the operational scheduling and price determination. The basic dispatch optimization formulation covers the key physical relationships between pressure, flow, storage, with flow controlled by valves, and assisted by compressors, where flow and storage are measured with respect to energy rather than in terms of mass. But we also discuss a range of sophisticated mathematical techniques which have had to be employed to create a practical dispatch tool, including iterating between piecewise and successive linearization; iterating between barrier and simplex algorithms to manage numerical accuracy and solution speed issues, and special methods developed to deal with scheduling flexibility. The market clearing model is a variation on the dispatch optimization model which replaces the gas network with an infinite storage tank with unlimited transport capacity. We address the performance of the model including accuracy and run time.

Many electricity markets now co-optimize production and pricing of ancillary services such as contingency reserve and regulation with that of energy. This approach has proved successful in reducing ancillary service costs and in providing consistent pricing incentives for potential providers of both energy and ancillary services Here we discuss the basic concepts involved, the optimization formulations employed to clear such co-optimized markets, and some of the practical issues that arise.

Dynamic programming (DP) is a well established technique for optimization of reservoir management strategies in hydro generation systems, and elsewhere. Computational efficiency has always been a major issue, though, at least for multi-reservoir problems. Although the dual of the DP problem has received little attention in the literature, it yields insights that can be used to reduce computational requirements significantly. The stochastic dual DP algorithm (SDDP) is one well known optimization model that combines insights from DP and mathematical programming to deal with problems of much higher dimension that could be addressed by DP alone. Here, though, we describe an alternative "constructive" dual DP technique, which has proved to be both efficient and flexible when applied to both optimization and simulation for reservoir problems of modest dimension. The approach is illustrated by models from New Zealand and the Nordic region.

"Top-down" models, based on observation of market price patterns, may be used to forecast prices in competitive electricity markets, once a reasonable track record is available and provided the market structure is stable. But many studies relate to potential changes in market structure, while prices in hydro-dominated markets are driven by inflow fluctuations and reservoir management strategies, operating over such a long timescale that an adequate track record may not be available for decades, by which time the system itself will be very different. "Bottom-up" analysis can readily model structural change and hydro variation, but must make assumptions about fundamental system data, commercial drivers, and rational optimizing behavior that leave significant unexplained price volatility. Here we describe a technique for fitting a hybrid model, in which a "top-down" approach is used to estimate parameters for a simplified "bottom-up" model of participant behavior, from market data, along with a stochastic process describing residual price volatility. This fitted model is then used to simulate market behavior as fundamental parameters vary. We briefly survey actual and potential applications in other markets, with differing characteristics, but mainly illustrate the application of this hybrid approach to the hydro-dominated New Zealand Electricity Market, where participant behavior can be largely explained by fitted "marginal water value curves." A second application of a hybrid model, to the Australian National Electricity Market, is also provided.

Efficient management of water requires balancing environmental needs, externality considerations, and economic efficiency. Toward that end, this paper presents a deterministic linear program that could be used to operate a smart spot market for groundwater. The market design uses the existing hydrological programs MODFLOW and GWM along with standard linear programming methods. In principle, a market could be set up anywhere that a MODFLOW model is available. The market design has parallels to markets in the electricity and gas sectors, which we discuss. We present a case study with notional bids for Marlborough, New Zealand. Our approach would reduce transaction costs for a water market, reduce users' risk, and increase the reliability of environmental flows. We discuss a number of cautions and limitations to the model and recommend further work on introducing a stochastic framework to the model.

Recent econometric models of spot market prices are particularly well-suited for thermaldominated systems, in which extreme short-term price volatility and strong mean-reversion are dominant characteristics. In order for a purely econometric price model to perform well in the context of the New Zealand market, which is dominated by hydro generation, it needs to be modified to incorporate the physical factors that influence the price level. Hydrological factors, such as storage levels and inflows, are major drivers of hydro generator behaviour. Assuming that generators use modern reservoir management optimisations, both factors are taken into account in the calculation of marginal water values (MWVs), which, theoretically, form the basis of their supply offers. However, the MWVs are assessed internally and are not public knowledge; therefore some proxy for the MWV is required for modelling spot prices. We compare the estimated parameters of a leading econometric model when fitted to two spot price time series from the New Zealand Electricity Market, to show how water-shortages leading to high prices can complicate the price-modelling process. We then use reservoir management theory to extend the price model. The storage level is transformed into a crude measure of the MWV, and incorporated into the model as the major driver of the deterministic price level. Our analysis shows that underlying spot price levels in New Zealand (especially during dry years) can be modelled with increased accuracy when the storage level is included in the price model.

Payment for reactive power services is a part of generator revenue in most established electricity markets today. However, such payments do not have a sound technical basis taking into account the critical role dynamic sources of reactive power play in maintaining voltage stability of the power system. It is also expected that the reactive power will eventually be traded under a spot market arrangement as the markets mature. This paper proposes a spot pricing mechanism for reactive power that takes into account the contributions made by generators both by providing reactive power to meet the current demand and also holding reactive power ‘reserve’ to maintain the system voltage stability in the face of a potential contingency. An OPF based dispatch/pricing model is proposed that optimally allocates real and dynamic reactive reserve among the generators to meet a pre-specified voltage stability margin. Nodal MW and MVAr spot prices are affected by the desired stability margin because an increase in MVAr and/or MW demand has the cascading effect of increasing the dynamic MVAr/MW reserve requirement. Simplistic illustrative examples are presented to develop insights on the theoretical premise. Experiments conducted for a reduced New Zealand North Island power system reveal some interesting impacts of the voltage stability constraint on real and reactive power prices.

The reservoir management problem for a hydrothermal power system is well suited to modeling via dynamic programing. In this paper we describe a dual approach which we term “constructive dynamic programming” (CDP) which has been successfully applied to optimize releases in a stochastic two‐reservoir model of the New Zealand power system. That model ignores serial correlations of inflows, though, and hence assumes that current inflow observations do not have any impact on future release decisions. Tests show, however, that better decision rules can be produced by accounting for inflow correlation. Hence we have developed an extension to the standard CDP to explicitly deal with serial correlation of reservoir inflows, and we report on those extensions also.

Over the past few years many authors have documented the demise of some traditional forms of OR practice. But at the same time, significant success has been reported in other areas. It is suggested that this pattern of failure and success reflects the degree to which OR has proved adaptable to a shift away from a reliance on top-down planning, toward reliance on pricing mechanisms; that is from a “primal” to a “dual” orientation, both in business and in Government. If so, this suggests significant potential in the adoption of a Dual OR paradigm which emphasises the use of OR to create and support “market” structures within which independent decision-makers can operate. Conversely, though, the OR profession must ask whether its current tool-kit is really adequate to the job ahead.

Market clearing models based on Linear Programming are being adopted in a number of electricity markets in different countries, with New Zealand and Australia being among the pioneers in this respect. This represents an exciting development for OR, worldwide, not only because these models are controlling a sector of vital national importance, but because large sums of money are being traded, virtually in real time, on the basis of model results. Here we reflect on experience with these models, draw out some lessons, and discuss the implications of these developments for OR practice in the sector, and elsewhere.

The scheduling of hydro stations has stochastic, integer, non-linear, and continuous time aspects, with all the approaches described to date making some simplifying assumption about one or other of these aspects. The (integer) unit commitment decision has received relatively little attention, partly because the resulting problem was deemed intractable given the potential gains in efficiency. However, with the advent of deregulated energy markets, the implications of ignoring these integer effects may be costly, and so they must be considered in some way. We discuss some of the managerial and modelling issues relating to this problem and some ideas for heuristics that incorporate management priorities into an Integer Programming framework.

Over the past few years many-authors, including Fildes and Ranyard(1,2,3), have documented the demise of some traditional forms of OR practice. The same trends have been evident in New Zealand, but we have also seen the development of new areas of OR endeavour, and numerical growth in the profession, as a result of adopting a "dual OR" paradigm which emphasises the use of OR to create and support structures, particularly internal or external "markets", within which independent decision-makers can operate. This implies the need to train a small group of OR people specialising in high level design issues, and a much larger group able to assist relatively small decision-making units.