AI is increasingly being used to help with AI R D. Under certain conditions this feedback loop might be able to produce an intelligence explosion, with rapidly escalating AI capabilities. I explore the mathematics of the most explosive possibilities, with an eye to understanding what drives the dynamics. I show that singular growth (towards a vertical asymptote) is harder to achieve than would be expected from recent economics-inspired modelling, and that there is an important but neglected class of growth rates that are faster than exponential but don't lead to a vertical asymptote. I draw out the generation time (the time to go around the feedback loop) as a neglected parameter that plays a pivotal role in determining the behaviour of any intelligence explosion — one cannot have singular growth unless the generation time rapidly approaches zero.

This paper introduces the concept of hyperpolation: a way of generalising from a limited set of data points that is a peer to the more familiar concepts of interpolation and extrapolation. Hyperpolation is the task of estimating the value of a function at new locations that lie outside the subspace (or manifold) of the existing data. We shall see that hyperpolation is possible and explore its links to creativity in the arts and sciences. We will also examine the role of hyperpolation in machine learning and suggest that the lack of fundamental creativity in current AI systems is deeply connected to their limited ability to hyperpolate.

The Lindy effect is a statistical tendency for things with longer pasts behind them to have longer futures ahead. It has been experimentally confirmed to apply to some categories, but not others, raising questions about when it is applicable and why. I shed some light on these questions by examining the mathematical properties required for the effect and generating mechanisms that can produce them. While the Lindy effect is often thought to require a declining hazard rate, I show that it arises very naturally even in cases with constant (or increasing) hazard rates -- so long as there is a probability distribution over the size of that rate. One implication is that even things which are becoming less robust over time can display the Lindy effect.

This paper explores the fundamental causal limits on how much of the universe we can observe or affect. It distinguishes four principal regions: the affectable universe, the observable universe, the eventually observable universe, and the ultimately observable universe. It then shows how these (and other) causal limits set physical bounds on what spacefaring civilisations could achieve over the longterm future.

AbstractWe introduce a summary wellbeing measure for economic evaluation of cross‐sectoral public policies with impacts on health and living standards. We show how to calculate period‐specific and lifetime wellbeing using quality‐adjusted life years based on widely available data on health‐related quality of life and consumption and normative assumptions about three parameters—minimal consumption, standard consumption, and the elasticity of the marginal value of consumption. We also illustrate how these three parameters can be tailored to the decision‐making context and varied in sensitivity analysis to provide information about the implications of alternative value judgments. As well as providing a general measure for cost‐effectiveness analysis and cost‐benefit analysis in terms of wellbeing, this approach also facilitates distributional analysis in terms of how many good years different population subgroups can expect to live under different policy scenarios.

We present a model where some proportion of extraterrestrial civilizations expand uniformly over time to reach a cosmological scale. We then ask what humanity could infer if a sky survey were to find zero, one, or more such civilizations. We show how the results of this survey, in combination with an approach to anthropics called the Self Indication Assumption (SIA), would shift any prior estimates of two quantities: 1) The chance that a technological civilization like ours survives to embark on such expansion, and 2) the maximum feasible speed at which it could expand. The SIA gives pessimistic estimates for both, but survey results (even null results) can reverse some of the effect. The SIA-induced shift in expectations is strong enough to be regarded as a falsifiable technological prediction, and a test of the SIA in cosmological reasoning.

The Repugnant Conclusion is an implication of some approaches to population ethics. It states, in Derek Parfit's original formulation, For any possible population of at least ten billion people, all with a very high quality of life, there must be some much larger imaginable population whose existence, if other things are equal, would be better, even though its members have lives that are barely worth living. (Parfit 1984: 388)

We introduce a novel approach to the problem of decision-making under moral uncertainty, based on an analogy to a parliament. The appropriate choice under moral uncertainty is the one that would be reached by a parliament comprised of delegates representing the interests of each moral theory, who number in proportion to your credence in that theory. We present what we see as the best specific approach of this kind (based on proportional chances voting), and also show how the parliamentary approach can be used as a general framework for thinking about moral uncertainty, where extant approaches to addressing moral uncertainty correspond to parliaments with different rules and procedures.

La pandemia de COVID-19 pone al descubierto la estrecha interrelación que existe en la humanidad. Con el incremento de las nuevas tecnologías, nuestro poder humano sigue creciendo, al igual que los riesgos letales. Las amenazas para el futuro humano se conocen como riesgos existenciales y la forma como los abordemos determinará el futuro de nuestra especie. Este escenario impone una necesidad de replantear la ética. Reflexionar y tomar acciones no solamente desde el punto de vista individual, sino como colectivo humano, y tampoco solamente centrado al presente o futuro inmediato, sino con una nueva ética que piense en las innumerables generaciones por nacer.

A major problem for interpersonal aggregation is how to compare utility across individuals; a major problem for decision-making under normative uncertainty is the formally analogous problem of how to compare choice-worthiness across theories. We introduce and study a class of methods, which we call statistical normalization methods, for making interpersonal comparisons of utility and intertheoretic comparisons of choice-worthiness. We argue against the statistical normalization methods that have been proposed in the literature. We argue, instead, in favor of normalization of variance: we claim that this is the account that most plausibly gives all individuals or theories ‘equal say’. To this end, we provide two proofs that variance normalization has desirable properties that all other normalization methods lack, though we also show how different assumptions could lead one to axiomatize alternative statistical normalization methods.

This chapter discusses how to take into account moral uncertainty over interval-scale measurable but non-comparable theories. Once again, we make use of the analogy between decision-making under moral uncertainty and voting. We give examples of interval-scale theories where it’s plausible to think that these theories are incomparable with each other and discuss what to do in such cases. Arguing against the Borda Rule and Ted Lockhart’s Principle of Equity Among Moral Theories, we argue in favour of an account we call variance voting. Finally, we discuss what to do in conditions where one has positive credence in some merely ordinal theories, some interval-scale but non-comparable theories, and some theories that are both interval-scale measurable and comparable with each other. We discuss whether the normalization used by this account should be done only within the decision-situation at hand, or whether it should be done over all possible decision-situations.

In this chapter, we discuss two further problems that face accounts of decision-making under moral uncertainty, and are particularly pressing for theories that involve maximizing expected choice worthiness. First, the fanaticism problem—that the expected choice worthiness of options might be primarily determined by tiny credences in theories that posit huge amounts of value. Second, the infectious incomparability problem—that any credence in theories with radical incomparability might render the expected choice worthiness of almost every option undefined. We argue that both problems can be overcome. The fanaticism problem is a problem for decision-making under uncertainty in general, not just for decision-making in the face of moral uncertainty. The infectious incomparability problem requires an account of decision-making in the face of both moral uncertainty and incomparability. We suggest one possible account.

In this chapter we introduce the topic of moral uncertainty, argue that moral uncertainty is a real and significant issue, and argue that there are non-trivial answers to the question, ‘Given that we are morally uncertain, how ought we to act in light of that uncertainty?’ In particular, we argue that there is an answer to the question of what a morally conscientious agent rationally should do when she is morally uncertain. We shall also consider and defend against some recent objections to the very project of trying to develop an account of decision-making under moral uncertainty: we’ll call these the fetishism objection; the regress objection; the blameworthiness objection; the conscientiousness objection; and the disanalogy with prudence objection.

We introduce and discuss the problems of intertheoretic incomparability and merely ordinal theories. We then introduce the analogy between decision-making under moral uncertainty and social choice, and explain how this analogy can help us to overcome these problems. The rest of the chapter is spent fleshing out how this idea can help us to develop a theory of decision-making under moral uncertainty that is applicable even when all theories under consideration are merely ordinal, and even when there is neither level-nor unit- comparability between those theories. We consider whether My Favourite Theory or My Favourite Option might be the right theory of decision-making under moral uncertainty in conditions of merely ordinal theories and incomparability, but reject both of these accounts. We defend the idea that, when maximizing choice worthiness is not possible, one should use the Borda Rule instead.

In this chapter we consider the extent to which different theories are unit-comparable, and what makes them comparable when they are. We consider three arguments for the conclusion that intertheoretic comparisons are always impossible: the appeal to cases argument, the swamping argument, and the arbitrary unit arguments. We argue against all three arguments. We distinguish between structural and non-structural accounts of intertheoretic comparisons. We argue in favour of non-structural accounts: we argue that intertheoretic comparisons are grounded in substantive facts about the theories themselves (rather than merely statistical properties of their choice worthiness function). We discuss a number of possible accounts of intertheoretic comparisons, ultimately arguing in favour of a ‘universal scale’ account.

In this chapter, we show how the theory we’ve given can shed light on the question of how to value gaining new moral information. We explain how we should assess the expected utility of new empirical information, and how we could use an analogous framework to work out the expected choice worthiness of new moral information. We apply this framework to two examples: the choice of how a large foundation should spend its resources, and the choice of career for an individual. Finally, we consider to what extent the lessons from this framework change when we consider ‘imperfect’ information.

In this chapter we argue that, in conditions of interval-scale measurability and unit-comparability, one should maximize expected choice worthiness. Though this position has often been suggested in the literature and is often taken to be the ‘default’ view, it has so far received little in the way of positive argument in its favour. We start, in section I, by providing new arguments against two rival theories that have been proposed in the literature—the accounts which we call My Favourite Theory and My Favourite Option. Then we give a novel argument for the view that, under moral uncertainty, one should take into account both probabilities of different theories and magnitudes of choice-worthiness. Finally, we argue in favour of maximizing expected choice-worthiness (MEC).

This paper argues in favor of a particular account of decision-making under normative uncertainty: that, when it is possible to do so, one should maximize expected choice-worthiness. Though this position has been often suggested in the literature and is often taken to be the ‘default’ view, it has so far received little in the way of positive argument in its favor. After dealing with some preliminaries and giving the basic motivation for taking normative uncertainty into account in our decision-making, we consider and provide new arguments against two rival accounts that have been offered— the accounts that we call ‘My Favorite Theory’ and ‘My Favorite Option’. We then give a novel argument for comparativism — the view that, under normative uncertainty, one should take into account both probabilities of different theories and magnitudes of choice-worthiness. Finally, we further argue in favor of maximizing expected choice-worthiness and consider and respond to five objections.

This chapter proposes a practical measure of individual well-being to facilitate the economic evaluation of public policies. The authors propose to evaluate policies in terms of years of good life gained, in a practical and flexible way that complements and builds upon the standard outcome measures used in cost-effectiveness and cost–benefit analysis. The authors show how to do this by adjusting years of life lived for consumption-related quality of life—that is, the material standard of living—as well as health-related quality of life. This is a straightforward extension of the quality-adjusted life year metric used in health economics for measuring years of healthy life. The authors’ approach allows for differences between people in the marginal value of money. It also permits distributional impact analysis in terms of lifetime well-being—that is, how many good years of life different people can expect over the course of their lives. The authors aim to show how years of good life could be measured in practice by harnessing readily available data on three important elements of individual well-being: consumption, health-related quality of life, and mortality. They also aim to identify the main ethical assumptions needed to use this measure.

In this chapter, we show that moral uncertainty creates a challenge for another metaethical view, namely non-cognitivism, according to which moral judgements are desires, or some other desire-like states, rather than beliefs. We show that it is surprisingly difficult, though perhaps not impossible, for non-cognitivists to accommodate moral uncertainty, for they lack the resources to adequately distinguish degrees of moral confidence and degrees of value ascribed to things. We discuss Lenman’s and Ridge’s versions of ecumenical expressivism and argue that neither are able to satisfactorilty explain moral uncertainty. We consider Sepielli’s defense, based on the ‘being for’ account of normative certitude, but argue that it, too, suffers from grave problems.