The exhilaration and enthusiasm which followed the passing of the Digital Services Act (DSA) is long over. No matter one’s perspective on the DSA, it seems clear that the party is over and the work begins. One of the perhaps oddest provisions of the DSA is Article 21. It calls for the creation of private quasi-courts that are supposed to adjudicate content moderation disputes. User Rights, based in Berlin, is one of the first organisations to assume this role.
The exhilaration and enthusiasm which followed the passing of the Digital Services Act (DSA) is long over. No matter one’s perspective on the DSA, it seems clear that the party is over and the work begins. One of the perhaps oddest provisions of the DSA is Article 21. It calls for the creation of private quasi-courts that are supposed to adjudicate content moderation disputes. User Rights, based in Berlin, is one of the first organisations to assume this role.
The European Union’s Digital Services Act (DSA) introduces a new regulatory approach to address the societal harms of online platforms: Systemic risk assessments. While a core component of the DSA, the regulation only outlines the standards and processes governing systemic risk assessments in broad strokes. It remains unclear what these systemic risk assesments will entail in practice. This Article develops a proposal of how systemic risk assessments should be implemented. It situates systemic risk assessments as a critical step toward platform accountability as they address societal harms, while existing approaches, such as remedy mechanisms, only protect user rights. Engaging with intangible harms and regulating speech and public discourse, risk assessments also entail significant challenges. Conventional reference points for content moderation regulation, such as terms and conditions, contractual freedom, fundamental rights and expertise, do not provide practical and legitimate bases to concretize risk assessment obligations. Public actors, such as the European Commission, should refrain from defining substantive standards, too, as they are directly bound by freedom of expression guarantees. Instead, the Article argues, the Commission should foster a procedural framework, a “virtuous loop,” which empowers civil society and allows it to specify and refine the standards governing systemic risks over time. Developing this framework, the Article explains how systemic risk assessment can fix “multistakeholderism,” and “multistakeholderism,” in turn, can help make systemic risk assessments work.
Artificial Intelligence (AI) is becoming the dominant instrument of generating knowledge in the 21st century — thereby shaping the future of humanity itself. It is frequently hailed as a miracle cure: a human-made tool that surpasses man in its magnificence, solving our problems and taking us to the next level of technical (and for that matter, human) development. In this article, however, we demonstrate that AI relies on a deterministic worldview, which contradicts our most fundamental cultural narratives. AI-based decision making systems turn predictions into self-fulfilling prophecies; not simply revealing the patterns underlying our world, but creating and enforcing them, to the detriment of the underprivileged, the exceptional, the unlikely. The widespread utilisation of AI dramatically aggravates the tension between the constraints of environment, society, and past behavior, and individuals' ability to alter the course of their lives, and to be masters of their own fate. Exposing hidden costs of the economic exploitation of AI, the article facilitates a philosophical discussion on responsible uses. It provides foundations of an ethical principle which allows us to shape the employment of AI in a way which aligns with our narratives and values.
Personalization has long been a feature of online services, shaping targeted advertisement and online manipulation. Corporations now seek to exploit the enormous economic potential of personalization beyond the confinements of the online space. In recent years, the proliferation of machine learning-based decision-making has led to personalization in all spheres of life. Corporations rely on machine learning-based systems to decide if and under what conditions they contract with individuals. They determine who is invited for job interviews and who is eligible for loans. They shape how we are perceived and tailor the way in which we are treated. The implications of these systems are already immense, and they foreshadow a larger transformation. Over the course of the 21st century, ubiquitous, machine learning-based personalization will likely permeate the economy and become a fundamental condition of human existence. The shape of this transformation is still uncertain; before it concretizes, we have the opportunity to guide its direction by articulating concepts that allow us to describe and critically examine it. * Niklas Eder is a Visiting Fellow at the Information Society Project at Yale Law School, Senior Policy Officer at the Oversight Board and founder of the project “Law in the Algorithmic Society.” I am grateful to Luise Durstewitz, Nawid Aludin, Yasmine Janah, Milky T. Asefa, Louis Hunter, Andrew Hadler, Nikolas Guggenberger, Artur Pericles Lima Monteiro, Przemek Palka, Yuval Goldfus, and the entire community of the Information Society Project for their support of this project. I also want to thank Jack Balkin, Frank Pasquale, Rebecca Wexler, and the organisers and participants of presentations at Yale Law School, the Privacy Workshop at Northeastern University, the European University Institute, the Legal Priorities Project, University of Tilburg, and Tartu University. Special thanks to Stan. Tech. L. Rev.’s Erich Remiker, Tanner Kuenneth, Kathryn Larkin, Olivia Malone, and Alex Evelson for their excellent editing. All errors are my own. Fall 2021 BEYOND AUTOMATION 2 Legal scholarship needs a conceptual foundation to address urgent questions about how personalization, driven by machine learning-based decision-making, affects liberty and other liberal democratic values. This article draws from surveillance theory and develops that foundation. It constructs a novel approach to examine the normative and constitutional implications of machine learning-based decision systems and ubiquitous personalization. The article builds on the concepts of panopticism and the surveillance assemblage to analyze how corporate machine learning-based decision-making affects the lives of individuals and transforms society. It is the first to develop an account of how ubiquitous personalization influences human agency and behavior. The article describes how machine learningbased decision systems amplify corporate power. It provides theoretical support for what Jack Balkin calls “normalization (or regimentation)”–the idea that algorithmic evaluations and decisions will govern human behavior. The article shows that existing legal responses focusing on rights, explainability, and transparency fail to prevent the already fragile balance of power between individuals and corporations from tipping in corporations’ interest. It argues that legal scholarship, to adequately respond to machine learning-based decision-making, must overcome its individualistic focus and engage in a debate on the legitimacy of corporate surveillance. Finally, the article explains why and how we should measure legal responses to machine learning-based decision-making against standards of legitimacy. The notion of legitimacy provides a foundation to tackle one of the great challenges with which machine learning-based systems confront liberal democracies—which is to reconcile corporate power with the values and freedoms central to these democracies. A legitimacy focus suggests that for legal responses to be adequate, they must not only educate individuals on the functions of algorithmic systems and endow them with legal rights. Rather, they must also entail general principles, such as data minimization and limitation, shaping the conditions based on which the personalization economy operates. TABLE OF CONTENTS
