As generative AI technologies find more and more real-world applications, the importance of testing their performance and safety is paramount. “Red-teaming” has quickly become the primary approach to testing AI models—prioritized by AI companies, and enshrined in AI policy and regulation. Members of red teams act as adversaries, probing AI systems to test their safety mechanisms and uncover vulnerabilities. Yet we know far too little about this work or its implications. In this article, we highlight the importance of understanding the values and assumptions behind red-teaming, the labor arrangements involved, and the psychological impacts on red-teamers, drawing insights from lessons learned around the work of content moderation. Red-teaming should be a deeply interdisciplinary concern. To avoid repeating the mistakes of the recent past, we call for a coordinated network of scholars, from the full range of the computational and social sciences, to study the technical, social, critical, and policy dimensions of red-teaming and of the emerging sociotechnical system that is AI.

Proponents of generative AI tools claim they will supplement, even replace, the work of cultural production. This raises questions about the politics of visibility: what kinds of stories do these tools tend to generate, and what do they generally not? Do these tools match the kind of diversity of representation that marginalized populations and non-normative communities have fought to secure in publishing and broadcast media? I tested three widely available generative AI tools with prompts designed to reveal these normative assumptions; I prompted the tools multiple times with each, to track the diversity of the outputs to the same query. I demonstrate that, as currently designed and trained, generative AI tools tend to reproduce normative identities and narratives, rarely representing less common arrangements and perspectives. When they do generate variety, it is often narrow, maintaining deeper normative assumptions in what remains absent.

As generative AI technologies find more and more real-world applications, the importance of testing their performance and safety seems paramount. "Red-teaming" has quickly become the primary approach to test AI models–prioritized by AI companies, and enshrined in AI policy and regulation. Members of red teams act as adversaries, probing AI systems to test their safety mechanisms and uncover vulnerabilities. Yet we know far too little about this work or its implications. This essay calls for collaboration between computer scientists and social scientists to study the sociotechnical systems surrounding AI technologies, including the work of red-teaming, to avoid repeating the mistakes of the recent past. We highlight the importance of understanding the values and assumptions behind red-teaming, the labor arrangements involved, and the psychological impacts on red-teamers, drawing insights from the lessons learned around the work of content moderation.

Within academia, industry, and government, the terms ‘health’ and ‘toxicity’ are widely used to describe and justify decisions around online content and its removal. However, the meanings of these terms are assumed to be self-evident and therefore are rarely examined. This article turns a critical eye to the health and toxicity metaphor to unpack its hidden political work. We trace the metaphor through three different discourses: the historical political economy of the term, the usage by cultural elites in the last two decades, and finally through its contemporary instrumental usage by volunteer content moderators on Facebook. By linking these discourses together, we argue that the metaphor of health and toxicity serves as a means for justification and legitimacy under contemporary neoliberalized orders that typically chafe at modes of public intervention and the language of democratic statecraft. Rather than elucidating the challenges of online content, we find that the metaphor often serves to obfuscate or sidestep the hardest problems in democratic governance. This analysis therefore has practical significance for researchers, policymakers, journalists, and other speakers that publicly traffic in this discourse at large.

Recent social science concerning the information technology industries has been driven by a sense of urgency around the problems social media platforms face. But it need not be our job to solve the problems these industries have created, at least not on the terms in which they offer them. When researchers are enlisted in solving the industry’s problems, we tend to repeat some of the missteps common to the study of technology and society.

Public debate about content moderation has overwhelmingly focused on removal: social media platforms deleting content and suspending users, or opting not to do so. However, removal is not the only available remedy. Reducing the visibility of problematic content is becoming a commonplace element of platform governance. Platforms use machine learning classifiers to identify content they judge misleading enough, risky enough, or offensive enough that, while it does not warrant removal according to the site guidelines, warrants demoting them in algorithmic rankings and recommendations. In this essay, I document this shift and explain how reduction works. I then raise questions about what it means to use recommendation as a means of content moderation.

In recent years, a variety of cultural industries have been transformed by platformization – a process in which technology companies serve as intermediaries connecting different parties (most importantly cultural producers and audiences) through websites and applications. From music to book publishing, movie production, and the visual arts, cultural production has undergone massive changes due to platforms such as Spotify, Apple Music, Amazon, Goodreads, YouTube, Facebook, Instagram, TikTok, Etsy, etc. In most cultural sectors, creators now have to grapple with the platforms that make their work visible to online audiences. This often means paying close attention to the quantitative infrastructure of platforms, namely their algorithms and analytics, which drive visibility and commercial success. This panel examines what these economic and technological changes imply for the independence of cultural production. Classical studies of culture often emphasized the role that the values of independence and autonomy play in shaping artists’ worldviews and practices. From Bourdieu’s analysis of “fields of cultural production” as “the economic world reversed” to Becker’s theory of “art worlds” where internal dynamics redefine external constraints, or the Frankfurt School’s critical take on the demise of aura through mechanical reproduction, sociological approaches have paid close attention to the threats to independence emerging under modern capitalism. In fact, most classical sociology saw cultural producers in the mass cultural industries as having little independence, often assuming (sometimes without empirical research) that the massification of culture would destroy original and critical art works. Here we revisit the question of independence in the context of platformed creation – a term that embraces all forms of cultural production that are mediated, in part or completely, through digital platforms. By bringing together scholars studying different aspects of platformed creation and reflecting on the concept of independence through diverse disciplinary lenses, we ask: what does independence look like in the context of platformed creation? What are some of the theoretical and methodological tools available to scholars for making sense of cultural, economic, and technological independence in the case of platformed creation? And how do these evolving forms of independence affect the kinds of art works and cultural tropes that circulate online? These different studies aim to put the concept of independence in dialogue with the question of interdependence (among cultural producers, audiences, and platforms) in a mediated digital world.

This article considers how media workers and organizations make use of the abundance of metrics available in the contemporary online environment. The expansion of audience measurement on digital music platforms, dashboard analytics, and third-party providers raises broad societal concerns about the quantification of culture; however, less attention has been paid to how professionals in the music industries approach, understand, and deploy these metrics in their work. Drawing on survey and interview data, we found that music workers do not take metrics on faith or reject them out of hand; rather, they make sense of them, deploy them strategically, and narrate their meanings to give themselves rationales to make investments and predictions and to persuade others to do so.

Les fournisseurs de contenu en ligne tels que YouTube se positionnent soigneusement vis-à-vis des utilisateurs, des clients, des annonceurs et des décideurs politiques, en communiquant de manière stratégique à propos de ce sur quoi ils interviennent et n'interviennent pas, ainsi que sur la façon dont leur activité doit être comprise dans le paysage de l'information. Un terme en particulier, celui de « plateforme », révèle les contours de ce travail discursif. Il a été mobilisé aussi bien dans les argumentaires marketing que pour faire appel à la dimension populaire de certains services – qu'il s'agisse de plateformes techniques, de scènes [platform] d'où l'on peut parler, ou encore, de tremplin [platform] vers des opportunités nouvelles. Les tensions qui peuvent apparaître, à partir du moment où l'on prétend servir tous ces différents groupes d'intérêt, sont soigneusement éludées. Le terme de « plateforme » convient aussi aux fournisseurs de contenu lorsqu'ils essaient d'intervenir dans les politiques de l'information afin de chercher à être protégés dans leur rôle de soutien à l'expression des utilisateurs, sans être pour autant tenus pour responsables des contenus qui circulent. Alors que ces fournisseurs deviennent les programmateurs du débat public, nous devons examiner les rôles qu'ils prétendent jouer ainsi que les termes à l'aune desquels ils espèrent être jugés.

Hosted by Northumbria and Birmingham City Universities, the Deplatforming Sex roundtable took place via Teams in October 2021. Participants included Danielle Blunt, Stefanie Duguay, Tarleton Gillespie and Sinnamon Love. Clarissa Smith chaired the discussion, which was transcribed and then edited to cut digressions and repetitions for publication. The roundtable provided the opportunity to reflect on recent moves to excise sex and forms of sexual commerce and performance from online spaces, while marking out some key issues for future research with and about sex workers, performers and other content providers. Our discussion provided critical engagement with ongoing legislative changes that are impacting content and providers directly and indirectly.

Content moderation has exploded as a policy, advocacy, and public concern. But these debates still tend to be driven by high-profile incidents and to focus on the largest, US based platforms. In order to contribute to informed policymaking, scholarship in this area needs to recognise that moderation is an expansive socio-technical phenomenon, which functions in many contexts and takes many forms. Expanding the discussion also changes how we assess the array of proposed policy solutions meant to improve content moderation. Here, nine content moderation scholars working in critical internet studies propose how to expand research on content moderation, with implications for policy.

In this paper, nine scholars consider how to expand research on Internet platforms’ content moderation. We consider research on encrypted platforms such as WhatsApp; on short-lived but briefly popular platforms; on cloud computing platforms and throughout the infrastructure. We address the implications of content moderation for civic action in open societies, content moderation as regulation, for our political lives, the quality of our cultures and daily lives; and opportunities and for regulatory approaches.

AI seems like the perfect response to the growing challenges of content moderation on social media platforms: the immense scale of the data, the relentlessness of the violations, and the need for human judgments without wanting humans to have to make them. The push toward automated content moderation is often justified as a necessary response to the scale: the enormity of social media platforms like Facebook and YouTube stands as the reason why AI approaches are desirable, even inevitable. But even if we could effectively automate content moderation, it is not clear that we should.

In this interdisciplinary roundtable discussion, five scholars interested in political communication work through the democratic dilemmas created when privately owned social media platforms are used as digital public squares by elected officials in the United States. This conversation unfolds in the context of ongoing legal cases that challenge politicians’ efforts to block select interlocutors and bar them from participation. We grapple with the tension between politicians’ use of social media to broadcast their own messages as a form of publicity with the desire by some members of the public that politicians be transparent online by allowing the electorate to question or even criticize them. Through this discussion, we weigh the importance of the right to criticize the government and its leaders alongside the realities of contentious content on social media platforms that are rife with abusive content, in a cultural context marked by social inequalities.

Social media platforms have profoundly transformed cultural production, in part by restructuring the terms by which culture is distributed and paid for. In this article, we examine the YouTube Partner Program and the controversies around the “demonetization” of videos, to understand these arrangements and what happens when they shift beneath creators’ feet. We use the testimony of YouTubers, provided in their own videos, to understand how creators square the contradiction between YouTube’s increasingly cautious rules regarding “advertiser-friendly” content, its shifting financial and algorithmic incentive structure, and its stated values as an open platform of expression. We examine YouTube’s tiered governance strategy, in which different users are offered different sets of rules, different material resources, and different procedural protections when content is demonetized. And we examine how, especially when the details of that tiered governance are ambiguous or poorly conveyed, creators develop their own theories for why their content has been demonetized—which can provide some creators a tactical opportunity to advance politically motivated accusations of bias against the platform.

A revealing and gripping investigation into how social media platforms police what we post online—and the large societal impact of these decisions Most users want their Twitter feed, Facebook page, and YouTube comments to be free of harassment and porn. Whether faced with “fake news” or livestreamed violence, “content moderators”—who censor or promote user†‘posted content—have never been more important. This is especially true when the tools that social media platforms use to curb trolling, ban hate speech, and censor pornography can also silence the speech you need to hear. In this revealing and nuanced exploration, award†‘winning sociologist and cultural observer Tarleton Gillespie provides an overview of current social media practices and explains the underlying rationales for how, when, and why these policies are enforced. In doing so, Gillespie highlights that content moderation receives too little public scrutiny even as it is shapes social norms and creates consequences for public discourse, cultural production, and the fabric of society. Based on interviews with content moderators, creators, and consumers, this accessible, timely book is a must†‘read for anyone who’s ever clicked “like” or “retweet.”