We Can Detect Your Bias: Predicting the Political Ideology of News Articles

EMNLP 2020, pp.4982-4991, (2020)

被引用1|浏览237
下载 PDF 全文
引用
微博一下

摘要

We explore the task of predicting the leading political ideology or bias of news articles. First, we collect and release a large dataset of 34,737 articles that were manually annotated for political ideology –left, center, or right–, which is well-balanced across both topics and media. We further use a challenging experimental setup where...更多

代码

数据

0
简介
  • In any piece of news, there is a chance that the viewpoint of its authors and of the media organization they work for, would be reflected in the way the story is being told.
  • It could enable news exploration from a left/center/right angle
  • It could be an important building block in a system that detects bias at the level of entire news media (Baly et al, 2018, 2019, 2020), such as the need to offer explainability, i.e., if a website is classified as left-leaning, the system should be able to pinpoint specific articles that support this decision
重点内容
  • In any piece of news, there is a chance that the viewpoint of its authors and of the media organization they work for, would be reflected in the way the story is being told
  • Media bias can come in many different forms, e.g., by omission, by over-reporting on a topic, by cherry-picking the facts, or by using propaganda techniques such as appealing to emotions, prejudices, fears, etc. (Da San Martino et al, 2019, 2020a,b) Bias can occur with respect to a specific topic, e.g., COVID-19, immigration, climate change, gun control, etc. (Darwish et al, 2020; Stefanov et al, 2020) It could be more systematic, as part of a political ideology, which in the Western political system is typically defined as left vs. center vs. right political leaning
  • We show that adversarial media adaptation is quite helpful in that respect, and we further propose to use a triplet loss, which shows sizable improvements over state-of-the-art pretrained Transformers
  • Baseline Results The results in Table 3 show the performance for Long Short-Term Memory networks (LSTMs) and for Bidirectional Encoder Representations from Transformers (BERT) at predicting the political ideology of news articles for both the media-based and the random splits
  • We have explored the task of predicting the leading political ideology of news articles
  • We further proposed an adversarial media adaptation approach, as well as a special triplet loss in order to prevent modeling the source instead of the political bias in the news article, which is a common pitfall for approaches dealing with data that exhibit high correlation between the source of a news article and its class, as is the case with our task here
方法
  • 4.1 Classifiers

    The task of predicting the political ideology of news articles is typically formulated as a classification problem, where the textual content of the articles is encoded into a vector representation that is used to train a classifier to predict one of C classes.
  • The authors' goal is to develop a model that can predict the political ideology of a news article.
  • Most articles published by a given source have the same ideological leaning
  • This might confuse the model and cause it to erroneously associate the output classes with features that characterize entire media outlets.
  • The authors apply two techniques to de-bias the models, i.e., to prevent them from learning the style of a specific news medium rather than predicting the political ideology of the target news article
结果
  • The authors explore the benefits of incorporating information describing the target medium, which can serve as a complementary representation for the article
  • While this seems to be counter-intuitive to what the authors have been proposing in Subsection 4.2, the authors believe that medium-level representation can be valuable when combined with an accurate representation of the article.
  • The authors observe sizable differences in performance between the two splits
  • Both models perform much better when they are trained and evaluated on the random split, whereas they both fail on the mediabased split, where they are tested on articles from media that were not seen during training.
结论
  • The authors have explored the task of predicting the leading political ideology of news articles.
  • The authors created a new large dataset for this task, which features article-level annotations and is well-balanced across topics and media.
  • The authors plan to explore topic-level bias prediction as well as going beyond left-centerright bias.
  • The authors plan to experiment with other languages, and to explore to what extent a model for one language is transferable to another one given that the left-center-right division is not universal and does not align perfectly across countries and cultures, even when staying within the Western political world
表格
  • Table1: Statistics about our dataset
  • Table2: Statistics about our dataset and its two splits: media-based and random
  • Table3: Baseline experiments (without de-biasing or media-level representation) for the two splits
  • Table4: Predicting the medium in which a target news article was published
  • Table5: Impact of de-biasing (adversarial adaptation and triplet loss) on article-level bias detection
  • Table6: Impact of adding media-level representations to the article-level representations (with and without debiasing). Note that the results in rows 3 and 6 are the same for both LSTM and BERT because no articles were involved, and the media-level representations were directly used to train the classifier
Download tables as Excel
相关工作
  • Most existing datasets for predicting the political ideology at the news article level were created by crawling the RSS feeds of news websites with known political bias (Kulkarni et al, 2018), and then projecting the bias label from a website to all articles crawled from it, which is a form of distant supervision. The crawling could be also done using text search APIs rather than RSS feeds (Horne et al, 2019; Gruppi et al, 2020).

    The media-level annotation of political leaning is typically obtained from specialized online platforms, such as News Guard,2 AllSides,3 and Media Bias/Fact Check,4 where highly qualified journalists use carefully designed guidelines to make the judgments.

    2http://www.newsguardtech.com 3http://allsides.com/ 4http://mediabiasfactcheck.com

    As manual annotation at the article level is very time-consuming, requires domain expertise, and it could be also subjective, such annotations are rarely available at the article level. As a result, automating systems for political bias detection have opted for using distant supervision as an easy way to obtain large datasets, which are needed to train contemporary deep learning models.

    Distant supervision is a popular technique for annotating datasets for related text classification tasks, such as detecting hyper-partisanship (Horne et al, 2018; Potthast et al, 2018) and propaganda/satire/hoaxes (Rashkin et al, 2017). For example, Kiesel et al (2019) created a large corpus for detecting hyper-partisanship (i.e., articles with extreme left/right bias) consisting of 754,000 articles, annotated via distant supervision, and additional 1,273 manually annotated articles, part of which was used as a test set for the SemEval-2019 task 4 on Hyper-partisan News Detection. The winning system was an ensemble of character-level CNNs (Jiang et al, 2019). Interestingly, all topperforming systems in the task achieved their best results when training on the manually annotated articles only and ignoring the articles that were labeled using distant supervision, which illustrates the dangers of relying on distant supervision.
研究对象与分析
articles: 34737
We explore the task of predicting the leading political ideology or bias of news articles. First, we collect and release a large dataset of 34,737 articles that were manually annotated for political ideology -left, center, or right-, which is well-balanced across both topics and media. We further use a challenging experimental setup where the test examples come from media that were not seen during training, which prevents the model from learning to detect the source of the target news article instead of predicting its political ideology

articles: 754000
Distant supervision is a popular technique for annotating datasets for related text classification tasks, such as detecting hyper-partisanship (Horne et al, 2018; Potthast et al, 2018) and propaganda/satire/hoaxes (Rashkin et al, 2017). For example, Kiesel et al (2019) created a large corpus for detecting hyper-partisanship (i.e., articles with extreme left/right bias) consisting of 754,000 articles, annotated via distant supervision, and additional 1,273 manually annotated articles, part of which was used as a test set for the SemEval-2019 task 4 on Hyper-partisan News Detection. The winning system was an ensemble of character-level CNNs (Jiang et al, 2019)

articles: 594
Finally, their dataset is not freely available, and their approach of randomly crawling articles does not ensure that topics and events are covered from different political perspectives. Lin et al (2006) built a dataset annotated with the ideology of 594 articles related to the IsraeliPalestinian conflict published on bitterlemons. org. The articles were written by two editors and 200 guests, which minimizes the risk of modeling the author style

articles: 34737
Note that the center class covers articles that are biased towards a centrist political ideology, and not articles that lack political bias (e.g., sports and technology), which commonly exist in news corpora that were built by scraping RSS feeds. We collected a total of 34,737 articles published by 73 news media and covering 109 topics.6. In this dataset, a total of 1,080 individual articles (3.11%) have a political ideology label that is different from their source’s

individual articles: 1080
We collected a total of 34,737 articles published by 73 news media and covering 109 topics.6. In this dataset, a total of 1,080 individual articles (3.11%) have a political ideology label that is different from their source’s. This suggests that, while the distant supervision assumption generally holds, we would still find many articles that defy it

articles: 1200
In order to ensure fair one-toone comparisons between experiments, we created these two different sets of splits, while making sure that they share the same test set, as follows: Figure 2: Political ideology for the most frequent topics: elections, immigration, coronavirus, and politics. • Media-based Split: We sampled 1,200 articles from 12 news media (100 per medium) and used them as the test set, and we excluded the remaining 5,470 articles from these media. Then, we used the articles from the remaining 61 media to create the training and the validation sets, where all articles from the same medium would appear in the same set: training, development, or testing

articles: 5470
• Random Split: Here, the test set is the same as in the media-based split. The 5,470 articles that we excluded from the 12 media are now added to the articles from the 61 remaining media. Then, we split this collection of articles (using stratified random sampling) into training and validation sets

引用论文
  • Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018. Predicting factuality of reporting and bias of news media sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 3528–3539, Brussels, Belgium.
    Google ScholarLocate open access versionFindings
  • Ramy Baly, Georgi Karadzhov, Jisun An, Haewoon Kwak, Yoan Dinkov, Ahmed Ali, James Glass, and Preslav Nakov. 2020. What was written vs. who read it: News media profiling using text analysis and social media context. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL ’20, pages 3364–3374.
    Google ScholarLocate open access versionFindings
  • Ramy Baly, Georgi Karadzhov, Abdelrhman Saleh, James Glass, and Preslav Nakov. 2019. Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news media. In Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’19, pages 2109– 2116, Minneapolis, MN, USA.
    Google ScholarLocate open access versionFindings
  • Alberto Barron-Cedeno, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019. Proppy: Organizing the news based on their propagandistic content. Information Processing & Management, 56(5):1849–1864.
    Google ScholarLocate open access versionFindings
  • Ceren Budak, Sharad Goel, and Justin M Rao. 2016. Fair and balanced? Quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly, 80(S1):250–271.
    Google ScholarLocate open access versionFindings
  • Giovanni Da San Martino, Alberto Barron-Cedeno, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020a. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’20, Barcelona, Spain.
    Google ScholarLocate open access versionFindings
  • Giovanni Da San Martino, Stefano Cresci, Alberto Barron-Cedeno, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020b. A survey on computational propaganda detection. In Proceedings of
    Google ScholarLocate open access versionFindings
  • Giovanni Da San Martino, Seunghak Yu, Alberto Barron-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news articles. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP ’19, pages 5636–5646, Hong Kong, China.
    Google ScholarLocate open access versionFindings
  • Kareem Darwish, Michael Aupetit, Peter Stefanov, and Preslav Nakov. 2020. Unsupervised user stance detection on Twitter. In Proceedings of the International AAAI Conference on Web and Social Media, ICWSM ’20, pages 141–152, Atlanta, GA, USA.
    Google ScholarLocate open access versionFindings
  • Stefano DellaVigna and Ethan Kaplan. 2007. The Fox News effect: Media bias and voting. The Quarterly Journal of Economics, 122(3):1187–1234.
    Google ScholarLocate open access versionFindings
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT ’19, pages 4171–4186, Minneapolis, MN, USA.
    Google ScholarLocate open access versionFindings
  • Yoan Dinkov, Ahmed Ali, Ivan Koychev, and Preslav Nakov. 2019. Predicting the leading political ideology of YouTube channels using acoustic, textual, and metadata information. In Proceedings of the 20th Annual Conference of the International Speech Communication Association, INTERSPEECH ’19, pages 501–505, Graz, Austria.
    Google ScholarLocate open access versionFindings
  • Erick Elejalde, Leo Ferres, and Eelco Herder. 2018. On the nature of real and perceived bias in the mainstream media. PloS one, 13(3):e0193765.
    Google ScholarLocate open access versionFindings
  • Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096–2030.
    Google ScholarLocate open access versionFindings
  • Doris A Graber and Johanna Dunaway. 2017. Mass media and American politics. SAGE Publications.
    Google ScholarFindings
  • Maurıcio Gruppi, Benjamin D. Horne, and Sibel Adalı. 2020. NELA-GT-2019: A large multi-labelled news dataset for the study of misinformation in news articles. arXiv preprint arXiv:2003.08444.
    Findings
  • Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780.
    Google ScholarLocate open access versionFindings
  • Benjamin D. Horne, William Dron, Sara Khedr, and Sibel Adali. 20Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news. In Proceedings of the The Web Conference, WWW ’18, pages 235–238, Lyon, France.
    Google ScholarLocate open access versionFindings
  • Benjamin D Horne, Jeppe Nørregaard, and Sibel Adalı. 20Different spirals of sameness: A study of content sharing in mainstream and alternative media. In Proceedings of the International AAAI Conference on Web and Social Media, ICWSM ’19, pages 257– 266, Munich, Germany.
    Google ScholarLocate open access versionFindings
  • Shanto Iyengar and Kyu S Hahn. 2009. Red media, blue media: Evidence of ideological selectivity in media use. Journal of communication, 59(1):19–39.
    Google ScholarLocate open access versionFindings
  • Ye Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, and Diana Maynard. 2019. Team Bertha von Suttner at SemEval-2019 Task 4: Hyperpartisan news detection using ELMo sentence representation convolutional network. In Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval ’19, pages 840–844, Minneapolis, MN, USA.
    Google ScholarLocate open access versionFindings
  • Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval2019 Task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval ’19, pages 829–839, Minneapolis, Minnesota, USA.
    Google ScholarLocate open access versionFindings
  • Vivek Kulkarni, Junting Ye, Steven Skiena, and William Yang Wang. 2018. Multi-view models for political ideology detection of news articles. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’18, pages 3518–3527, Brussels, Belgium.
    Google ScholarLocate open access versionFindings
  • Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006. Which side are you on? Identifying perspectives at the document and sentence levels. In Proceedings of the Tenth Conference on Computational Natural Language Learning, CoNLL ’06, pages 109–116.
    Google ScholarLocate open access versionFindings
  • Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, ACL ’04, pages 271– 278, Barcelona, Spain.
    Google ScholarLocate open access versionFindings
  • Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP ’14, pages 1532–1543, Doha, Qatar.
    Google ScholarLocate open access versionFindings
  • Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL ’18, pages 231–240, Melbourne, Australia.
    Google ScholarLocate open access versionFindings
  • Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, Yejin Choi, and Paul G Allen. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP ’17, pages 2931– 2937, Copenhagen, Denmark.
    Google ScholarLocate open access versionFindings
  • Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP ’19, pages 3973–3983, Hong Kong, China.
    Google ScholarLocate open access versionFindings
  • Diego Saez-Trumper, Carlos Castillo, and Mounia Lalmas. 2013. Social media news communities: Gatekeeping, coverage, and statement bias. In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, CIKM ’13, page 1679–1684, San Francisco, CA, USA.
    Google ScholarLocate open access versionFindings
  • Abdelrhman Saleh, Ramy Baly, Alberto BarronCedeno, Giovanni Da San Martino, Mitra Mohtarami, Preslav Nakov, and James Glass. 2019. Team QCRI-MIT at SemEval-2019 Task 4: Propaganda analysis meets hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval ’19, pages 1041– 1046, Minneapolis, MN, USA.
    Google ScholarLocate open access versionFindings
  • Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’15, pages 815–823, Boston, MA, USA.
    Google ScholarLocate open access versionFindings
  • Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013. Measuring ideological proportions in political speeches. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP ’13, pages 91–101, Seattle, Washington, USA.
    Google ScholarLocate open access versionFindings
  • Peter Stefanov, Kareem Darwish, Atanas Atanasov, and Preslav Nakov. 2020. Predicting the topical stance and political leaning of media using tweets. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL ’20, pages 527–537.
    Google ScholarLocate open access versionFindings
  • Yifan Zhang, Giovanni Da San Martino, Alberto Barron-Cedeno, Salvatore Romeo, Jisun An, Haewoon Kwak, Todor Staykovski, Israa Jaradat, Georgi Karadzhov, Ramy Baly, Kareem Darwish, James Glass, and Preslav Nakov. 2019. Tanbih: Get to know what you are reading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP ’19, pages 223–228, Hong Kong, China.
    Google ScholarLocate open access versionFindings
您的评分 :
0

 

标签
评论
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科