AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We describe the problem of pretrained word embeddings conflating denotation and connotation

Do “Undocumented Workers” == “Illegal Aliens”? Differentiating Denotation and Connotation in Vector Spaces

EMNLP 2020, (2020)

Cited by: 0|Views13
Full Text
Bibtex
Weibo

Abstract

In politics, neologisms are frequently invented for partisan objectives. For example, “undocumented workers” and “illegal aliens” refer to the same group of people (i.e., they have the same denotation), but they carry clearly different connotations. Examples like these have traditionally posed a challenge to referencebased semantic theori...More

Code:

Data:

0
Introduction
  • Language carries information through both denotation and connotation. For example, a reporter writing an article about the leftmost wing of the Democratic party can choose to refer to the group as “progressives” or as “radicals”.
  • The word choice does not change the individuals referred to, but it does communicate significantly different sentiments about the policy positions discussed
  • This type of linguistic nuance presents a significant challenge for natural language processing systems, most of which fundamentally assume words to have similar meanings if they are surrounded in similar shtoorrrieosr tafuxnpdaeyder totalitarian goverrunnment afhfoeradlathble inpsrougraranmce spianygeler pooblaicmieas awmeaelrtihciaensst tdroillliloanr freedomworks stimbiulllus specnudtsing burydagnet word contexts.
  • A right-leaning query like “taxpayer-funded healthcare” could make one likely to see articles about “totalitarian” and “horror stories” than about “affordable healthcare”
Highlights
  • Language carries information through both denotation and connotation
  • We evaluate on words sampled in three different ways: Random is a random sample of 500 words drawn from each corpus’ vocabulary that occur at least 100 times in order to filter out web scraping artifacts, e.g., URLs and author bylines
  • We see that the magnitude of change is greater across the board for the highly partisan words than for random words, which is expected as the highly partisan words are usually loaded with more denotation or connotation information that can be manipulated
  • This is understandable because Congressional Record (CR) PROXY is not trained with ground truth denotation labels. (We evaluate it with the labels from CR BILL)
  • We describe the problem of pretrained word embeddings conflating denotation and connotation
  • This threshold is chosen based on manual inspection, but we have evaluated on other thresholds as well with no significant difference in results
  • We show that our decomposed spaces are capable of improving the diversity of document rankings in an information retrieval task
Results
  • The authors see that the Vdeno and Vconno spaces demonstrate the desired shift in homogeneities and structures, which is intuitively illustrated by Figure 4.
  • For Vdeno, the authors see that denotation homogeneity hdeno consistently increases and con-.
  • The only exception is CR PROXY’s Vdeno, which sees no significant movement in either direction.
  • This is understandable because CR PROXY is not trained with ground truth denotation labels.
  • The authors report the overall distribution of political leanings among the top 100 documents and the rank-weighted αnDCG (Clarke et al, 2008) diversity score.
Conclusion
  • The authors describe the problem of pretrained word embeddings conflating denotation and connotation.
  • The authors address this issue by introducing an adversarial network that explicitly represents the two properties as two different vector spaces.
  • The authors confirm that the decomposed spaces encode the desired structure of denotation or connotation by both quantitatively measuring their homogeneity and qualitatively evaluating their clusters and their representation of well-known political euphemisms.
  • The authors show that the decomposed spaces are capable of improving the diversity of document rankings in an information retrieval task
Summary
  • Introduction:

    Language carries information through both denotation and connotation. For example, a reporter writing an article about the leftmost wing of the Democratic party can choose to refer to the group as “progressives” or as “radicals”.
  • The word choice does not change the individuals referred to, but it does communicate significantly different sentiments about the policy positions discussed
  • This type of linguistic nuance presents a significant challenge for natural language processing systems, most of which fundamentally assume words to have similar meanings if they are surrounded in similar shtoorrrieosr tafuxnpdaeyder totalitarian goverrunnment afhfoeradlathble inpsrougraranmce spianygeler pooblaicmieas awmeaelrtihciaensst tdroillliloanr freedomworks stimbiulllus specnudtsing burydagnet word contexts.
  • A right-leaning query like “taxpayer-funded healthcare” could make one likely to see articles about “totalitarian” and “horror stories” than about “affordable healthcare”
  • Results:

    The authors see that the Vdeno and Vconno spaces demonstrate the desired shift in homogeneities and structures, which is intuitively illustrated by Figure 4.
  • For Vdeno, the authors see that denotation homogeneity hdeno consistently increases and con-.
  • The only exception is CR PROXY’s Vdeno, which sees no significant movement in either direction.
  • This is understandable because CR PROXY is not trained with ground truth denotation labels.
  • The authors report the overall distribution of political leanings among the top 100 documents and the rank-weighted αnDCG (Clarke et al, 2008) diversity score.
  • Conclusion:

    The authors describe the problem of pretrained word embeddings conflating denotation and connotation.
  • The authors address this issue by introducing an adversarial network that explicitly represents the two properties as two different vector spaces.
  • The authors confirm that the decomposed spaces encode the desired structure of denotation or connotation by both quantitatively measuring their homogeneity and qualitatively evaluating their clusters and their representation of well-known political euphemisms.
  • The authors show that the decomposed spaces are capable of improving the diversity of document rankings in an information retrieval task
Tables
  • Table1: Summary of model variants experimented
  • Table2: Intrinsic evaluation results across models and test sets. ∆ is change relative to Vpretrained (Table 3). Arrows in parentheses mark the desired directions of change. Note that because denotation labels have far more classes than connotation labels, the magnitude of hdeno and hconno are not directly comparable with each other
  • Table3: Baseline homogeneity scores of embeddings pretrained on each corpus
  • Table4: Changes in cosine similarity (relative to Vpretrained) for known political euphemism’ pairs, i.e. words with the same denotation but opposite partisan connotation. Omitted entries are out of vocabulary
  • Table5: Example right- and left-leaning queries generated using the procedure described
  • Table6: Retrieval metrics. For α-nDCG, higher means more diverse; for Gini, lower means more diverse
  • Table7: Sample words from each of our test sets as described in §5.2
  • Table8: Corpus with regular expression search for bill titles
  • Table9: Example bill topics
  • Table10: CR TOPIC
  • Table11: Seven random samples of bill mentions from the 111th Congress. Speeches truncated to fit the table
Download tables as Excel
Related work
  • Embedding Augmentation. At the lexical level, there is substantial literature that supplements pretrained representations with desired information (Faruqui et al, 2015; Bamman et al, 2014) or improves their interpretability (Murphy et al, 2012; Arora et al, 2018; Lauretig, 2019). However, existing works tend to focus on evaluating the dictionary definitions of words, less so on grounding words to specific real world referents and, to our knowledge, no major attempt yet in interpreting and manipulating the denotation and connotation dimensions of meaning as suggested by the semantic theories discussed in §2. While we do not claim to do full justice to conceptual role semantics either, this paper furnishes a first attempt at implementing a school of semantics introduced by philosophers of language and increasingly popular among cognitive scientists.

    Style Transfer. At the sentence level, adversarial setups similar to ours have been previously explored for differentiating style and content. For example, Romanov et al (2019); Yamshchikov et al (2019); John et al (2019) converted informal English to formal English and Yelp reviews from positive to negative sentiment. The motivation for such models is primarily natural language generation and the personalization thereof (Li et al, 2016). Additionally, our framing in terms of Frege’s sense and reference adds clarity to the sometimes illdefined problems explored in style transfer (e.g., treating sentiment as “style”). For example, “she is an undocumented immigrant” and “she is an illegal alien” have the same truth conditions but different connotations, whereas “the cafe is great” and “the cafe is terrible” have different truth conditions.
Funding
  • This research was supported by the Google Faculty Research Awards Program
Study subjects and analysis
most relevant documents: 5000
6.2 Models. We generate a ranked list of documents for each query in a two-step manner: (1) We pre-select the 5,000 most relevant documents according to a traditional BM25 model (Robertson et al, 1995) with default parameters. (2) This initial set of documents is then ranked using DRMM (Guo et al, 2016), a neural relevance matching model for adhoc retrieval

most relevant documents: 100
While (1) is purely based on tf-idf style statistics and remains static for all compared conditions, (2) is repeated for every proposed word embedding. This results in a ranked list of the top 100 most relevant documents for each query and word embedding. 6.3 Results

documents: 100
We compare the results of the DRMM retrieval model using different word embeddings in terms of quality and diversity of viewpoints reflected in the ranked results. To measure diversity, we report the overall distribution of political leanings among the top 100 documents and the rank-weighted αnDCG (Clarke et al, 2008) diversity score. For α-nDCG, higher values indicate a more diverse list of results whose political leanings are evenly distributed across result list ranks

random samples: 7
CR TOPIC. Seven random samples of bill mentions from the 111th Congress. Speeches truncated to fit the table. Nearest neighbors of government-run healthcare (triangles) and economic stimulus (circles). Note that words cluster as strongly by policy denotation (shapes) as by partisan connotation (colors); namely, pretrained representations conflate denotation with connotation. Plotted by t-SNE with perplexity = 10

documents: 100
Neighborhood of “deficit” in Vpretained, Vdeno, and Vconno of PN PROXY. Arrows point to the top-10 nearest neighbors. Colors reflect partisan leaning, where more opaque dots are more heavily partisan words. Note that in Vpretained and in Vconno, the nearest neighbors are all Republican-leaning words, whereas they are balanced in Vdeno. Distribution of partisanship of news source for top 100 documents for right-leaning and leftleaning queries. Red = right-leaning news sources; blue = left-leaning; gray = nonpartisan or apolitical.

Reference
  • Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2018. Linear algebraic structure of word senses, with applications to polysemy.
    Google ScholarFindings
  • Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
    Findings
  • David Bamman, Chris Dyer, and Noah A. Smith. 2014. Distributed representations of geographically situated language. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 828– 834, Baltimore, Maryland. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Eric Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, and Geri Gay. 2015. Testing and comparing computational approaches for identifying the language of framing in political news. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1472–1482, Denver, Colorado. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ned Block. 1986. Advertisement for a semantics for psychology. Midwest studies in philosophy, 10:615– 678.
    Google ScholarLocate open access versionFindings
  • Susan Carey. 2009. The Origin of Concepts. Oxford series in cognitive development. Oxford University Press.
    Google ScholarFindings
  • Eunsol Choi, Chenhao Tan, Lillian Lee, Cristian Danescu-Niculescu-Mizil, and Jennifer Spindel. 2012. Hedge detection as a lens on framing in the GMO debates: A position paper. In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 70–79, Jeju, Republic of Korea. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Charles LA Clarke, Maheedhar Kolla, Gordon V Cormack, Olga Vechtomova, Azin Ashkan, Stefan Buttcher, and Ian MacKinnon. 200Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 659–666.
    Google ScholarLocate open access versionFindings
  • Shiri Dori-Hacohen, Elad Yom-Tov, and James Allan. 2015. Navigating controversy as a complex search task. In SCST@ ECIR. Citeseer.
    Google ScholarLocate open access versionFindings
  • Nikita Duseja and Harsh Jhamtani. 2019. A sociolinguistic study of online echo chambers on twitter. In Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science, pages 78–83, Minneapolis, Minnesota. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1606–1615, Denver, Colorado. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Gottlob Frege. 1892. Uber sinn und bedeutung. Zeitschrift fur Philosophie und philosophische Kritik, 100:25–50.
    Google ScholarLocate open access versionFindings
  • Dean Fulgoni, Jordan Carpenter, Lyle Ungar, and Daniel Preotiuc-Pietro. 2016. An empirical exploration of moral foundations theory in partisan news sources. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3730–3736, Portoroz, Slovenia. European Language Resources Association (ELRA).
    Google ScholarLocate open access versionFindings
  • Luca Gasparri and Diego Marconi. 2019. Word Meaning. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, fall 2019 edition. Metaphysics Research Lab, Stanford University.
    Google ScholarLocate open access versionFindings
  • Matthew Gentzkow, Jesse M Shapiro, and Matt Taddy. 2019. Measuring group differences in highdimensional choices: method and application to congressional speech. Econometrica, 87(4):1307– 1340.
    Google ScholarLocate open access versionFindings
  • Goran Glavas, Federico Nanni, and Simone Paolo Ponzetto. 2019. Computational analysis of political texts: Bridging research efforts across communities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 18–23, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Mark Greenberg and Gilbert Harman. 2005. Conceptual role semantics.
    Google ScholarFindings
  • Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 503–511, Boulder, Colorado. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 55–64.
    Google ScholarLocate open access versionFindings
  • Jiyoung Han, Youngin Lee, Junbum Lee, and Meeyoung Cha. 2019. The fallacy of echo chambers: Analyzing the political slants of user-generated news comments in Korean media. In Proceedings of the 5th Workshop on Noisy User-generated Text (WNUT 2019), pages 370–374, Hong Kong, China. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424–434, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829–839, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Ezra Klein. 2020. Why We’re Polarized. Profile Books.
    Google ScholarFindings
  • Saul A Kripke. 1972. Naming and necessity. In Semantics of natural language, pages 253–355. Springer.
    Google ScholarLocate open access versionFindings
  • Adam Lauretig. 2019.
    Google ScholarFindings
  • Identification, interpretability, and Bayesian word embeddings. In Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science, pages 7–17, Minneapolis, Minnesota. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Frances E Lee. 2009. Beyond ideology: Politics, principles, and partisanship in the US Senate. University of Chicago Press.
    Google ScholarFindings
  • Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994–1003, Berlin, Germany. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Gabriel Machado Lunardi. 2019. Representing the filter bubble: Towards a model to diversification in news. In International Conference on Conceptual Modeling, pages 239–246. Springer.
    Google ScholarLocate open access versionFindings
  • Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc.
    Google ScholarLocate open access versionFindings
  • Brian Murphy, Partha Talukdar, and Tom Mitchell. 2012. Learning effective and interpretable semantic models using non-negative sparse embedding. In Proceedings of COLING 2012, pages 1933–1950, Mumbai, India. The COLING 2012 Organizing Committee.
    Google ScholarLocate open access versionFindings
  • Sebastian Pado, Andre Blessing, Nico Blokker, Erenay Dayanik, Sebastian Haunss, and Jonas Kuhn. 2019. Who sides with whom? towards computational construction of discourse networks for political debates. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2841–2847, Florence, Italy. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018. A stylometric inquiry into hyperpartisan and fake news. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 231–240, Melbourne, Australia. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Daniel Preotiuc-Pietro, Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017. Beyond binary labels: Political ideology prediction of twitter users. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729–740, Vancouver, Canada. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Cornelius Puschmann. 2019. Beyond the bubble: Assessing the diversity of political search results. Digital Journalism, 7(6):824–843.
    Google ScholarLocate open access versionFindings
  • Hilary Putnam. 1975. The meaning of ‘meaning’. Philosophical papers, 2.
    Google ScholarLocate open access versionFindings
  • Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
    Google ScholarLocate open access versionFindings
  • Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931–2937, Copenhagen, Denmark. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
    Google ScholarFindings
  • Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. 2019. Adversarial decomposition of text representation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 815–825, Minneapolis, Minnesota. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013. Measuring ideological proportions in political speeches. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 91–101, Seattle, Washington, USA. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • Shayan A Tabrizi and Azadeh Shakery. 2019. Perspective-based search: a new paradigm for bursting the information bubble. FACETS, 4(1):350–388.
    Google ScholarLocate open access versionFindings
  • Nava Tintarev, Emily Sullivan, Dror Guldin, Sihang Qiu, and Daan Odjik. 2018.
    Google ScholarFindings
  • Ivan P. Yamshchikov, Viacheslav Shibaev, Aleksander Nagaev, Jurgen Jost, and Alexey Tikhonov. 2019. Decomposing textual information for style transfer. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 128–137, Hong Kong. Association for Computational Linguistics.
    Google ScholarLocate open access versionFindings
  • The skip-gram loss follows the parameters recommended by Mikolov et al. (2013). Context window radius = 5. Negative samples per true context word = 10. We also subsample frequent words in exactly the same way as the original paper (equation 5) did with their threshold of 10−5.
    Google ScholarLocate open access versionFindings
  • We use Stanford Stanza (Qi et al., 2020) for tokenization, part-of-speech tag, dependency parsing, and named entity recognition. We replace multiword phrases with an atomic token. We source our phrases of interests from the following three pipelines: 1. Named entity recognizer.
    Google ScholarFindings
  • 2. Frequency-based collocation. (We experimented with PMI-based collocation, which yielded results that were more prone to artifacts and arbitrary threshold setting.)
    Google ScholarFindings
  • 3. Bigram and trigram constituents of parse trees that are (a) POS-tagged as noun phrase or verb phrase; (b) contain no stop words as in nltk.corpus.stop words; (c) contain no parliamentary procedural words as in {“yield”, “motion”, “order”, “ordered”, “quorum”, “roll”, “unanimous”, “mr.”, “madam”, “speaker”, “chairman”, “president”, “senator”, “gentleman”, “colleague”, “colleagues”}
    Google ScholarFindings
Author
Albert Webson
Albert Webson
Zhizhong Chen
Zhizhong Chen
Carsten Eickhoff
Carsten Eickhoff
Ellie Pavlick
Ellie Pavlick
Your rating :
0

 

Tags
Comments
小科