Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes

Nicholas Lourie
Nicholas Lourie
Cited by: 0|Bibtex|Views30
Other Links: arxiv.org
Weibo:
One explanation is that unlike a traditional softmax layer trained on hard labels, the Dirichlet likelihood leverages all annotations without the need for a majority vote

Abstract:

As AI systems become an increasing part of people's everyday lives, it becomes ever more important that they understand people's ethical norms. Motivated by descriptive ethics, a field of study that focuses on people's descriptive judgments rather than theoretical prescriptions on morality, we investigate a novel, data-driven approach t...More

Code:

Data:

0
Introduction
  • State-of-the-art techniques excel at syntactic and semantic understanding of text, reaching or even exceeding human performance on major language understanding benchmarks (Devlin et al 2019; Lan et al 2019; Raffel et al 2019).
  • Reading between the lines with pragmatic understanding of text still remains a major challenge, as it requires understanding social, cultural, and ethical implications underlying the text.
  • People read not just what is stated literally and explicitly, and the rich non-literal implications based on social, cultural, and moral conventions.
  • Closing the door in a salespersons face?
  • The other day a salespersons knocked on the door and it was obvious he was about to sell something.
Highlights
  • State-of-the-art techniques excel at syntactic and semantic understanding of text, reaching or even exceeding human performance on major language understanding benchmarks (Devlin et al 2019; Lan et al 2019; Raffel et al 2019)
  • Developing machine ethics poses major open research questions, ranging from how to define and represent abstract moral concepts to how to design computational models that make concrete moral judgments on complex real-world situations described in language. Our work investigates the latter, drawing inspiration from descriptive ethics, the field of study that focuses on people’s descriptive judgements, in contrast to prescriptive ethics which focuses on theoretical prescriptions on morality (Gert and Gert 2017)
  • As a first step toward computational models of descriptive ethical judgments in language, we present a study based on people’s diverse ethical judgements over a wide spectrum of social situations shared in online communities
  • One explanation is that unlike a traditional softmax layer trained on hard labels, the Dirichlet likelihood leverages all annotations without the need for a majority vote
  • We introduce a new task: WHO’S IN THE WRONG?, and a dataset, SCRUPLES, to study it
  • With Dirichlet-multinomial layers fully utilizing all annotations, rather than just the majority vote, we’re able to improve the performance of current techniques
Methods
  • The authors found communal judgments on real-life anecdotes reflect this fact in that some situations are clean-cut, while others can be divisive.
  • This inherent subjectivity in people’s judgements is an important facet of human intelligence, and it raises unique technical challenges compared to tasks that can be defined as clean-cut categorization, as many existing NLP tasks are often framed.
  • The authors compare to an oracle classifier and present a novel Bayesian estimator for its score, called the BEST performance, available at https://scoracle.apps.allenai.org
Results
  • Following the goal to model norms’ distribution, the authors compare models with cross-entropy.
  • RoBERTa with a Dirichlet likelihood (RoBERTa + Dirichlet) outperforms all other models on both the ANECDOTES and the DILEMMAS.
  • One explanation is that unlike a traditional softmax layer trained on hard labels, the Dirichlet likelihood leverages all annotations without the need for a majority vote
  • It can separate the subjectivity of the question from the model’s uncertainty, making the predictions more expressive.
  • You can demo the model at https://norms.apps.allenai.org
Conclusion
  • SCRUPLES provides simple ethical dilemmas that enable models to learn basic ethical understanding as well as complex anecdotes that challenge existing models.
  • With Dirichlet-multinomial layers fully utilizing all annotations, rather than just the majority vote, the authors are able to improve the performance of current techniques.
  • These layers separate model uncertainty from norms’ controversiality.
Summary
  • Introduction:

    State-of-the-art techniques excel at syntactic and semantic understanding of text, reaching or even exceeding human performance on major language understanding benchmarks (Devlin et al 2019; Lan et al 2019; Raffel et al 2019).
  • Reading between the lines with pragmatic understanding of text still remains a major challenge, as it requires understanding social, cultural, and ethical implications underlying the text.
  • People read not just what is stated literally and explicitly, and the rich non-literal implications based on social, cultural, and moral conventions.
  • Closing the door in a salespersons face?
  • The other day a salespersons knocked on the door and it was obvious he was about to sell something.
  • Methods:

    The authors found communal judgments on real-life anecdotes reflect this fact in that some situations are clean-cut, while others can be divisive.
  • This inherent subjectivity in people’s judgements is an important facet of human intelligence, and it raises unique technical challenges compared to tasks that can be defined as clean-cut categorization, as many existing NLP tasks are often framed.
  • The authors compare to an oracle classifier and present a novel Bayesian estimator for its score, called the BEST performance, available at https://scoracle.apps.allenai.org
  • Results:

    Following the goal to model norms’ distribution, the authors compare models with cross-entropy.
  • RoBERTa with a Dirichlet likelihood (RoBERTa + Dirichlet) outperforms all other models on both the ANECDOTES and the DILEMMAS.
  • One explanation is that unlike a traditional softmax layer trained on hard labels, the Dirichlet likelihood leverages all annotations without the need for a majority vote
  • It can separate the subjectivity of the question from the model’s uncertainty, making the predictions more expressive.
  • You can demo the model at https://norms.apps.allenai.org
  • Conclusion:

    SCRUPLES provides simple ethical dilemmas that enable models to learn basic ethical understanding as well as complex anecdotes that challenge existing models.
  • With Dirichlet-multinomial layers fully utilizing all annotations, rather than just the majority vote, the authors are able to improve the performance of current techniques.
  • These layers separate model uncertainty from norms’ controversiality.
Tables
  • Table1: BEST’s relative error when estimating the oracle score in simulations. Anecdotes simulates the ANECDOTES, 3 Annotators simulates 3 annotators per example, and Mixed Prior simulates a Dirichlet mixture as the true prior
  • Table2: Comparison of objective vs. subjective tasks
  • Table3: Dataset statistics for the ANECDOTES. Tokens combine stories’ titles and texts. Token types count distinct items
  • Table4: Instance statistics for the ANECDOTES’ dev set. Columns are percentiles. TTR is the token-type ratio
  • Table5: Label descriptions and frequencies from dev. Frequencies tally individual judgments (not the majority vote)
  • Table6: Baselines for ANECDOTES. Best scores are in bold. Calibration smooths models worse than the uniform distribution to it, giving a cross-entropy of 1.609
  • Table7: Baselines for DILEMMAS. Best scores are bold
  • Table8: Likelihood comparisons on the ANECDOTES (dev). Soft uses cross-entropy on soft labels, Counts uses crossentropy on label counts, and Dirichlet uses a Dirichletmultinomial layer. Best scores are in bold
  • Table9: Likelihood comparisons on the DILEMMAS (dev). Soft uses cross-entropy on soft labels, Counts uses crossentropy on label counts, and Dirichlet uses a Dirichletmultinomial layer. Best scores are in bold
  • Table10: Verbs associated with the more or less ethical choice from DILEMMAS. LR is the likelihood ratio, Better and Worse are the numbers of times the verb was the better or worse action, and Total is their sum
  • Table11: Top 5 words for the DILEMMAS’ topics (train), learned through LDA (<a class="ref-link" id="cBlei_et+al_2003_a" href="#rBlei_et+al_2003_a">Blei, Ng, and Jordan 2003</a>)
  • Table12: Filtering metrics. Spam is the negative class. The accuracy on comments and posts is 95% and 99%
  • Table13: Metrics for extracting labels from comments and post types from post titles
  • Table14: Verbs significantly associated with more or less ethical choices from the DILEMMAS (train). p is the p-value, LR is the likelihood ratio, Better and Worse count the times the verb was a better or worse action, and Total is their sum
  • Table15: Top 25 words for the DILEMMAS’ topics (train), learned through LDA (<a class="ref-link" id="cBlei_et+al_2003_a" href="#rBlei_et+al_2003_a">Blei, Ng, and Jordan 2003</a>)
Download tables as Excel
Related work
  • From science to science-fiction, people have long acknowledged the need to align AI with human interests. In his 1942 short story, “Runaround”, Isaac Asimov famously coined three laws of robotics to safeguard human welfare (Asimov 1942). Beyond fiction, pioneers of modern computing expressed similar concerns. Early on, I.J. Good raised the possibility of an “intelligence explosion” and the great benefits, as well as dangers, such an event may pose (Good 1966). In the intervening years, many researchers cautioned about super-intelligence and the need for AI to understand ethics (Vinge 1993; Weld and Etzioni 1994).
Funding
  • We thank Mark Neumann, Maxwell Forbes, Hannah Rashkin, Doug Downey, and Oren Etzioni for their helpful feedback and suggestions while we developed this work. This research was supported in part by NSF (IIS1714566), DARPA under the CwC program through the ARO (W911NF-15-1-0543), and DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031). A Estimating Oracle Performance As discussed in Section 2.1, given class label counts, Yi, for each instance, we can model them using a DirichletMultinomial distribution: θi ∼ Dirichlet(α) Yi ∼ Multinomial(θi, Ni) where Ni is the fixed (or random but independent) number of annotations for example i. First, we estimate α by minimizing the Dirichletmultinomial’s negative log-likelihood, marginalizing out the θi’s: − log Γ(Ni)Γ( k αk) Γ(Yik + αk) i Γ(Ni + k αk) k Γ(Yik)Γ(αk) Pushing the log inside the products leaves only log-gamma terms
Study subjects and analysis
simulation studies: 3
See Appendix A for the mathematical details. Simulation Experiments To validate BEST, we ran three simulation studies comparing it’s estimate and the true oracle score. First, we simulated the ANECDOTES’ label distribution using a Dirichlet prior learned from the data (Anecdotes)

posts: 625
the label and scores from the comments. To evaluate the extraction, we sampled and manually annotated 625 posts and 625 comments. Comments and posts were filtered with an F1 of 97% and 99%, while label extraction had an average F1 of 92% over the five classes

tweets: 35000
One body of work draws on moral foundations theory (Haidt and Joseph 2004; Haidt 2012; Graham et al 2013), a psychological theory explaining ethical differences in terms of how people weigh a small number of moral foundations (e.g. care/harm, fairness/cheating, etc.). The Moral Foundation Twitter Corpus (MFTC) collects more than 35,000 tweets labeled with the foundations they express, and has been used for supervised moral sentiment prediction (Hoover et al 2020). Similarly, lexicons annotating words’ alignment with moral foundations aid in analyzing things like political groups’ moral sensitivies (Graham, Haidt, and Nosek 2009) as well as predicting the foundations expressed in social media posts (Araque, Gatti, and Kalimeri 2019)

posts: 625
Each example in the ANECDOTES derives from a reddit post and its comments. First, we filtered and extracted content from a data dump21 using rules; then, we evaluated the extraction by manually annotating a random sample of 625 posts and 625 comments. Tables 12 and 13 report these results

samples: 100000
P (word|more ethical). We permuted the class labels to simulate the independent condition and tested association at the 0.05 level of significance using 100,000 samples in our Monte Carlo estimates for the permutation distribution. Some of the p-values were computed as zero due to the likelihood ratio in the original data being higher than in any of the sampled permutations

Reference
  • Abel, D.; MacGlashan, J.; and Littman, M. L. 2016. Reinforcement learning as a framework for ethical decision making. In Workshops at the thirtieth AAAI conference on artificial intelligence.
    Google ScholarLocate open access versionFindings
  • Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schulman, J.; and Mane, D. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565.
    Findings
  • Anderson, M.; and Anderson, S. L. 2007. Machine Ethics: Creating an Ethical Intelligent Agent. AI Magazine 28(4): 15. doi: 10.1609/aimag.v28i4.2065. URL https://www.aaai.org/ojs/index.php/aimagazine/article/view/2065.
    Locate open access versionFindings
  • Anderson, M.; and Anderson, S. L., eds. 2011. Machine Ethics. Cambridge University Press. doi:10.1017/CBO9780511978036.
    Findings
  • Anderson, M.; Anderson, S. L.; and Armen, C. 200Machine ethics: papers from the AAAI Fall Symposium.
    Google ScholarFindings
  • Araque, O.; Gatti, L.; and Kalimeri, K. 2019. MoralStrength: Exploiting a Moral Lexicon and Embedding Similarity for Moral Foundations Prediction. Knowl. Based Syst. 191: 105184.
    Google ScholarLocate open access versionFindings
  • Asimov, I. 1942. Runaround. Astounding Science-Fiction 94–103.
    Google ScholarFindings
  • Balakrishnan, A.; Bouneffouf, D.; Mattei, N.; and Rossi, F. 2019. Incorporating Behavioral Constraints in Online AI Systems. In AAAI.
    Google ScholarFindings
  • Blei, D. M.; Ng, A. Y.; and Jordan, M. I. 2003. Latent dirichlet allocation. Journal of machine Learning research 3(Jan): 993–1022.
    Google ScholarLocate open access versionFindings
  • Bolukbasi, T.; Chang, K.-W.; Zou, J. Y.; Saligrama, V.; and Kalai, A. T. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, 4349–4357.
    Google ScholarLocate open access versionFindings
  • Breiman, L. 2001. Random Forests. Machine Learning 45: 5–32.
    Google ScholarLocate open access versionFindings
  • Bringsjord, S.; Arkoudas, K.; and Bello, P. 2006. Toward a General Logicist Methodology for Engineering Ethically Correct Robots. IEEE Intelligent Systems 21: 38–44.
    Google ScholarLocate open access versionFindings
  • Chen, T.; and Guestrin, C. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, 785–794. New York, NY, USA: ACM. ISBN 978-14503-4232-2. doi:10.1145/2939672.2939785. URL http://doi.acm.org/10.1145/2939672.2939785.
    Locate open access versionFindings
  • Chevalier, C.; and Ginsbourger, D. 2013. Fast Computation of the Multi-Points Expected Improvement with Applications in Batch Selection. In Revised Selected Papers of the 7th International Conference on Learning and Intelligent Optimization - Volume 7997, LION 7, 59–69. New York, NY, USA: Springer-Verlag New York, Inc. ISBN 978-3-642-44972-7. doi:10.1007/978-3-642-449734
    Locate open access versionFindings
  • 7. URL http://dx.doi.org/10.1007/978-3-642-44973-4
    Findings
  • 7. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics. doi:10.18653/v1/N19-1423. URL https://www.aclweb.org/anthology/N19-1423.
    Locate open access versionFindings
  • Frazier, S.; Nahian, M. S. A.; Riedl, M. O.; and Harrison, B. 2020. Learning Norms from Stories: A Prior for Value Aligned Agents. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society.
    Google ScholarLocate open access versionFindings
  • Gal, Y. 2016. Uncertainty in Deep Learning. Ph.D. thesis, University of Cambridge.
    Google ScholarFindings
  • Gal, Y.; and Ghahramani, Z. 2016. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML-16).
    Google ScholarLocate open access versionFindings
  • Galbraith, J.; Moustaki, I.; Bartholomew, D.; and Steele, F. 2002. The Analysis and Interpretation of Multivariate Data for Social Scientists. Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences. Taylor & Francis. ISBN 9781584882954.
    Google ScholarLocate open access versionFindings
  • Gelman, A.; Carlin, J.; Stern, H.; and Rubin, D. 2003. Bayesian Data Analysis, Second Edition. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis. ISBN 9781420057294.
    Google ScholarLocate open access versionFindings
  • Gert, B.; and Gert, J. 2017. The Definition of Morality. In Zalta, E. N., ed., The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, fall 2017 edition.
    Google ScholarLocate open access versionFindings
  • Good, I. J. 1966. Speculations Concerning the First Ultraintelligent Machine. volume 6 of Advances in Computers, 31 – 88. Elsevier. doi:https://doi.org/10.1016/S0065-2458(08)60418-0.
    Locate open access versionFindings
  • Gordon, J.; and Van Durme, B. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 workshop on Automated knowledge base construction, 25–30. ACM.
    Google ScholarLocate open access versionFindings
  • Graham, J.; Haidt, J.; Koleva, S.; Motyl, M.; Iyer, R.; Wojcik, S. P.; and Ditto, P. H. 2013. Moral foundations theory: The pragmatic validity of moral pluralism. In Advances in experimental social psychology, volume 47, 55–130. Elsevier.
    Google ScholarLocate open access versionFindings
  • Graham, J.; Haidt, J.; and Nosek, B. A. 2009. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology 96(5): 1029–46.
    Google ScholarLocate open access versionFindings
  • Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, 1321– 1330. JMLR.org.
    Google ScholarLocate open access versionFindings
  • Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S.; and Smith, N. A. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 107–112. New Orleans, Louisiana: Association for Computational Linguistics. doi:10.18653/v1/N18-2017. URL https://www.aclweb.org/anthology/N18-2017.
    Locate open access versionFindings
  • Haidt, J. 2012. The Righteous Mind: Why Good People Are Divided by Politics and Religion. Pantheon Books.
    Google ScholarFindings
  • Haidt, J.; and Joseph, C. 2004. Intuitive Ethics: How Innately Prepared Intuitions Generate Culturally Variable Virtues. Daedalus 133(4): 55–66. ISSN 00115266.
    Google ScholarLocate open access versionFindings
  • Head, T.; MechCoder; Louppe, G.; Shcherbatyi, I.; fcharras; Vinıcius, Z.; cmmalone; Schroder, C.; nel215; Campos, N.; Young, T.; Cereda, S.; Fan, T.; rene rex; Shi, K. K.; Schwabedal, J.; carlosdanielcsantos; Hvass-Labs; Pak, M.; SoManyUsernamesTaken; Callaway, F.; Esteve, L.; Besson, L.; Cherti, M.; Pfannschmidt, K.; Linzberger, F.; Cauet, C.; Gut, A.; Mueller, A.; and Fabisch, A. 2018. scikit-optimize/scikit-optimize: v0.5.2. doi:10.5281/zenodo. 1207017. URL https://doi.org/10.5281/zenodo.1207017.
    Findings
  • Holm, S. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics 65–70.
    Google ScholarFindings
  • Hoover, J.; Portillo-Wightman, G.; Yeh, L.; Havaldar, S.; Davani, A. M.; Lin, Y.; Kennedy, B.; Atari, M.; Kamel, Z.; Mendlen, M.; Moreno, G.; Park, C.; Chang, T. E.; Chin, J.; Leong, C.; Leung, J. Y.; Mirinjian, A.; and Dehghani, M. 2020. Moral Foundations Twitter Corpus: A Collection of 35k Tweets Annotated for Moral Sentiment. Social Psychological and Personality Science doi:10.1177/1948550619876629. URL https://doi.org/10.1177/1948550619876629.
    Locate open access versionFindings
  • Horvitz, E. 2009. Asilomar Meeting on Long-Term AI Futures. URL http://erichorvitz.com/AAAI Presidential Panel 2008-2009.htm.[Online; accessed 23-March-2020].
    Locate open access versionFindings
  • Hovy, D.; Berg-Kirkpatrick, T.; Vaswani, A.; and Hovy, E. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1120– 1130.
    Google ScholarLocate open access versionFindings
  • Jentzsch, S.; Schramowski, P.; Rothkopf, C.; and Kersting, K. 2019. Semantics derived automatically from language corpora contain human-like moral choices. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 37–44.
    Google ScholarLocate open access versionFindings
  • Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; and Soricut, R. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. ArXiv abs/1909.11942.
    Findings
  • Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M. V.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L. S.; and Stoyanov, V. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv abs/1907.11692.
    Findings
  • McCullagh, P.; and Nelder, J. 1989. Generalized Linear Models, Second Edition. Chapman & Hall/CRC Monographs on Statistics & Applied Probability. Taylor & Francis. ISBN 9780412317606.
    Google ScholarFindings
  • Murphy, K. P. 2012. Machine Learning: A Probabilistic Perspective. The MIT Press. ISBN 0262018020.
    Google ScholarFindings
  • Niven, T.; and Kao, H.-Y. 2019. Probing Neural Network Comprehension of Natural Language Arguments. In Proceedings of the 57th Conference of the Association for Computational Linguistics, 4658–4664.
    Google ScholarLocate open access versionFindings
  • Florence, Italy: Association for Computational Linguistics. URL https://www.aclweb.org/anthology/P19-1459.
    Findings
  • Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017. Automatic Differentiation in PyTorch. In NIPS Autodiff Workshop.
    Google ScholarLocate open access versionFindings
  • Paun, S.; Carpenter, B.; Chamberlain, J.; Hovy, D.; Kruschwitz, U.; and Poesio, M. 2018. Comparing Bayesian Models of Annotation. Transactions of the Association for Computational Linguistics 6: 571–585.
    Google ScholarLocate open access versionFindings
  • Pavlick, E.; and Kwiatkowski, T. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics 7(0): 677–694. ISSN 2307-387X. URL https://transacl.org/index.php/tacl/article/view/1780.
    Locate open access versionFindings
  • Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; and Duchesnay, E. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12: 2825–2830.
    Google ScholarLocate open access versionFindings
  • Poliak, A.; Naradowsky, J.; Haldar, A.; Rudinger, R.; and Van Durme, B. 2018. Hypothesis Only Baselines in Natural Language Inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, 180–191. New Orleans, Louisiana: Association for Computational Linguistics. doi: 10.18653/v1/S18-2023. URL https://www.aclweb.org/anthology/ S18-2023.
    Locate open access versionFindings
  • Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. ArXiv abs/1910.10683.
    Findings
  • Rennie, J. D. M.; Shih, L.; Teevan, J.; and Karger, D. R. 2003. Tackling the Poor Assumptions of Naive Bayes Text Classifiers. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning, ICML’03, 616–623. AAAI Press. ISBN 1-57735-189-4. URL http://dl.acm.org/citation.cfm?id=3041838.3041916.
    Locate open access versionFindings
  • Rossi, F. 2016. Moral Preferences. In 10th Workshop on Advances in Preference Handling (MPREF) at AAAI.
    Google ScholarLocate open access versionFindings
  • Rossi, F.; and Mattei, N. 2019. Building ethically bounded AI. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 9785–9789.
    Google ScholarLocate open access versionFindings
  • Russell, S.; Dewey, D.; and Tegmark, M. 2015. Research priorities for robust and beneficial artificial intelligence. Ai Magazine 36(4): 105–114.
    Google ScholarLocate open access versionFindings
  • Schlesinger, A.; O’Hara, K. P.; and Taylor, A. S. 2018. Let’s talk about race: Identity, chatbots, and AI. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 315. ACM.
    Google ScholarLocate open access versionFindings
  • Schmidt, A.; and Wiegand, M. 2017. A Survey on Hate Speech Detection using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, 1–10.
    Google ScholarLocate open access versionFindings
  • Valencia, Spain: Association for Computational Linguistics. doi:10.18653/v1/W17-1101. URL https://www.aclweb.org/anthology/W17-1101.
    Findings
  • Schramowski, P.; Turan, C.; Jentzsch, S.; Rothkopf, C. A.; and Kersting, K. 2019. BERT has a Moral Compass: Improvements of ethical and moral values of machines. ArXiv abs/1912.05238.
    Findings
  • Tange, O. 2011. GNU Parallel - The Command-Line Power Tool.;login: The USENIX Magazine 36(1): 42–47. doi:10.5281/zenodo. 16303. URL http://www.gnu.org/s/parallel.
    Locate open access versionFindings
  • Tsuchiya, M. 2018. Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). Miyazaki, Japan: European Languages Resources Association (ELRA). URL https://www.aclweb.org/anthology/L18-1239.
    Locate open access versionFindings
  • Vinge, V. 1993. The Coming Technological Singularity. Whole Earth Review.
    Google ScholarLocate open access versionFindings
  • Waldrop, M. M. 1987. A Question of Responsibility. AI Magazine 8(1): 28. doi:10.1609/aimag.v8i1.572. URL https://www.aaai.org/ojs/index.php/aimagazine/article/view/572.
    Locate open access versionFindings
  • Wallach, W.; and Allen, C. 2009. Moral Machines: Teaching Robots Right from Wrong. Oxford University Press. doi:10.1093/ acprof:oso/9780195374049.001.0001.
    Findings
  • Weld, D.; and Etzioni, O. 1994. The First Law of Robotics (a Call to Arms). In Proceedings of the Twelfth AAAI National Conference on Artificial Intelligence, AAAI’94, 1042–1047. AAAI Press.
    Google ScholarLocate open access versionFindings
  • Wolf, M. J.; Miller, K.; and Grodzinsky, F. S. 2017. Why we should have seen that coming: comments on Microsoft’s tay experiment, and wider implications. ACM SIGCAS Computers and Society 47(3): 54–64.
    Google ScholarLocate open access versionFindings
  • Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; and Brew, J. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. ArXiv abs/1910.03771.
    Findings
  • Wu, Y.-H.; and Lin, S.-D. 2018. A low-cost ethics shaping approach for designing reinforcement learning agents. In ThirtySecond AAAI Conference on Artificial Intelligence.
    Google ScholarLocate open access versionFindings
  • Yu, H.; Shen, Z.; Miao, C.; Leung, C.; Lesser, V. R.; and Yang, Q. 2018. Building Ethics into Artificial Intelligence. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, 5527–5533. International Joint Conferences on Artificial Intelligence Organization. doi:10.24963/ijcai.2018/ 779. URL https://doi.org/10.24963/ijcai.2018/779.
    Locate open access versionFindings
  • Yudkowsky, E. 2008. Artificial intelligence as a positive and negative factor in global risk. Global catastrophic risks 1(303): 184.
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments