Since the rise of social media, the authority of traditional professional literary critics has been supplemented – or undermined, depending on the point of view – by technological developments and the emergence of community-driven online layperson literary criticism. So far, relatively little research (Allington 2016, Kellermann et al. 2016, Kellermann and Mehling 2017, Bogaert 2017, Pianzola et al. 2020) has examined this layperson user-generated evaluative “talk of literature” instead of addressing traditional forms of consecration. In this paper,1 we examine the professional and layperson literary criticism pertaining to a prominent German-language literary award: the Ingeborg-Bachmann-Preis, awarded during the Tage der deutschsprachigen Literatur (TDDL). We propose an aspect-based sentiment analysis (ABSA) approach to discern the evaluative criteria used to differentiate between ‘good’ and ‘bad’ literature. To this end, we collected a corpus of German social media reviews retrieved from Twitter, Instagram and Goodreads and enriched it with manual ABSA annotations: aspects and aspect categories (E.g., TEXT Motifs Themes, JURY Discussion Valuation), sentiment expressions and named entities. In a next step, the manual annotations are used as training data for our ABSA pipeline including 1) aspect term extraction, 2) aspect term category prediction and 3) aspect term polarity classification. Each pipeline ncomponent is developed using state-of-the-art pre-trained BERT models. Two sets of experiments were conducted for the aspect polarity detection: one where only the aspect embeddings were used and another where an additional context window of five adjoining words in either direction of the aspect was considered. We present the classification results for the aspect category and aspect sentiment prediction subtasks for the Twitter corpus as well as the next steps to tackle aspect term extraction. These preliminary experimental results show a good performance and accuracy for the aspect category classification, with an F1-score of 0.81, and for the aspect sentiment subtask, which uses an additional context window, with an F1-score of 0.72.
We present experiments on implicit sentiment processing based on a fine-grained event-with-sentiment dataset for the economic domain. In information extraction, events encode factual information about real-world occurrences reported in news text, while sentiment analysis concerns expressions of opinions and subjectivity. We contribute to the under-researched task of `polar fact’ or `implicit sentiment’ processing (Wilson (2008), Toprak et al. (2010)) by combining extraction of events and sentiment to automatically assign common-sense connotational opinion to facts. The SENTiVENT dataset of English economic news articles contains token-level event annotations (event annotation described in Jacobs and Hoste (forthc.)) on which we manually annotate aspect-based sentiment consisting of sentiment expressions (the words expressing implicit/explicit sentiment), targets (the entity to which the sentiment is directed), and polarities (‘positive’, ‘negative’, and ‘neutral’ investor opinion). The sentiment annotations are validated in an inter-annotator agreement study and a series of polarity classification experiments on coarse- and fine-grained sentiment-target representations. For the coarse-grained experiments, we cast implicit polarity categorization as a text-classification task of sentiment expressions with target spans using large-scale pretrained language model embeddings. We fine-tune several models on the classification task and experiment with the integration of existing lexicons comparing performance of in-domain and general sentiment lexicons. For fine-grained experiments, we apply sentiment-target-polarity triplet extraction to test the feasibility of token-level extraction of implicit sentiment. To this end, we apply a state-of-the-art Grid Tagging end-to-end model (Wu et al. 2020) and compare performance to benchmark explicit sentiment datasets. The SENTiVENT dataset fills the need for a manually annotated dataset in financial text mining applications while also being useful for implicit sentiment processing. These experiments validate the data resource and the results suggest best-practices for the creation and natural language engineering of domain-specific implicit sentiment applications.
Recent studies (Blevins et al. 2018, Tenney et al. 2019, etc) have presented evidence that linguistic information, such as Part-of-Speech (PoS), is stored in the word representations (embeddings) learned by neural networks, with the neural networks being trained to perform next word prediction and other NLP tasks. In this work, we focus on so-called probing tasks or diagnostic classifiers that train linguistic feature classifiers on the activations of a trained neural model and interpret the accuracy of such classifiers on a held-out set as a measure of the amount of linguistic information captured by that model. In particular, we show that the overlap between training and test set vocabulary in such experiments can lead to over-optimistic results, as the effect of memorization on the linguistic classifier’s performance is overlooked. We then present our technique to split the vocabulary across the linguistic classifier’s training and test sets, so that any given word type may only occur in either the training or the test set. This technique makes probing tasks more informative and consequently assess more accurately how much linguistic information is actually stored in the token representation. To the best of our knowledge, only a few studies such as Bisazza and Tump (2018) have reported on the effect of vocabulary splitting in this context and we corroborate their findings. From our experiments we found that incorporating such a technique for PoS classification, clearly shows the effect of memorization when the vocabulary is not split, especially at the word-type representation level (that is, the context-independent embeddings, or layer 0). For our experiments, we trained a language model on next-word-prediction. We then extracted the word representations from the encoder, for all the layers. These representations are then taken as the input to a logistic regression model, that is trained on PoS classification. The model is run for the two different settings: with and without vocabulary splitting. Finally, the output is analysed and compared between the different split settings. Across all layers, the full vocabulary setting gave high accuracy values (85-90%), compared to when the vocabulary split was enforced (35 – 50%). To further substantiate that this is due to memorization, we also compared the results to that from a LM with randomly initialized embeddings. The difference of around 70% further suggests that the model is memorizing words, but not truly learning syntax. Our work provides evidence that the results of linguistic probing tasks only partially account for the linguistic information stored in neural word representations. Splitting the vocabulary provides a solution to this problem, but is not itself a trivial task and comes with its own set of issues, such as large deviations across random runs. We conclude that more care must be taken when setting up probing task experiments and, even more, when interpreting them.
In this paper, we explore self-distillation as a means to improve statistical dependency parsing models for Dutch and German over purely supervised training. Self-distillation (Furlanello et al. 2018) trains a new student model on the output of an existing (weaker) teacher model. In contrast to most previous work on self-distillation, we perform distillation using a large, unannotated corpus. We show that in dependency parsing as sequence labeling (Spoustov´a and Spousta 2010, Strzyz et al. 2019), self-distillation plus finetuning provides large improvements over models that use supervised training. We carry out experiments on the German T¨uBa-D/Z universal dependency (UD) treebank (C¸ ¨oltekin et al. 2017) and the UD conversion of the Dutch Lassy Small treebank (Bouma and van Noord 2017). We find that self-distillation improves German parsing accuracy of a bidirectional LSTM parser from 92.23 to 94.33 Labeled Attachment Score (LAS). Similarly, on Dutch we see improvement from 89.89 to 91.84 LAS.
Modern language models, especially those based on deep neural networks, frequently use bottom-up vocabulary generation techniques like Byte Pair Encoding (BPE) to create word pieces enabling them to model any sequence of text, even with a fixed-size vocabulary significantly smaller than the full training vocabulary. The resulting language models often prove extremely capable. Yet, when included into traditional Automatic Speech Recognition (ASR) pipelines, these languages models can sometimes perform quite unsatisfyingly for rare or unseen text, because the resulting word pieces often don’t map cleanly to phoneme sequences (consider for instance Multilingual BERT’s unfortunate breaking of Sonnenlicht into Sonne+nl+icht). This impairs the ability for the acoustic model to generate the required token sequences, preventing good options from being considered in the first place. While approaches like Morfessor attempt to solve this problem using more refined algorithms, these approaches only make use of the written form of a word as an input, splitting words into parts disregarding the word’s actual meaning. Meanwhile, word embeddings for languages like Dutch have become extremely common and high-quality; in this project, the question of whether this knowledge about a word usage in context could be leveraged to yield better hyphenation quality will be investigated. For this purpose, the following approach is evaluated: A baseline Transformer model is tasked to generate hyphenation candidates for a given word based on its written form, and those candidates are subsequently reranked based on the embedding of the hyphenated word. The obtained results will be compared with the results yielded by Morfessor based on the same dataset. Finally, a new set of linguistic rules to perform Dutch hyphenation (suitable for use with Liang’s hyphenation algorithm from TEX82) will be presented. The resulting output of these rules will be compared to currently available rule-sets.
In this paper we investigate methods for improving the sentiment analysis functionality of Pattern.nl, the Dutch submodule of Pattern, an open-source library for web mining and natural language processing. We discuss the impact on performance of three different potential improvements: extending the module’s internal sentiment lexicon; removing subsets of neutral words from the sentiment lexicon; and improving the algorithm for combining multiple word-level sentiment ratings into a sentence-level sentiment rating. We evaluated the improvements on datasets from the product review domain (books, clothing and music) and a dataset of short emotional stories. The experiments show that lexicon expansion does not lead to better results; new normalization techniques, on the other hand, show a limited but consistent performance increase for sentiment ratings.
SPOD is a tool for Dutch syntax in which a given corpus is analysed according to a large number of predefined syntactic characteristics. SPOD is an extension of the PaQu (”Parse and Query”) tool (Odijk et al. 2017). SPOD is available for a number of standard Dutch corpora and treebanks. In addition, you can upload your own texts which will then be syntactically analysed. SPOD will run a potentially large number of syntactic queries in order to show a variety of corpus properties, such as the number of main and subordinate clauses, types of main and subordinate clauses, and their frequencies, average length of clauses (per clause type: e.g. relative clauses, indirect questions, finite complement clauses, infinitival clauses, finite adverbial clauses, etc.). Other syntactic constructions include comparatives, correlatives, various types of verb clusters, separable verb prefixes, depth of embedding etc. SPOD allows linguists to obtain a quick overview of the syntactic properties of texts, for instance with the goal to find interesting differences between text types, or between authors with different backgrounds or different age. In the paper, we describe the SPOD tool in some more detail, and we provide a case study, illustrating the type of investigations which are enabled andfacilitated by SPOD. Most of the syntactic properties are implemented in SPOD by means of relatively complicated XPath 2.0 queries, and as such SPOD also provides examples of relevant syntactic queries, which may otherwise be relatively hard to define for non-technical linguists. SPOD is available via https://www.let.rug.nl/alfa/paqu/spod
Pattern (https://www.clips.uantwerpen.be/pages/pattern-nl) is an open-source Python package for NLP that is developed and maintained by the CLiPS Computational Linguistics group at Universiteit Antwerpen. The submodule for Dutch, pattern.nl, contains a rule-based sentiment analyzer, which is based on a built-in lexicon of about 4,000 Dutch lemmas. The lexicon contains a subjectivity and polarity score for each word, which are used to calculate a score for an input sentence. The usefulness of the lexicon was evaluated in 2012 by using it to classify book reviews. However, the applicability of Pattern in more general-domain sentiment analysis tasks is limited. For example, the sentences During the war, my youngest daughter died. or just broke up with my significant other and I don't want to live anymore. will receive a neutral judgement from the sentiment analysis function of pattern.nl. In order to generalise pattern.nl's sentiment analysis functionality, we propose to supplement its emotion lexicon with additional Dutch words and an associated subjectivity/polarity score. In this talk, we describe our attempt to extend pattern.nl with words from Moors lexicon (http://crr.ugent.be/papers/Moors_et_al_BRM_norms_Valence_Arousal_Dominance_AoA.pdf). Moors lexicon contains manually-annotated scores of valence, arousal and dominance for about 4,300 Dutch words. The ratings of valence were first rescaled to the [-1;1] range used by pattern.nl, and then added to its lexicon, increasing the coverage to a total of 6,877 unique words. We compared the effect of this extension by measuring the mean average error (MAE) of the original version of pattern.nl and our extended version against a balanced dataset of 11,180 book reviews and the associated ratings (1 to 5 stars) collected from bol.com. Preliminary experiments on the correlation between the common subset between pattern.nl and Moors' (0.8) bode well, but despite the increase in coverage, preliminary results are negative: the original version of pattern.nl seems to perform better than after the lexicon expansion. This was also confirmed by further tests, where we tried, to no avail, – removing Moors' words centered around 0 (i.e., neutral ones); – removing stopwords from Moors'; – replicating the original pattern.nl evaluation by binarizing the dataset and using F1 score. Part of the problem might lie in the dataset used for the evaluation: reviews are related to sentiment, but indirectly; furthermore, the dataset is very noisy. Different results could also be obtained by PoS-tagging and lemmatizing the data, a step that is not technically required but might be beneficial to increase the coverage of Moors' lexicon in sentences. We are currently looking for suitable datasets for Dutch that can be used to evaluate our extension, preferably datasets that are more general domain than product reviews.
In this work, we investigate automatic controversy detection in Dutch news using a distant supervised approach based on entropy. We collected a total of 1859 news articles from Facebook from five different Dutch news providers (NOS, RTL Nieuws, de Volkskrant, het Parool, NRC and de Telegraaf) together with their Facebook users’ reactions (LIKE, LOVE, HAHA, WOW, SAD and ANGRY). We used the reactions as proxies for controversies, assuming that the higher the entropy of the reactions, the more controversial is the news. A manual exploration the 10-top and 10-bottom news of the dataset ordered by entropy confirmed the validity of the intuition. We then developed a linear regression model to predict the controversy of news based on token and character n-grams. As a baseline, we used a dummy regressor always predicting the average of the entropy. We investigate three experimental settings: i.) all-news, a 10-fold cross validated model on the full corpus; ii.) in-source, a 10-fold cross validated model on each news source separately; and iii.) across-source, where we trained on one news source and tested on the other 5 (e.g. train on NOS and test on het Parool). In all experimental settings, the model beat the baseline. In particular, in all-news the model MSE=0.033 (baseline MSE= 0.049); in in-source the average of the model MSE=0.036 (baseline MSE= 0.042); and in across-source the average of the model MSE=0.052 (baseline MSE= 0.59). We extended the model to predict topics, the number of reactions and their type to form a complete pipeline.
Machine translation (MT) quality has improved enormously since the arrival of neural machine translation (NMT). The most noticeable improvement compared to statistical MT systems is the increased grammaticality and fluency of the produced MT output. At the lexical level, the quality of NMT systems is less promising. New types of lexical mistakes appear in NMT output, such as the occurrence of non existing words, i.e. words that are not part of the vocabulary of the target language and were thus invented by the NMT system. For MT use cases in which readers only have access to the MT output without the source text, such non-existing words can affect comprehension as the intended source meaning may not be recovered. To investigate if and to what extent non-existing words in English-to-Dutch NMT output impair comprehension, an experiment was set up in SurveyMonkey. Eighty-six participants were given 15 non-existing words (5 single words and 10 noun compounds) and were either asked to describe the meaning of these words or to select the correct meaning from a predefined list. The words were presented either in isolation or in sentence context. Participants were asked to indicate how confident they were about their answer. Results show that non existing words indeed impair comprehension as in 60% of the cases the participants gave a wrong answer. Sentence context had a positive impact and made it easier for the participants to determine the meaning of the non-existing word. Participants were also more confident about their answer when the words were presented in sentence context.
This paper describes the systems designed by the Fraunhofer IAIS team at the CLIN29 shared task on cross-genre gender detection in Dutch. We show two alternative classification approaches: a rather standard one consisting of feature engineering and a random forest classifier; and an alternative one involving a LSTM classifier. Both are enhanced by a LDA model trained on stems. We considered various features such as frequency of function words, parts-of-speech and sentiment among others. We achieved 53.77% average accuracy in the cross-genre settings.
We present our system for the CLIN29 shared task on cross-genre gender detection for Dutch. We experimented with a multitude of neural models (CNN, RNN, LSTM, etc.), more "traditional" models (SVM, RF, LogReg, etc.), different feature sets as well as data pre-processing. The final results suggested that using tokenized, non-lowercased data works best for most of the neural models, while a combination of word clusters, character trigrams and word lists showed to be most beneficial for the majority of the more "traditional" (that is, non-neural) models, beating features used in previous tasks such as n-grams, character n-grams, part-of-speech tags and combinations thereof. In contradiction with the results described in previous comparable shared tasks, our neural models performed better than our best traditional approaches with our best feature set-up. Our final model consisted of a weighted ensemble model combining the top 25 models. Our final model won both the in-domain gender prediction task and the cross-genre challenge, achieving an average accuracy of 64.93% on the in-domain gender prediction task, and 56.26% on cross-genre gender prediction.
We present the results of cross-genre and in-genre gender classification performed on the data sets of Dutch tweets, YouTube comments and news prepared for the CLIN 2019 shared task. We propose a recurrent neural network architecture for gender classification, in which the input word and part-of-speech sequences are fed to the LSTM layer, which is followed by average and max pooling layers. The best cross-genre accuracy of 55.2% was achieved by the model trained on YouTube comments and tweets, and tested on the balanced news corpus, while the best ingenre accuracy of 61.33% was achieved on YouTube comments. Overall, the proposed approach ranked 2nd in the global cross-genre ranking and 6th in the global in-genre ranking of CLIN 2019 shared task.
This work is a result of participation in shared task on gender detection in Dutch. The task was to predict gender within and across different genres. This work applies some existing ideas about using lexical and more abstract text representations (morphological, syntactical labels, text bleaching). It provides a comparison of different features across genres in two types of tasks and presents two pipelines. Using three types of features, we found that lexical features are more significant, although other features also show good results making the model more robust. Final scores where in range 0.61-0.64 for ingenre and 0.53-0.56 for cross-genre prediction.
