ABSTRACT Visual components of communication interact with the core linguistic system at all levels and make an important contribution to the meaning of the utterance expressed. This does not only apply to sign languages, which exclusively rely on the visual channel, but also co‐speech gestures and emojis, which are visual means that accompany and enhance spoken and written languages respectively. Visual meaning components often have iconic properties and are used to directly depict properties of the referent or event referred to. Formal semantic theories have only quite recently started to address the relevance of visual meaning components and to integrate insights from cognitive linguistics and semiotics into formal semantic and pragmatic analyses. In this review, we discuss the impact of visual components on linguistic interpretation in all three modalities (sign language, spoken language and written language). We argue that so far visual components are mainly analysed through direct applications and modest extensions of the formal linguistic vocabulary established for the semantic analysis of spoken languages; that is, depictive (iconic) meaning components are mostly analysed similarly to non‐depictive (arbitrary) descriptive meaning components. However, it is becoming more and more evident that the traditional formal linguistic repertoire developed for spoken languages needs to be extended in non‐trivial ways to meet the specific requirements of visual components and the non‐descriptive properties of visual depictions.

Sentences are not just mere strings of words or signs, but manifest a complex internal structure. Linguistic research has demonstrated that sign languages and spoken languages both exhibit hierarchical constituent structure which determines how individual elements in a sentence relate to each other. Here, we report the first adaptation of the psycholinguistic click paradigm, which aims to demonstrate an impact of hierarchical constituent structure on cognitive processing load during auditory language processing, to the visuo-spatial modality of sign languages. We performed two independent online experiments: The main experiment with a group of 53 deaf signers using German Sign Language (DGS) as their primary means of communication and a control experiment with a group of 53 hearing non-signers. Both groups were shown videos of syntactically complex sentences in DGS. A white flash (mimicking the click in the auditory domain) to which participants had to respond could occur as an overlay to the video at different levels in the constituent structure. Our pre-registered inferential analyses yielded no effect for our syntactic manipulations, neither in the group of signers nor in the group of non-signers. Additional exploratory analyses suggest general effects of attention during the processing of communicative signals, as even the group of non-signers' behaviour was influenced by non-manual cues (i.e., eyebrow raising), despite their lack of knowledge of DGS. We conclude that the simultaneous and time-shifted presence of different syntax-relevant cues (i.e., hands, mouthings, and non-manuals) makes the processing of the sign stream robust against disruption by extraneous visual signals and argue that non-signers attend to some non-manual cues, due to their resemblance to communicative gestures independent of language. (* Trettenbrein and Maran share first authorship.)

The processing of referential expressions is influenced by their form, but also by the accessibility of their antecedents. In sign languages, discourse referents are typically assigned to referential loci (R-loci) in the signing space, which can be used to identify discourse referents later on. This study on German Sign Language (DGS) examines whether overt manual localization of a discourse referent with an index sign can be used to increase the prominence and, hence, accessibility of the corresponding discourse referent. Using an eye tracking paradigm, deaf adult DGS signers were presented with DGS videos containing two-sentence discourses introducing two discourse referents in the first sentence. The marking of the two new discourse referents varied by either lack of overt localization or localization of one of the two discourse referents with the index sign. The second sentence started with either the subject or object of the first sentence as a bare noun. Overall, signers fixated the discourse referent that was continued with in the second sentence. However, overt manual localization did not affect signers’ gaze suggesting that overt localization did not lead to facilitatory processing of referential expressions. We speculate that localization may either not carry the linguistic function of enhancing prominence or that the stimuli used in this experiment did not allow for revealing that function.

Sign languages are perceived visually and externalized using a signer’s hands, face, and upper body. During sign language comprehension, deaf signers primarily focus their gaze on the face, while hearing non-signers attend more to the hands of a signer. Little is known about whether deaf signers adapt their gaze behaviour when processing unknown signs. Here, we report eye-tracking data from 15 deaf native signers of German Sign Language (DGS) and 15 hearing non-signers who were presented with videos in either DGS or an unknown sign language, all containing no linguistic mouth actions. Our data confirm that deaf signers generally fixate more on the face of a signer than hearing non-signers who attend to the hands in sign space. Moreover, only deaf signers increase their attention to the hands when processing video stimuli consisting of unknown signs compared to familiar signs, suggesting similar adjustment behaviours as observed in spoken languages.

Lexical variables such as iconicity or age of acquisition are known to be important sources of variance in psycholinguistic experiments. To control for such variables, researchers working on German Sign Language (DGS) need to use stimuli rated for these constructs (e.g., iconicity) by an independent group of participants before implementing their actual experiment. Up to now, several research groups have made such rating data publicly available but a central resource is currently still lacking. Against this background, this short paper provides a roadmap for the collaborative creation of a so-called “DGS-LEX”, a lexical database for psycholinguistic research on DGS, similar to ASL-LEX. By integrating relevant data from different published and forthcoming studies, this joint effort aims to establish a new database for lexical variables in DGS primarily based upon subjective ratings. (Note: This paper is bilingual. The contents of the English and German versions are identical. The German version can be retrieved from https://doi.org/10.6084/m9.figshare.28282907. A summary in DGS is available from https://doi.org/10.6084/m9.figshare.28282898).

In the present experimental studies, we investigated the interaction of co-speech palm up gestures and prosody in sentences typically triggering free choice (FC) interpretation. Results show that multimodal sentences boost the exclusive interpretation compared to utterances displayed only in the spoken modality, which only received the FC interpretation (inclusive reading). On the basis of this evidence, we suggest that visual cues should more systematically be used as a tool to shed light on linguistic theories, and enrich experimental methods.

Research on spoken languages has shown that response particles may indicate the truth of a previous utterance or the polarity of the response. In responses to negative antecedents, the two functions come apart and particles become ambiguous. We present the first quantitative study on response strategies in sign languages by discussing data from a production experiment in German Sign Language ( Deutsche Geb & auml;rdensprache; DGS). The results indicate that DGS does not exploit the potential of simultaneous manual and nonmanual strategies to disambiguate responses. Still, the type of articulator influences the choice of response element. We propose an optimality-theoretic model to account for the role of articulator type, the disambiguation potential, and the morphosyntax of response elements in DGS.

No AccessWhy artichokes and palms don’t grow on trees – the grammaticalization of question particles from co-speech gesture in sign languageMarkus Steinbach, Marco Coniglio, Katharina PaulMarkus SteinbachSearch for more papers by this author, Marco ConiglioSearch for more papers by this author, Katharina PaulSearch for more papers by this authorhttps://doi.org/10.14220/9783737015530.81SectionsPDF/EPUB ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinkedInRedditEmail About Previous chapter Next chapter FiguresReferencesRelatedDetails Download book coverWiener Arbeiten zur LinguistikVolume 7 1st editionISBN: 978-3-8471-1553-3 eISBN: 978-3-7370-1553-0HistoryPublished online:March 2023 PDF download

While the first mention bias has been well investigated in spoken languages, little is known about the presence of a similar bias in sign languages. In sign languages, pronominal pointing signs are directed towards referential locations in the ipsilateral and contralateral area of the signing space usually associated with referents in previous contexts. The present event-related potential study investigates the presence of a first mention effect during pronoun resolution in German Sign Language. We present participants with sentence sets containing two referents in the first sentence and a pronominal sign at the beginning of the second sentence directed to the ipsilateral or contralateral area of the signing space. Results show an N400 component for contralateral compared to ipsilateral pronominal signs suggesting increased processing costs associated with the second referent assigned to the contralateral area. Thus, the current study provides evidence for a first mention effect highlighting its modality independent nature.

Sign languages make use of the full expressive power of the visual-gestural modality to report the utterances and/or actions of another person. A signer can shift into the perspective of one or more persons and reproduce the utterances or actions from the perspective of these persons. This modality-specific device of utterance and action report is called role shift or constructed action. Especially in sign language narration, role shift is a productive and expressive means that can be used to demonstrate linguistic and non-linguistic actions. Recent developments in sign language linguistics put forth new formal semantic analyses of role shift at the interface between sign language and gesture, integrating insights from classical cognitive and formal analyses of quotation, demonstration and perspective/context shift in spoken and sign languages. In this article, I build on recent accounts of role shift as a modality-specific device of demonstration and show that a modified version of this theory even accounts for cases of complex demonstrations including hybrid demonstrations, multiple demonstrations and demonstrations involving a complex interaction of gestural and linguistic components.

Abstract This paper aims to investigate the so-called gehen+infinitive construction in German, in which an inflected form of the (movement) verb gehen ‘go’ is combined with the infinitive of another main verb and, thus, seems to behave like an auxiliary syntactically. Supported by two questionnaire studies, we will argue that (i) this construction undergoes a currently observable grammaticalization process, and that (ii) it is generally used to encode an aspectual reading, namely ingressivity. Finally, we provide a proposal for the diachronic development of the gehen+infinitive construction, arguing for a transition from a biclausal structure to a monoclausal one, and discuss the consequence of this shift at the syntax-semantics-interface.

AbstractDifferential object marking has been described for many typologically different spoken languages and seems to be also a prevalent property of agreement marking in the visual-gestural modality. Many studies on agreement in sign languages have shown that object agreement in sign languages is subject to specific restrictions which are very similar to the restrictions described for differential object marking in spoken languages. In addition, unrelated sign languages have developed a specific marker which seems to be mainly used to mark object agreement. It is thus not a surprise that recent studies have analysed (at least certain instances of) object agreement marking as an instance of differential object marking. This chapter proceeds as follows. First, the discussion of differential object marking in sign language is embedded in a broad evaluation of recent empirical studies on agreement marking in German Sign Language. Secondly, based on the results of the empirical studies, a unified analysis of agreement marking in DGS is developed. Two main insights are presented: (a) DGS has two different kinds of agreement markers, a preverbal differential object marker and a postverbal agreement marker. (b) These two markers constitute two different stages of the same grammaticalization process.

This paper presents the results of a Sentence Reproduction Task (SRT) investigating conditional sentences in German Sign Language (DGS). We found that participants mark conditional sentences in DGS by systematically using different non-manual markers on the antecedent and the consequent. In addition, these non-manual markers were frequently used in combination with one or two manual signs. However, the manual markers were omitted in the test sentences, i.e., the input stimuli the participants were asked to reproduce. The results of our experimental study are, on the one hand, consistent with descriptions of manual and non-manual strategies used to mark conditional sentences in different unrelated sign languages. On the other hand, our findings provide new insights on the multi-layered marking of conditional sentences in DGS.

Languages offer various ways to present what someone said, thought, imagined, felt, and so on from their perspective. The prototypical example of a perspective-shifting device is direct quotation. In this review we define perspective shift in terms of indexical shift: A direct quotation like “Selena said, ‘Oh, I don't know.’” involves perspective shift because the first-person indexical ‘I’ refers to Selena, not to the actual speaker. We then discuss a variety of noncanonical modality-specific perspective-shifting devices: role shift in sign language, quotatives in spoken language, free indirect discourse in written language, and point-of-view shift in visual language. We show that these devices permit complex mixed forms of perspective shift which may involve nonlinguistic gestural as well as visual components.

Various studies provide evidence for a phonological priming effect in the recognition of single signs based on phonological parameters, i.e., handshape, location and movement. In addition, some of these studies show that phonological parameters influence this effect differently. The current eye tracking study on German Sign Language examined the presence of a phonological priming effect at the sentence level depending on the phonological relation of prime-target sign pairs. We recorded participants’ eye movements while presenting a video of sentences containing either related or unrelated prime-target sign pairs, and a picture of the target and the distractor. The data provided evidence for a phonological priming effect for sign pairs sharing handshape and movement while differing in location. Moreover, a difference between parameters in their contribution to sign recognition was suggested such that recognition was facilitated for signs sharing handshape, but was inhibited for signs sharing location. Showing that sub-lexical features influence sign language processing.

In sign languages, just as in many spoken languages, number can be marked on nouns, pronouns, and verbs, and quantifiers are used to specify quantity within noun phrases. The chapter does not address the expression of grammatical number in one specific sign language, but rather describes patterns found in various sign languages, focusing on modality-independent and modality-specific properties of number marking. As for the former, nominal and verbal plurals are commonly realized by reduplication. As for number-marking strategies specific to visual–spatial languages, it is found that sign languages employ the two hands (e.g. lexical plurality), the signing space in front of the signer's body (e.g. plural marking on predicates), and specific reduplication types that are not attested in spoken languages (e.g. sideward reduplication of certain nouns). In addition, the choice of pluralization strategy is determined by modality-specific phonological features, and we are thus dealing with phonologically conditioned allomorphy.

When referring to non-present entities, speakers and signers can select from a range of different strategies to create expressions that range from extremely concise to highly elaborate. This design of referring expressions is based partly on the availability of contextual information that can aid addressee understanding. In the small signing community of Providence Island, signers’ heavy reliance on extra-linguistic information has led to their language being labelled as context-dependent (Washabaugh, de Santis & Woodward 1978). This study investigates the semiotic strategies that deaf signers in Providence Island use to introduce non-present third person referents, and examines how signers optimise specificity and minimise ambiguity by drawing on shared context. We examined first introductions to non-present people in spontaneous dyadic conversations between deaf signers and analysed the semiotic strategies used. We found that signers built referring expressions using the same strategies found in other sign languages, yet designed expressions that made use of contextual knowledge shared through community membership, such as geography, local spoken languages and traits of fellow islanders. Our signers also used strategies described as unusual or unattested in other sign languages, such as unframed constructed action sequences and stand-alone mouthings. This study deepens our understanding of context dependence by providing examples of how context is drawn upon by communities with high degrees of shared knowledge. Our results call into question the classification of sign languages as context-dependent or context-independent and highlights the differences in data collection across communities and the resulting limitations of cross-linguistic comparisons.

Response elements (REs) like English yes and no fulfil two functions. They may affirm or reject a previous utterance, or they may indicate that the response to the previous utterance has positive or negative polarity. In responses to negative sentences, these two functions come apart. Spoken languages investigated so far seem to display different preferences for the interpretation of REs to signal either the positive/negative polarity of the response clause or the affirmation/rejection of the truth of the previous utterance. The present paper investigates the meaning and use of REs in German Sign Language (DGS). We present the results of a discourse completion experiment in DGS, which is the first quantitative study of the response system of a sign language, and provide a preliminary theoretical analysis of this system. Sign languages are of particular interest in this context since they systematically use multiple articulatory channels, which can, in principle, encode truth and polarity at the same time. The results show that DGS employs manual and non-manual REs which encode both truth and polarity, i.e. are ambiguous, as well as REs that encode only truth. The ambiguous REs are used more often to encode truth than polarity, and are rarely disambiguated by simultaneous non-manual REs. Hence, DGS does not use the potential made available by the visual-gestural modality in the domain of response strategies.