THE IMPORTANCE OF SOCIOCOGNITIVE ELEMENTS IN THE PROCESS OF MULTIMODAL DISCOURSE LOCALIZATION IN THE DIGITAL MEDIA

Jasmina Đorđević

Faculty of Philosophy, University of Niš

jasmina.djordjevic@filfak.ni.ac.rs

Multimodal discourse is the social use of language based on different semiotic entities employed both as meaning-making and meaning-exchanging tools (Bateman et al., 2017) while the digital media are a means of enabling interactivity via a digitally supported channel of communication (Đorđević, 2022). Therefore, multimodal discourse in the digital media is the social use of language based on various semiotic resources via a digitally supported channel of communication. In the digital media, parts of the meaning of multimodal discourse may derive from the interaction between words, visuals (cartoons, memes, giffs, illustrations, etc.), sounds (songs, musical compositions, jingles, etc.) and various other semiotic resources (e.g. typography, icons, colour schemes, hashtags, etc.). If such discourse is to be presented to users of different language and cultural backgrounds, the semiotic resources have to be adapted for consumption by those users. The process that enables both linguistic and cultural adaptation of content to meet the requirements of a particular target audience is referred to as localization, an interdisciplinary field relying on linguistics, cultural studies, language engineering and sociology (Jiménez-Crespo 2013). Since multimodal discourse is a complex multi-faceted presentation of a multiplicity of meanings, a significant aspect to consider in its localization are sociocognitive elements, an essential constituent in the discourse – cognition – society triangle van Dijk (2018) identified as crucial in the understanding of discourse. As such, sociocognitive elements are underlying the cognitive and cultural models that will shape the way in which the target audience will understand the localized discourse. Although a lot of research is available on multimodal discourse, localization and the sociocognitive aspect of discourse, a research gap is obvious referring to the role of sociocognitive elements in the process of multimodal discourse localization. Starting from the assumption that multimodal discourse localization is a sociocognitive practice navigating the cognitive and cultural differences between the source and target audiences, this presentation will demonstrate the importance of sociocognitive elements in that process. Based on an adapted annotation relying on Kress and van Leeuwen’s (2006) understanding of ideational metafunction, four examples of localized content (two commercials and two scenes from animated shows) localized in different language/culture pairs will be presented. The annotation follows the rules of an immediate constituent analysis used to identify five sociocognitive elements: cultural references, visual metaphors, social norms, communicative practices and cognitive models, each linked to specific semiotic resources in the individual examples, which contributed to the adequate or inadequate localization. The results of the analyses clearly point to the necessity of understanding sociocognitive elements as an essential precondition for an effective and appropriate localization of multimodal discourse in the digital media.

References

Bateman, J., Wildfeuer, J., & Hiippala, T. (2017). Multimodality: Foundations, research and analysis – A problem-oriented introduction. De Groyter Mouton.

Đorđević, J. (2022). Digital media discourse in linguistic research. Faculty of Philosophy Niš.

Jiménez-Crespo, M. A. (2013). Translation and web localization. Routledge.

Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design. Second edition. Routledge.

van Dijk, T. A. (2018). Socio-cognitive discourse studies. In J. Flowerdew & J. E. Richardson (Eds.), The Routledge handbook of Critical Discourse Studies (pp. 26–44). Routledge.