Technically speaking: Semiotics and the Internet

One of the trends in linguistics in recent years is to view language in a larger context, incorporating not just the nature and structure of language, but how it is used in wider social contexts, sometimes referred to as language ecology. This is in line with an approach to culture and meaning known as semiotics, the study of how meaning is conveyed through signs. A “social semiotic” view of language has influenced the approach to language learning, viewing it as a social, dialogic process of meaning construction that includes different media, modes and symbols. Claire Kramsch explains:

Whereas folk notions of language learning see it as an incremental accumulation of atomistic structures that moves the learner from word to sentence, from sentence to paragraph, and from paragraph to text, a social semiotic approach considers language as a holistic network of various signs in the environment, including gestures, silences, body postures, graphic and other visual and acoustic symbols, which shape a context of meaning and invite us to respond to it. (2002)

This multimodal approach to language is particularly apt given the nature of communication in the Internet age. With the enhanced multimedia capabilities of mobile phones, everyday communication increasingly incorporates nonverbal resources such as photos and videos. Mobile apps like Instagram are used to communicate through images and video clips. Instagram users in turn can connect their accounts to social network services such as Facebook, Twitter, or Flickr. Increasingly, we are seeing multimodal communication mediated through networked devices and services. Users may initiate a conversation on a wearable device such as an Apple Watch (maybe sending a heartbeat), follow up with photos from a smartphone, and later continue the conversation on Facebook on their tablet or laptop.

Semioticians look not only at language use, but also examine the significance of cultural phenomena such as advertisements, films, or graffiti. One of the key figures in this field was Ferdinand de Saussure, who distinguished between a signifier (such as a word like “tree”) and the signified (the natural object in the forest). The sign is the combination of the signified and signifier, establishing the relationship between the two. Semioticians point to the fact that different cultures might have different relationships between the signifier and the signified. One might, for example, show respect in one culture by averting ones gaze, while in another culture one conveys the same meaning by looking directly into someone’s eyes.

Semoiticians today frequently use the concept of “symbiotic resources”:

[I] define semiotic resources as the actions and artifacts we use to communicate, whether they are produced physiologically – with our vocal apparatus; with the muscles we use to create facial expressions and gestures, etc. – or by means of technologies – with pen, ink and paper; with computer hardware and software; with fabrics, scissors and sewing machines, etc. Traditionally they were called ‘signs’. For instance, a frown would be a sign of disapproval, the color red a sign of danger, and so on. Signs were said to be the union of a signifier – an observable form such as a certain facial expression, or a certain color – and a signified – a meaning such as disapproval or danger. The sign was considered the fundamental concept of semiotics…In social semiotics the term ‘resource’ is preferred, because it avoids the impression that ‘what a sign stands for’ is somehow pre-given, and not affected by its use (van Leeuwen, 2005, p. 3).

This notion that signs are dynamic and change over time, and as used in different contexts, highlights the transformative nature of communication. On the Internet we are not just consumers but creators. From a symbiotic perspective, we are sign-makers who shape and combine semiotic resources to reflect our own interests. We might do mashups of YouTube videos (substituting our own soundtrack), write fanfiction transforming anime storylines, or use Google Photos to create instant image-based narratives to share. One of the online activities young people favor is playing multiplayer games, which bring together a variety of semiotic resources, including gaming history, eye-hand coordination, language ability (to chat with other gamers), strategic reasoning, and a host of other resources and skills.

Exploring the multiple dimensions in which cultures express meaning and identity is facilitated by the media capture capabilities now available on mobile devices. One can explore the increasingly rich cultural diversity of many urban “linguistic landscapes” through capturing and analyzing street signs, store displays, graffiti, billboards, posted personal ads, community bulletin boards, or restaurant menus. Such “realia” have long been a staple resource in language instruction, but they also offer rich fodder for cultural study. Scholars such as Jon Bloomeart have explored how the study of the variety of signs in a neighborhood can reveal its history, ethnic makeup, and intergroup dynamics (2013). Students of language and culture can go beyond capturing images and take advantage of the audio/video features of mobile devices to film street scenes and capture conversations. A particularly rich source of cultural and linguistic information are interviews with residents, which could be weaved into compelling digital stories. “Digital storytelling” is a powerful tool for exploring personal experiences and histories, incorporating photos and videos. In cross-cultural studies, this offers an opportunity to capture and reflect on “rich points”, those experiences that are revelatory in terms of both the other culture or individual and one’s own values and perspectives.

The built-in GPS capabilities of today’s smart phones allows images and videos to be geo-tagged, enabling the creation of personalized maps, place-based photo stories, or narrated city tours. That capability has been used to create innovative mobile games for language and culture learning. The ARIS platform (for Augmented Reality and Interactive Storytelling) enables sophisticated mobile game creation featuring augmented reality, the ability to overlay textural or other information overviews captured by a phone camera (Holden & Sykes, 2011). The tool allows for creation of games with quite interesting features, that combine a virtual environment with real-world locations. QR (Quick Response) codes, for example, can be posted in designated areas, which, when scanned with a camera, provide information on that location or further game directions. Game players have access within the app to recording audio and video, and there is even an image matching functionality, which compares photos taken with those in the game, triggering possible game events. One game created with ARIS is Mentira, which combines virtual experiences with real-world visits to locations in Albuquerque, New Mexico. The object of the game is to solve a murder mystery and involves users gaining information from site visits and from conversations with non-playing characters in the game. Another game created with ARIS is Chrono-Ops. The game has an ecological theme: players are tasked with inventing sustainability projects to save the planet. Directions are given in one of three different languages. As part of the game process, users write texts, record audio, and shoot video, all of which can become assets for future players. All ARIS games can be downloaded from the project site.

Another aspect of language learning in which images can play in important role is in vocabulary development. It has been known for some time in second-language acquisition that for many learners associating words with pictures aids in learning. Using flashcards with pictures rather than L1 equivalents can be helpful in bypassing native language interference. One of the memory techniques that has proven to be highly successful is the use of images associated with words and meanings to create a “memory palace”. It involves creating a walk-through of the rooms in a building with which one is well acquainted (or can imagine in detail) and associating each item in the room with a memorable image which in some way conjures the word and it is meaning. To recall an item, one walks back mentally through the rooms. One of the memory techniques psychologists have shown to be effective is known as “spaced repetition”. The idea is that there is a particular optimal rhythm for reviewing items to be learned until they are committed to long-term memory. Instead of studying or testing one’s knowledge of a set of items every day, it is better to study them one day, wait perhaps 3 days to study them again, then wait another 7 days after that. Programs that incorporate spaced repetition are set up to keep records of working with sets of words and automatically prompting review at optimal times. There are a number of digital flashcard programs which include that functionality.