AI in Language and Literacy Education: A Multimodal Future
Generative artificial intelligence (GenAI) has permeated second language education at an extraordinary rate, but thus far there has been little agreement on what it actually means for language teaching and learning.
AI for Multimodal Literacy
For Professor George Jiang, Assistant Professor in the Faculty’s Academic Unit of Language and Literacy Education, AI neither poses an existential crisis to second language education nor offers a simple technological solution to long-lasting pedagogical challenges. Instead, it reconfigures language and literacy education as multimodal and digital rather than merely language-centric and paper-bound. Professor Jiang’s work identifies three key areas in which AI can support this shift: multimodal literacy, multimodal assessment, and critical digital literacy.
“Multimodal literacy is high on the policy and curricular agenda of language education in Hong Kong,” Professor Jiang notes. More broadly, the term multimodal literacy refers to the ability to make meaning with language in combination with other semiotic modes, such as image and audio. In language classrooms, teachers can foster students’ multimodal literacy by guiding them to engage with and create multimodal texts. However, this process is often overlooked, because it can be labour-intensive and may shift students’ attention away from language use towards technological processes.
Students prompting with AI for creative images.
To address this, a novel solution was implemented by Professor Jiang and his team in a project on multimodal literacy development. Their aim was to guide teachers to integrate conversational and multimodal GenAI into digital multimodal composing. First, they helped teachers guide students in using GenAI to generate and gather basic elements for multimodal text creation. These included scripts, visuals, soundtracks, and on-screen captions.
This approach relieved teachers and students of the labour of, for instance, filming and editing, and gave them access to synthetic embodied learning through simulated experiences and virtual avatars.
The next step was to help students engage more productively with AI. “We highlighted prompting as a new space in which to teach students new ways of using language to communicate and interact with AI for an intended output,” explains Professor Jiang. Such output might include a script, a plan, an image, an animation, or a short video clip.
The team developed an original multimodal prompting framework to support teachers in guiding students to describe an intended output with linguistic knowledge, visual elements, and sociocultural awareness. Through prompting and interacting with multimodal GenAI tools, teachers’ and students’ attention was redirected to language use and content development, capitalising on AI’s remarkable visualisation power. Potential biases in AI-generated content were mitigated through ongoing evaluation and refinement.
In these two ways, the team was able to support multimodal literacy development in contemporary language and literacy classrooms.
AI for Multimodal Assessment
GenAI’s ability to generate human-like content has also raised questions about assessment in language education. Tasks such as essay writing are becoming more difficult to evaluate reliably, given the ease with which AI tools can generate text. There is a pressing need to develop alternative forms of assessment in language classrooms, according to Professor Jiang.
“To prepare language teachers for AI-supported assessment innovations,” he explains, “we supported teachers’ effective use of AI for multimodal assessment task design, implementation, and evaluation across four key learning stages.” This was done in projects funded by the Education Bureau of the Hong Kong Government.
The team considered AI-supported multimodal assessment a timely response to the need for assessment innovations in today’s language and literacy classrooms. Multimodal assessment refers to assessment tasks that invite students to demonstrate their learning through multiple modes of communication, which include language. It allows students to represent meaning and learning with all of their linguistic, cultural, and communicative repertoires. Examples of multimodal assessment tasks include infographic abstracts, e-book narratives, and video documentaries.
For instance, rather than asking students only to write an essay, a teacher might ask them to create a short video documentary. Students would still be required to plan, write, speak, edit, and evaluate language, while also learning how images, subtitles, and voice work together to shape and communicate meaning across different modes.
“To prepare language teachers to cater for diverse learning needs, we developed an original literacy continua model to specify clearly how classroom-based multimodal assessment can be designed across different key learning stages by attending to content, genre, mode, and design.”
Regarding content, the team guided teachers to set literacy purposes for multimodal assessment according to their classroom contexts. Next, they worked with teachers to identify purpose-specific genres as the target outputs of the classroom-based multimodal assessment. The teachers then considered modal accuracy, density, and cohesion to diversify task complexity as appropriate for their target learners. In the final dimension, design, the researchers suggested three patterns of AI use (automate, assist, augment) to facilitate multimodal meaning-making.
Using this literacy continua model, the team supported language teachers in designing and implementing coherent classroom-based multimodal assessment tasks across four key learning stages in primary and secondary education.
To help teachers evaluate students’ performance on multimodal assessment tasks, the team developed a genre-based model and rubrics that break evaluation into five distinct and assessable layers, including purpose, base unit, layout, navigation flow, and rhetoric. They further guided teachers to adapt the genre-based rubrics into peer feedback forms, based on which a series of formative assessment activities can be designed.
A student interacting with an AI tool in a multimodal assessment task.
AI for Critical Digital Literacy
The effective use of AI in second language education depends largely on whether and how critical digital literacies are supported and developed, particularly for minoritised students with linguistically and culturally diverse backgrounds.
“Students of ethnic minority backgrounds face additional challenges when using AI for learning English as a second or additional language,” Professor Jiang points out, “due to the unequal and uneven acceptance of those students’ home languages and cultures as learning resources by mainstream AI platforms.”
Not only are such platforms often modelled on English usage in mainstream Western societies, but language teachers – in Hong Kong and beyond – may not be fully equipped to teach critical digital literacies in today’s AI-saturated world.
Professor Jiang and his team addressed this challenge in a General Research Fund project supported by the Research Grants Council, in which they developed and validated a four-dimensional framework to prepare language teachers to teach critical digital literacies to Hong Kong students learning English as a second language.
The framework conceptualises the teaching of digital literacies in four dimensions: meaning-making as multimodal design; identity-related relationship building; supporting digital activism; and unpacking algorithmic biases in AI platforms. In each of these dimensions, the framework clearly specifies the knowledge and teaching strategies needed by language teachers.
“This framework was validated through two cycles of design-based research,” says Professor Jiang, “and the research findings have also been used to inform the design of literacy education courses in our teacher education programmes.”
Looking Ahead
Taken together, these initiatives point to a gradual shift in language and literacy education. AI is not replacing existing practices, but it is changing how language is used, taught, and assessed, opening up new possibilities for learning and communication. At the same time, careful implementation remains essential. Supporting teachers and students to engage with AI in thoughtful and informed ways will be key to widening its benefits.
Professor George Jiang
Associate Professor
Academic Unit of Language and Literacy Education
