Hiippala, Tuomo
(Helsingin yliopisto, 2013)
This dissertation studied the structure of multimodal artefacts, or how language, image and other semiotic modes combine and interact in documents. This places the study within the emerging field of multimodal research, which uses linguistic methods to study the interaction of multiple semiotic modes.
Despite the growing amount of multimodal research, the structure of multimodal artefacts has not received the attention it warrants. Previous studies have been either very detailed or exceedingly abstract, leaving a significant gap between data and theory, which this dissertation attempted to bridge. To do so, the dissertation adopted a data-driven approach to multimodal analysis, addressing the structure of multimodal artefacts, the factors that shape the artefact structure, and the role of structure in the recognition and interpretation of the artefacts.
The data consisted of tourist brochures produced by the city of Helsinki between 1967 and 2008, which allowed a longitudinal perspective to their multimodal structure. A total of 58 double-pages were annotated for their content, visual appearance, layout and rhetorical organisation, and compiled into an XML-based multimodal corpus. To study the corpus, the dissertation developed visualisation methods that combined information from multiple analytical layers of the corpus to represent the multimodal structures in the data.
The study revealed the functional motivation behind the structure of the tourist brochures, identifying patterns in their hierarchical and rhetorical organisation, which were used to fulfil specific communicative tasks. The configuration of these patterns, in turn, signalled how the brochure was to be interpreted. The results also showed that after the year 1985, which marked the introduction of desktop publishing software, the organising principles of the tourist brochures have shifted towards a more fragmented and non-linear structure.