Open Student Projects

Computer Vision Pipeline for Terapixel Visual Heritage

Master projects (DH/IC)

Supervisor: Tsz Kin Chau and Sarah Kenderdine

Project description:

Computer vision has been applied to very large and/or intricate visual objects with techniques such as tiling object detection, multi-resolution object detection, and bounding box fusion. While these techniques have seen noticeable success in domains such as remote sensing, pedestrian tracking, and pathology detection, they have yet to be developed in Cultural Heritage. The Panorama of the Battle of Murten is the world’s largest image of a single object, measuring 1.6 terapixels. Its physical size (10×100 m²) and the depicted elements, such as armaments, characters, heraldic representations, and costumes, are incredibly rich and intricate. An object detection pipeline for large, intricate visual heritage can deliver a rich inventory of these entities, potentially opening up a new epistemological opportunity to understand how and what the painting represents and transmits visually, through Topological Data Analysis or Topological Narratives (Rodríguez-Ortega, 2022).

The core objective is to identify a pipeline for object detection in intricate visual heritages with a thorough evaluation, where further downstream tasks such as Distant Viewing, Pose Detection, and Segmentation can follow. The student is expected to learn how to tackle the issues of lack of training data and domain shift problems common in cultural heritage.

Main activities:

• Development of a labelled dataset using the image of the Panorama of the Battle of Murten. The student will be given a list of classes of interest.
• Review and experiment with various multi-resolution object detection pipelines using state-of-the-art foundation vision models.
• The student is welcome to suggest improvements to the pipeline.
• Evaluate the performance of these pipelines using the developed dataset.

Prerequisite: Excellent Python Skills, Experience in Computer Vision

The project offers the opportunity to process the world’s largest image of a single object (1.6 terapixels) and experiment with newly emerged object detection methods using foundation vision models.

Reference:

Rodríguez-Ortega, N. (2022). Techno-Concepts for the Cultural Field: N-Dimensional Space and Its Conceptual Constellation. Multimodal Technologies and Interaction, 6(11), Article 11. https://doi.org/10.3390/mti6110096

Swiss cultural history of the Middle Ages: Augmenting the Panorama of the Battle of Murten

Master projects (DH)

Supervisor: Jaquet Daniel Norbert and Sarah Kenderdine

Project description:

The Panorama of the Battle of Murten was painted in 1893 by a team of artists led by the military painter Louis Braun. This gigantic painting measures 10m by 100m (1000m2 of painting). It depicts the moment that Swiss Confederates gained the upper hand against the Duchy of Burgundy during its 1476 invasion, a turning point in European history. specialised in public entertainment with this main media of the end of the 19th century (an early form of immersive experiences before cinema in rotunda theatres in Zürich and Geneva). The EPFL project Diagram is currently imaging the painting as the largest dataset ever created for a single object – 1.600 terapixel (phase 1 2022-2023) and enters the phase of augmenting the painting for immersive experiences (phase 2 2023-2026).

As a DH master’s student, you will work with a team of researchers and data scientists to provide additional data for the augmentation of the digital twin. A range of options is available:

• Production of narratives and metadata based on original archival documents;
• Documentation and metadata based on iconographical representations of the battle, the objects, the weapons and the characters;
• 3D models of preserved material culture (mainly late Medieval weapons);
• 3D animations of reconstructions of fighting movements and costume patterns.

 

Prerequisite: A motivation to learn history through a late 19^th-century representation of a 15^th-century battle; Interest in objects and costumes of the Middle Ages, as well as in interactive immersive museum installations; [optional] Basic notions of Medieval History and Medievalism (contemporary reception of the Middle Ages); [optional] Notion of 3D modelling and 3D animation; [optional] Notion of semantic annotation.

Creative Exploration of Natural Language Processing for the Design of a Virtual Reality Installation

Masters Thesis Project (DH/IC)

Supervisor: Dhruva Gowda Storz and Prof. Sarah Kenderdine

Project description:

The Laboratory for Experimental Museology and the Visualization Research Center in Hong Kong have acquired a collection of 100 digitized stereographic photographs, along with a corpus of text describing each one of them in the context of the author’s journey through China in the wake of the boxer rebellion. Our lab aims to design an interactive virtual reality installation for exploring this dataset, through an investigation of how new narrative threads or interaction modalities can be created through linking the text corpus with images through natural language processing techniques. 

This interdisciplinary master thesis is a vital part of our aims with this dataset, and involves an unbiased creative exploration of various NLP techniques of various levels of complexity (TFIDF, word2vec, NER, CLIP, LLMs, RAG, etc) applied to our dataset, to determine their potential for linking text to images in both functional and expressive ways, and imagining how these links could be implemented in an interactive installation. 

The thesis presents the students with the opportunity to engage with their engineering practice from a creative perspective, and is well suited for those who are interested in the interface between art and engineering. As it surveys various NLP methods, this project is a great opportunity to learn NLP from the ground up, or push the boundaries of your NLP knowledge if you are experienced. 

Main activities:

• Survey various NLP algorithms applied to our dataset to see what possibilities they present for the creation of interesting interactive experiences in VR
• Conceive of immersive visualization experiences on our stereoscopic virtual reality systems based on your exploration

Prerequisite: 

Python Skills, basic machine learning knowledge, Interest in new media art, creative mindset.