Latest revision as of 14:15, 1 March 2024

Overview

Status: Available

• Type: Semester Thesis
• Professor: Prof. Dr. L. Benini
• Supervisors:
  • Cristian Cioflan (IIS): cioflanc@iis.ee.ethz.ch
  • Dr. Miguel de Prado (Bonseyes Community Association): miguel.deprado@bonseyes.com
  • Dr. José Miranda Calero (Embedded Systems Laboratory, EPFL): jose.mirandacalero@epfl.ch

Introduction

Human-computer interaction evolved rapidly in the last years thanks to the hardware development enabling complex Deep Neural Networks (DNNs) to perform machine learning tasks. The scientific advancements led to the employment of Automated Speech Recognition (ASR) and Visual Speech Recognition (VSR) in human-machine communication, with the former representing the task of converting spoken speech into written words[1], whilst the latter being the conversion of lip movement into written words (i.e., lipreading)[6]. In the context of voice-controlled devices, it is of uttermost importance to additionally mention speaker separation[7]. This aim to mimic the cocktail party effect present in humans, which denotes our capability of focusing on an individual conversation, whilst filtering out other discussions and surrounding noises. This ensures that only a designated individual can control one target device at a time.

In the current work, we investigate the potential of performing Audio-Visual Speech Separation and Recognition (AVSSR) [2][8][3]. Through multi-task learning, we merge Audio-Visual Speech Separation (AVSS) and Audio-Visual Speech Recognition (AVSR). We propose to fuse audio and visual information, thus increasing the data dimensionality and, implicitly, the amount of useful information. The so-trained model would then provide the target user(s) with the transcript of their respective speech. Lastly, the proposed system must abide by the TinyML[5] constraints considering edge devices, namely reduced memory and storage requirements, as well as real-time operation on low-power, battery-operated devices.

Character

• 20% literature research
• 50% architectural implementation and optimizations
• 30% evaluation

Prerequisites

• Must be familiar with Python.
• Knowledge of deep learning basics, including some deep learning framework like PyTorch or TensorFlow from a course, project, or self-taught with some tutorials.

Project Goals

The main tasks of this project are:

• Task 1: Familiarize yourself with the project specifics (1-2 Weeks)

  Learn about DNN training and PyTorch, how to visualize results with TensorBoard. Read up on data fusion and multimodal learning, common approaches and recent advances on the topic. Read up on multi-task learning in the context of audio-visual time series. Read up on DNN models aimed at time series (e.g., TCNs, TASMs, Transformer and Conformer networks) and the recent advances in AVSR/AVSS.

• Task 2: Propose and evaluate AVSSR topolgies (4-6 weeks)

  Considering state-of-the-art works in AVSS[9] and AVSR[10][11][12] and previous IIS projects, propose and implement AVSSR architectures.

  Propose evaluation metrics and novel loss functions; analyse the models' performance on the GRID dataset.

• Task 3: Optimize proposed models considering TinyML constraints (2-3 weeks)

  Reduce the models' hardware-associated costs (i.e., memory, storage, computational complexity), evaluating the trade-offs on the proposed metrics.

  (Only if conducted as Master's thesis) Deploy the proposed architecture on novel parallel ultra-low-power platforms.

• Task 4: Dataset generalization and ablation study (1-2 Weeks)

  Investigate alternative datasets for AVSSR.

  Propose and implement dataset modifications to enable multi-speaker AVSSR.

  Evaluate and compare multimodal learning against audio- and video-only learning.

  Evaluate and compare speaker-overlapping and speaker-disjoint training and testing.

• Task 5 - Gather and Present Final Results (2-3 Weeks)

  Gather final results.

  Prepare presentation (15 min. + 5 min. discussion).

  Write a final report. Include all major decisions taken during the design process and argue your choice. Include everything that deviates from the very standard case - show off everything that took time to figure out and all your ideas that have influenced the project.

Project Organization

Weekly Meetings

The student shall meet with the advisor(s) every week in order to discuss any issues/problems that may have persisted during the previous week and with a suggestion of next steps. These meetings are meant to provide a guaranteed time slot for mutual exchange of information on how to proceed, clear out any questions from either side and to ensure the student’s progress.

Report

Documentation is an important and often overlooked aspect of engineering. One final report has to be completed within this project. Any form of word processing software is allowed for writing the reports, nevertheless the use of LaTeX with Tgif (See: http://bourbon.usc.edu:8001/tgif/index.html and http://www.dz.ee.ethz.ch/en/information/how-to/drawing-schematics.html) or any other vector drawing software (for block diagrams) is strongly encouraged by the IIS staff.

Final Report

A digital copy of the report, the presentation, the developed software, build script/project files, drawings/illustrations, acquired data, etc. needs to be handed in at the end of the project. Note that this task description is part of your report and has to be attached to your final report.

Presentation

At the end of the project, the outcome of the thesis will be presented in a 15-minutes talk and 5 minutes of discussion in front of interested people of the Integrated Systems Laboratory. The presentation is open to the public, so you are welcome to invite interested friends. The exact date will be determined towards the end of the work.

References