Revision as of 11:24, 14 May 2024

Status: Available

• Type: Semester Thesis
• Professor: Prof. Dr. L. Benini
• Supervisors:
  • Philip Wiese (IIS): wiesep@iis.ee.ethz.ch
  • Victor Kartsch Morinigo (IIS): victor.kartsch@iis.ee.ethz.ch

Introduction

Multi-modal sensing enables on-device feature extraction, reducing communication overhead, perform predictions, increases privacy and enable on-device learning.

Smart sensors, which are sensors equipped with advanced computation capabilities, present a unique opportunity for innovation in multiple critical domains such as environmental sensing to reduce global greenhouse gas emissions or infrastructure monitoring. Our infrastructure is responsible for almost 80% of all global greenhouse gas emissions. Therefore, it is very important to better understand the individual contributors and to be able to build precise models to plan future infrastructure.

A near-sensor processing of the gathered data that directly extracts meaning from the signal can lead to huge improvements in terms of energy efficiency and increased privacy. The reduced energy consumption comes from the fact that more specialized hardware such as the PULP platform including the GAP9 can be used, and, additionally, that the information density of the transmitted data is higher, and thus less data must be transmitted via the often very power-intensive wireless link.

Our long-term goal is to develop a low-power sensing node with very low power consumption. These nodes should incorporate multiple sensors to enable multi-modal sensing. In other words, combine the data of the different sensors to extract additional data. The idea is to use the very efficient GAP9 multicore processor and deploy multi-modal neural networks to perform feature extraction, prediction, anomaly detection, and even on-device learning to adaptively fine-tune the model based on the measured data.

Within the WP3 of the UrbanTwin Project [1], we have designed and produced a compact, low-power, highly efficient computational and sensing node targeting the extraction of features that can model a digital twin of a city. The device aims to process AI-based algorithms on-site, greatly reducing the need for energy-demanding transmission links while addressing data privacy and security concerns. Thanks to its modular design, the system can be easily tailored to various applications while leveraging the computational power and efficiency of the main SoC, the multicore ultra-low-power GAP9 processor. Together with this, a sensing node has been developed targeting the sensing of environmental parameters such as pressure, temperature, VOC, and other sensing devices. Students are invited to contribute to the further development of the platform by creating libraries and characterization tests to assess the device's capabilities. With the computational board's hand, students will also narrow down the work into a specific study case that takes advantage of the sensing method and the computational capabilities of GAP9 to perform edge processing.

Project

• Commissioning of the UTSensorNode: Test base functionalities of the UTSensorNode PCB.
• Base Framework for the UTSensorNode: Development of a supervisor system to manage the board resources and integrate or develop drivers for different sensors.
• Application Specific Device Optimization: Select an interesting application (air, image, sound...), investigate signal feature extraction, classification, and if possible on-device learning.

Prerequisites

• Fundamental deep learning concepts.
• PCB Design and embedded C programming.
• Familiarity with Python programming.
• Familiarity with the PyTorch deep learning framework.

References

[1] Urban Twin Project