Design of a Low Power Smart Sensing Multi-modal Vision Platform

Example of a Mulit-modal Sensor Node

Overview

Dynamic Vision Sensors (DVS) or also called Event-based cameras can detect (when stationary placed) fast-moving and small objects and open-up tons of new possibilities for AI and tinyML. We are creating a completely new system, with an autonomous base station and distributed smart sensor nodes to run cutting-edge AI algorithms and perform novel sensor fusion techniques.

Project description

Low Power Smart Sensing Devices often consist of a handful sensors such as IMUs, microphones, temperature and humidity sensors, barometer sensors, a communication module, to communicate with the cloud (Wifi, LoRa, BLE) and one or multiple processing units (STM32s, GAP9 or other microcontrollers). However, due to computing and resource limitations, camera sensors are often neglected. Adding vision to such a smart sensing platform not only adds value to its reusability but also opens the door into the world of tiny-edge AI for vision-based applications. IIS and PBL are creating such a new platform to not only enable real-time vision processing on microcontrollers, instead, we also want to push boundaries and perform sensor-fusion with RGB images and other sensors. In the long term, event-based vision sensors will be added to enable real-time responses to events within the sensor’s proximity and thus enable true sensor “smartness” by visually understanding the nodes understand its environment

Your task in this project will be one or several out of the tasks mentioned below. Depending on your thesis (Semester/Master thesis), tasks will be assigned accordingly to your interests and skills.

Tasks:

• PCB Design of the Platform
• Embedded Firmware Design for sensor read-out and wireless communication
• AI part (“the smartness”). Develop and deploy networks to run on microcontrollers

Prerequisites (not all needed!) depending of Tasks

• Embedded Firmware Design and experience in Free RTOS, Zephyr, etc…
• C-Code programming
• Circuit design tools (e.g. Altium)
• Experience in ML on MCU or deep knowledge of ML and strong will to deploy on the edge

Type of work

• 20% Literature study
• 60% Software and/or Hardware design
• 20% Measurements and validation

Status: Available

• Type: Semester or Master Thesis (multiple students possible)
• Professor: : Prof. Dr. Luca Benini
• Supervisors:

Julian Moosmann e-mail: julian.moosmann@pbl.ee.ethz.ch phone: +41 79 328 15 91 skype: mojulian_2 Office: ETF F 110

Philipp Mayer e-mail: mayerph@iis.ee.ethz.ch phone: +41 44 63 242 68 skype: mayer.philipp1 Office: ETF F 108

• Currently involved students:
  • None