Personal tools

Design of a Low Power Smart Sensing Multi-modal Vision Platform

From iis-projects

Jump to: navigation, search


Smart Glasses with custom integrated electronics
AI-Node for Multi-modal Sensor Nodes

Overview

Dynamic Vision Sensors (DVS) or also called Event-based cameras can detect (when stationary placed) fast-moving and small objects and open-up tons of new possibilities for AI and tinyML. We are creating a completely new system, with an autonomous base station and distributed smart sensor nodes to run cutting-edge AI algorithms and perform novel sensor fusion techniques.

Project description

Low Power Smart Sensing Devices often consist of a handful of sensors (IMUs, microphones, etc…), a communication module (Wifi, LoRa, BLE), and one or multiple processing units (STM32s, GAP9, or other microcontrollers). However, due to computing and resource limitations, camera sensors are often neglected due to high memory consumption and high computational complexity for on-board processing. Nevertheless, novel processing techniques slowly and steadily transition the world of the Internet of Things (IoT) into Artificial Intelligence of Things (AIoT) devices and as such increase processing capabilities of low-power microcontrollers using dedicated silicon to accelerate ML processing. As such, it started to enable the possibility of adding vision sensors to such a smart sensing platform to not only add value for its reusability but also open the door into the world of tiny-edge AI for vision-based applications. The newest creation of such a platform is shown on the right AI-node, enabling real-time vision processing on microcontrollers, while performing sensor fusion on RGB images and other sensors. In the short term, this platform leads to a smart glasses application, while In the long term, event-based vision sensors will be added to enable real-time responses to events within the sensor’s proximity and thus enable true sensor “smartness” by visually understanding the nodes’ environment.

Your task in this project will be one or several out of the tasks mentioned below. Depending on your thesis (Semester/Master thesis), tasks will be assigned according to your interests and skills.


Tasks:

  • High-end PCB Design for the smart-glasses
  • Embedded Firmware Design for sensors, communication and AI processing
  • AI part (“the smartness”). Develop and deploy networks to run on microcontrollers


Prerequisites (not all needed!) depending of Tasks

  • Embedded Firmware Design and experience in Free RTOS, Zephyr, etc…
  • C-Code programming
  • Circuit design tools (e.g. Altium)
  • Experience in ML on MCU or deep knowledge of ML and strong will to deploy on the edge


Type of work

  • 20% Literature study
  • 60% Software and/or Hardware design
  • 20% Measurements and validation

Status: Available

  • Type: Semester or Master Thesis (multiple students possible)
  • Professor: : Prof. Dr. Luca Benini
  • Supervisors:
Julian Moosmann.jpg

Julian Moosmann

Philippmayer.jpg

Philipp Mayer

  • Currently involved students:
    • None