Personal tools

Object Detection and Tracking on the Edge

From iis-projects

Jump to: navigation, search


Example of Drone Detection, comparing DVS and RGB images


Overview

Dynamic Vision Sensors (DVS) or also called Event-based cameras can detect (when stationary placed) fast-moving and small objects and open-up tons of new possibilities for AI and tinyML. We are creating a completely new system, with an autonomous base station and distributed smart sensor nodes to run cutting-edge AI algorithms and perform novel sensor fusion techniques.

Project description

Nowadays, Object detection has transitioned from classical approaches involving handcrafted feature extraction of images to advanced AI approaches. With the accelerated processing capabilities delivered by GPUs, these deep neural networks run incredibly fast and achieve detection accuracy comparable to or even superior to humans. However, when it comes to scenarios with low or high brightness, the dynamic range of standard CCD/CMOS cameras performs poorly and as such the networks start to fail. One approach to overcome this problem is to use a novel camera-sensor, called a dynamic vision Sensor, short DVS. These sensors do not record the intensity of pixels, instead, they record individual pixel intensity changes, similar to a human’s eye. With this novel technology, a dynamic range of >120 dB can be achieved. While literature is still in debate of the optimal approach to dealing with event data, we’re considering digital compute platforms for processing and as such developing new algorithms with a frame-based approach. However, this thesis aims to find new algorithm approaches for object detection and tracking (of e.g. UAVs, Cars, and People). The goal of such an algorithm is to be small-sized to fit edge computing platforms such as the Nvidia Jetson Orin, while ultimately end-up in the field of vision-based Artificial Intelligence of Things (AIoT) devices and therefore deploying the network on specialized microcontrollers.

Your task in this project will be one or several of the tasks mentioned below. Depending on your thesis (Semester/Master thesis), tasks will be assigned according to your interests and skills.


Tasks:

  • Event and Frame-based and/or 3D Imaging
  • Parallel Programming from MCU to Nvidia Jetson Orin


Prerequisites (not all needed!) depending of Tasks

  • Embedded Firmware Design and experience in Free RTOS, Zephyr, etc…
  • Experience in Machine Learning and/or neuromorphic computing
  • Parallel programming


Type of work

  • 20% Literature study
  • 60% Software and/or Hardware design
  • 20% Measurements and validation

Status: Available

  • Type: Semester or Master Thesis (multiple students possible)
  • Professor: : Prof. Dr. Luca Benini
  • Supervisors:
Julian Moosmann.jpg

Julian Moosmann

Philippmayer.jpg

Philipp Mayer

  • Currently involved students:
    • None