Personal tools

Difference between revisions of "Object Detection and Tracking on the Edge"

From iis-projects

Jump to: navigation, search
(Created page with "<!-- Creating Object Detection and Tracking on the Edge-> 200px|thumb|right|Example of Drone Detection, comparing DVS and RGB images = Overview = Dy...")
 
(Project description)
 
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
<!-- Creating Object Detection and Tracking on the Edge->
+
<!-- Creating Object Detection and Tracking on the Edge -->
  
 
[[File:Dvs_drones.png|200px|thumb|right|Example of Drone Detection, comparing DVS and RGB images]]
 
[[File:Dvs_drones.png|200px|thumb|right|Example of Drone Detection, comparing DVS and RGB images]]
 +
  
 
= Overview =
 
= Overview =
Line 8: Line 9:
  
 
= Project description =
 
= Project description =
Nowadays, Object detection has switched from classical approaches of finding handcrafted features within an image, to AI approaches. With the hardware speedup GPUs are delivering, these deep neural networks can run incredibly fast and achieve detection accuracy comparable or even superior to humans. However, when it comes to scenarios with low or high brightness, the dynamic range of standard CCD/CMOS cameras perform poorly and as such the networks start to fail. One approach to overcome these problems is to use a novel camera-sensor, called a dynamic vision Sensor, short DVS. These sensors do not record the intensity of pixels, instead, they record intensity changes, similar to a human’s eye. With this novel technology, a dynamic range of >120 dB can be achieved, which is comparable to a human’s eye. We aim of developing new object detection and tracking algorithms (for UAVs, Cars and People), use of bio-inspired processing hardware, implement a new stereo-vision algorithm, attacking existing algorithms with adversarial attacks and building up a new autonomous detection system. This can involve using/programming an MCU, Nvidia Jetson Orin, custom ASIC or even neuromorphic computing platforms.
+
Nowadays, Object detection has transitioned from classical approaches involving handcrafted feature extraction of images to advanced  AI approaches. With the accelerated processing capabilities delivered by GPUs, these deep neural networks run incredibly fast and achieve detection accuracy comparable to or even superior to humans. However, when it comes to scenarios with low or high brightness, the dynamic range of standard CCD/CMOS cameras performs poorly and as such the networks start to fail. One approach to overcome this problem is to use a novel camera-sensor, called a dynamic vision Sensor, short DVS. These sensors do not record the intensity of pixels, instead, they record individual pixel intensity changes, similar to a human’s eye. With this novel technology, a dynamic range of >120 dB can be achieved. While literature is still in debate of the optimal approach to dealing with event data, we’re considering digital compute platforms for processing and as such developing new algorithms with a frame-based approach. However, this thesis aims to find new algorithm approaches for object detection and tracking (of e.g. UAVs, Cars, and People). The goal of such an algorithm is to be small-sized to fit edge computing platforms such as the  Nvidia Jetson Orin, while ultimately end-up in the field of vision-based Artificial Intelligence of Things (AIoT) devices and therefore deploying the network on specialized microcontrollers.
 +
 
 +
Your task in this project will be one or several of the tasks mentioned below. Depending on your thesis (Semester/Master thesis), tasks will be assigned according to your interests and skills.  
  
Your task in this project will be one or several of the tasks mentioned below. Depending on your thesis (Semester/Master thesis), tasks will be assigned accordingly to your interests and skills.
 
  
 
== Tasks: ==
 
== Tasks: ==
 
* Event and Frame-based and/or 3D Imaging  
 
* Event and Frame-based and/or 3D Imaging  
 
* Parallel Programming from MCU to Nvidia Jetson Orin
 
* Parallel Programming from MCU to Nvidia Jetson Orin
* Adversarial Attacks
 
* New Object Detection Algorithms or Neuromorphic Algorithms
 
  
  
Line 57: Line 57:
 
** None
 
** None
  
[[Category:Available]] [[Category:Digital]] [[Category:Event-Driven Computing]] [[Category:Deep Learning Projects]] [[Category:EmbeddedAI]] [[Category:SmartSensors]] [[Category:System Design]] [[Category:2023]] [[Category:Semester Thesis]] [[Category:Master Thesis]] [[Category:Julian]] [[Category:Hot]]
+
[[Category:Available]] [[Category:Digital]] [[Category:Event-Driven Computing]] [[Category:Deep Learning Projects]] [[Category:EmbeddedAI]] [[Category:SmartSensors]] [[Category:System Design]] [[Category:2023]] [[Category:Semester Thesis]] [[Category:Master Thesis]] [[Category:Julian]] [[Category:Mayerph]] [[Category:Hot]]

Latest revision as of 09:55, 5 December 2023


Example of Drone Detection, comparing DVS and RGB images


Overview

Dynamic Vision Sensors (DVS) or also called Event-based cameras can detect (when stationary placed) fast-moving and small objects and open-up tons of new possibilities for AI and tinyML. We are creating a completely new system, with an autonomous base station and distributed smart sensor nodes to run cutting-edge AI algorithms and perform novel sensor fusion techniques.

Project description

Nowadays, Object detection has transitioned from classical approaches involving handcrafted feature extraction of images to advanced AI approaches. With the accelerated processing capabilities delivered by GPUs, these deep neural networks run incredibly fast and achieve detection accuracy comparable to or even superior to humans. However, when it comes to scenarios with low or high brightness, the dynamic range of standard CCD/CMOS cameras performs poorly and as such the networks start to fail. One approach to overcome this problem is to use a novel camera-sensor, called a dynamic vision Sensor, short DVS. These sensors do not record the intensity of pixels, instead, they record individual pixel intensity changes, similar to a human’s eye. With this novel technology, a dynamic range of >120 dB can be achieved. While literature is still in debate of the optimal approach to dealing with event data, we’re considering digital compute platforms for processing and as such developing new algorithms with a frame-based approach. However, this thesis aims to find new algorithm approaches for object detection and tracking (of e.g. UAVs, Cars, and People). The goal of such an algorithm is to be small-sized to fit edge computing platforms such as the Nvidia Jetson Orin, while ultimately end-up in the field of vision-based Artificial Intelligence of Things (AIoT) devices and therefore deploying the network on specialized microcontrollers.

Your task in this project will be one or several of the tasks mentioned below. Depending on your thesis (Semester/Master thesis), tasks will be assigned according to your interests and skills.


Tasks:

  • Event and Frame-based and/or 3D Imaging
  • Parallel Programming from MCU to Nvidia Jetson Orin


Prerequisites (not all needed!) depending of Tasks

  • Embedded Firmware Design and experience in Free RTOS, Zephyr, etc…
  • Experience in Machine Learning and/or neuromorphic computing
  • Parallel programming


Type of work

  • 20% Literature study
  • 60% Software and/or Hardware design
  • 20% Measurements and validation

Status: Available

  • Type: Semester or Master Thesis (multiple students possible)
  • Professor: : Prof. Dr. Luca Benini
  • Supervisors:
Julian Moosmann.jpg

Julian Moosmann

Philippmayer.jpg

Philipp Mayer

  • Currently involved students:
    • None