Realtime Gaze Tracking on Siracusa
From iis-projects
Overview
Augmented Reality/Virtual Reality wearables are gaining momentum due to their potential to revolutionize how we perceive the world. Fitting smartphone-like processing capabilities into a lightweight form factor is quite challenging. The real-time processing constraints in a limited power budget require a novel approach across the entire stack, system-level design, and optimization. These AR/VR wearable systems use multiple sensors to deliver an immersive experience through various algorithms.
One of the very common and important workloads is real-time eye gaze tracking. This helps to predict the eye gaze location of the user and thus could be exploited to render only the area where the user is looking at high resolution and render other areas at low resolution[1]. Eye gaze tracking is crucial for an immersive edge-device experience and requires fast and real-time prediction. Recently, there has been research to build lightweight Regions of Interest prediction[2]. However, the execution latency of the application depends on the temporal data distribution fetched from the sensor, the type of algorithm, the image resolution, and the underlying hardware architecture on which the algorithm would be deployed.
Status: Available
- Type: Semester Thesis
- Professor: Prof. Dr. L. Benini
- Supervisors:
Project Description
In [2], an event-driven approach is used to predict the RoIs. We have integrated Siracusa into GVSoC[3]- a highly configurable, fast and accurate full-platform simulator for RISC-V-based IoT Processors. An emulation model of one of our latest chips, Siracusa[4], is integrated into the GVSoC framework. The project aims to evaluate the performance of these gaze-tracking algorithms on the GVSoC Siracusa platform.
1. Develop application and Map the eye gaze tracking[2] on the GVSoC implementation of Siracusa
2. Evaluate the algorithm performance by exploring different resolutions, activities, etc.
3. If needed, extend the GVSoC model to accelerate part of the execution pipeline in the given algorithm
Workload Distribution
- 10% Literature research
- 20% Extending GVSoC model
- 60% Workload creation and deployment
- 10% Verification
Required Skills
To work on this project, you will need the following:
- Proficient in C, C++, Python
- Familiarity with PyTorch
- Strong interest in computer architecture.
- Familiarity with Deep Learning and traditional DSP algorithms
- To have worked in the past with at least one RTL language (SystemVerilog or Verilog or VHDL) - having followed (or actively following during the project) the VLSI1 / VLSI2 courses is strongly recommended.
References
[1] M. Abrash, "Creating the Future: Augmented Reality, the next Human-Machine Interface," 2021 IEEE International Electron Devices Meeting (IEDM), San Francisco, CA, USA, 2021, pp. 1.2.1-1.2.11, doi: 10.1109/IEDM19574.2021.9720526.
[2] Feng, Yu et al. "Real-time gaze tracking with event-driven eye segmentation." 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 2022.
[3] Bruschi, Nazareno, et al. "GVSoC: a highly configurable, fast and accurate full-platform simulator for RISC-V based IoT processors." 2021 IEEE 39th International Conference on Computer Design (ICCD). IEEE, 2021.
