On-Device Federated Continual Learning on Nano-Drone Swarms
From iis-projects
Contents
Overview
Status: Available
- Type: Semester Thesis
- Professor: Prof. Dr. L. Benini
- Supervisors:
Introduction
The increasing demand for machine learning at the edge, near the sensor, lead to machine learning inference being performed directly on edge devices with limited memory and computational resources -- an approach often referred to as TinyML[5]. Together with inference, the neural networks' training paradigm is also challenged. Rather than following a train-once-deploy-everywhere approach, continual learning[3][7] enables models deployed on devices to learn incrementally, adapting to new data or tasks over time. This approach is crucial for applications in dynamic environments where data distributions change or new tasks emerge. One of the challenges that such models must overcome is catastrophic forgetting, thus acquiring new knowledge without corrupting previously learned one. Federated learning[4] further enhances edge-adaptable models by allowing a collective, distributed learning process across multiple devices, exchanging knowledge without exchanging raw data. This is crucial for privacy-sensitive applications and in scenarios where data cannot be centralized due to bandwidth or regulatory constraints.
Federated Continual Learning tasks[1][8] face unique challenges at the extreme edge, on autonomous nano-drones. Given the limited memory, processing power, and the requirement for real-time computation, it is mandatory to develop efficient learning algorithms capable of solving real-world problems such as face recognition. This scenario becomes even more complex when considering a swarm of drones[2], where each of them can encounter different faces in diverse environments. Through federated learning, the collective knowledge contained in a shared model can be individually updated, without sharing sensitive information such as the raw biometrical data. This expands the number of recognizable faces over time, while ensuring that the learning process is efficient and adaptable to the ever-changing environments in which the nano-drones operate.
Character
- 10% literature research
- 50% deep learning
- 30% on-device implementation
- 10% evaluation
Prerequisites
- Must be familiar with Python, in the context of deep learning and quantization
- Must be familiar with C, for layer implementation and model deployment
Project Goals
The main tasks of this project are:
Task 1: Familiarize yourself with the project specifics (1-2 Weeks)
Learn about DNN training and PyTorch, how to visualize results with TensorBoard.
Read up on Continual Learning and Federated Learning, in the context of extreme edge devices.
Familiarize yourself with the PULP-TrainLib[6] on-device learning framework.
Task 2 - Develop an efficient Federated Continual Learning for Face Recognition. (5-6 weeks)
Create a PyTorch framework for face recognition, including Datasets and Dataloaders. Expand it considering the specific data management required in Continual Learning.
Propose a continual learning algorithm and benchmark it considering neural backbones feasible for the extreme edge, such as MobileNetv2.
Expand the algorithm to enable federated learning.
Task 3 - Deploy FCL algorithm on PULP platforms (5-6 weeks)
Evaluate the proposed algorithm considering hardware-associated metrics (e.g., latency, parameters, memory). Optimize the algorithm considering an accuracy-complexity trade-off.
Deploy the pretrained model, expanding PULP-TrainLib to include the proposed learning algorithm.
Perform in-field evaluation of the FCL algorithm considering a simple, two-drone system. Intermediate evaluation of CL could be beneficial for this task.
Task 4 - Gather and Present Final Results (2-3 Weeks)
Gather final results.
Prepare presentation (15/20 min. + 5 min. discussion).
Write a final report. Include all major decisions taken during the design process and argue your choice. Include everything that deviates from the very standard case - show off everything that took time to figure out and all your ideas that have influenced the project.
Project Organization
Weekly Meetings
The student shall meet with the advisor(s) every week in order to discuss any issues/problems that may have persisted during the previous week and with a suggestion for the next steps. These meetings are meant to provide a guaranteed time slot for a mutual exchange of information on how to proceed, clear out any questions from either side and ensure the student’s progress.
Report
Documentation is an important and often overlooked aspect of engineering. One final report has to be completed within this project. Any form of word processing software is allowed for writing the reports, nevertheless, the use of LaTeX with Tgif (See: http://bourbon.usc.edu:8001/tgif/index.html and http://www.dz.ee.ethz.ch/en/information/how-to/drawing-schematics.html) or any other vector drawing software (for block diagrams) is strongly encouraged by the IIS staff.
Final Report
A digital copy of the report, the presentation, the developed software, build script/project files, drawings/illustrations, acquired data, etc. needs to be handed in at the end of the project. Note that this task description is part of your report and has to be attached to your final report.
Presentation
At the end of the project, the outcome of the thesis will be presented in a 15-minute/20-minute talk and 5 minutes of discussion in front of interested people of the Integrated Systems Laboratory. The presentation is open to the public, so you are welcome to invite interested friends. The exact date will be determined towards the end of the work.
References
[1] Jiahua Dong and Lixu Wang and Zhen Fang and Gan Sun and Shichao Xu and Xiao Wang and Qi Zhu Federated Class-Incremental Learning 2022.
[2] Carl Friess and Vlad Niculescu and Tommaso Polonelli and Michele Magno and Luca Benini Fully Onboard SLAM for Distributed Mapping with a Swarm of Nano-Drone. 2023.
[3] M. De Lange and R. Aljundi and M. Masana and S. Parisot and X. Jia and A. Leonardis and G. Slabaugh and T. Tuytelaars A Continual Learning Survey: Defying Forgetting in Classification Tasks 2022.
[4] McMahan, Brendan and Moore, Eider and Ramage, Daniel and Hampson, Seth and Arcas, Blaise Aguera y Communication-Efficient Learning of Deep Networks from Decentralized Data 2017.
[5] Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Osborne, Gennady Pekhimenko, Arun Tejusve Raghunathm Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, and Yuchen Zhou. “MLperf inference benchmark. 2020
[6] Nadalini, Davide and Rusci, Manuele and Tagliavini, Giuseppe and Ravaglia, Leonardo and Benini, Luca and Conti, Francesco PULP-TrainLib: Enabling On-Device Training for RISC-V Multi-core MCUs Through Performance-Driven Autotuning 2022
[7] Liyuan Wang and Xingxing Zhang and Hang Su and Jun Zhu A Comprehensive Survey of Continual Learning: Theory, Method and Application 2024
[8] Jaehong Yoon and Wonyong Jeong and Giwoong Lee and Eunho Yang and Sung Ju Hwang Federated Continual Learning with Weighted Inter-client Transfer 2021
