View source for Knowledge Distillation for Embedded Machine Learning

==Description==
The vast majority of high-performance neural networks used on datasets like ImageNet use Millions or Billions of parameters and are trained and executed with several GPUs at once. Such networks can never be deployed to devices like microcontrollers. 
However, using novel training techniques, we can leverage these well-trained networks to transfer their knowledge to smaller networks that can be deployed to embedded devices.

Knowledge Distillation is a novel training approach for deep neural networks, which uses well-trained large networks or ensembles of specialized models to train smaller, more efficient networks. This technique shows a lot of potential for deploying models to embedded devices when used in conjunction with well-established quantization techniques. The goal of this thesis is to develop a knowledge distillation algorithm and evaluate it for the training of networks for embedded devices, comparing it to traditional training methods.


The main goals (not all have to be met in a single semester project) of the project are:
* Develop framework for distillation-based training in PyTorch

* Combine knowledge distillation with quantization to optimize model size

* Evaluate knowledge distillation as a method for deployment of networks to embedded devices


===Status: Available ===
Looking for a student for a Semester project. 

: Supervision: [[:User:xiaywang|Xiaying Wang]], [[:User:Scheremo|Moritz Scherer]]

===Prerequisites===
* Machine Learning
* Python

===Character===
: 20% Theory
: 80% Implementation

===Literature===
* [https://arxiv.org/abs/1503.02531] G. Hinton, et. al., Distilling the Knowledge in a Neural Network
* [https://arxiv.org/abs/1805.04770] T. Furlanello, et. al., Born Again Neural Networks


===Professor===
: [http://www.iis.ee.ethz.ch/portrait/staff/lbenini.en.html Luca Benini]
[[#top|↑ top]]

<!--
==Detailed Task Description==

===Meetings & Presentations===
The student(s) and advisor(s) agree on weekly meetings to discuss all relevant decisions and decide on how to proceed. Of course, additional meetings can be organized to address urgent issues. 
Around the middle of the project there is a design review, where senior members of the lab review your work (bring all the relevant information, such as prelim. specifications, block diagrams, synthesis reports, testing strategy, ...) to make sure everything is on track and decide whether further support is necessary. They also make the definite decision on whether the chip is actually manufactured (no reason to worry, if the project is on track) and whether more chip area, a different package, ... is provided. For more details confer to [http://eda.ee.ethz.ch/index.php/Design_review]. 
At the end of the project, you have to present/defend your work during a 15 min. presentation and 5 min. of discussion as part of the IIS colloquium (as required for any semester or master thesis at D-ITET).

===Deliverables===
* description of the most promising architectures, and argumentation on the decision taken (as part of the report)
* synthesizable, verified VHDL code
* generated test vector files
* synthesis scripts & relevant software models developed for verification
* synthesis results and final chip layout (GDS II data), bonding diagram
* datasheet (part of report)
* presentation slides
* project report (in digital form; a hard copy also welcome, but not necessary)
--->
<!--
===Timeline==
To give some idea on how the time can be split up, we provide some possible partitioning: 
* Literature survey, building a basic understanding of the problem at hand, catch up on related work 
* Development of a working software-based implementation running on the Zynq's ARM core
* Piece-by-piece off-loading of relevant tasks to the programmable logic
* Implementation of data interfaces (software or hardware)
* Report and presentation 
-->
<!-- 13.5 weeks total here -->

===Practical Details===
* '''[[Project Plan]]'''
* '''[[Project Meetings]]'''
* '''[[Final Report]]'''
* '''[[Final Presentation]]'''

[[#top|↑ top]]

<!-- 

COPY PASTE FROM THE LIST BELOW TO ADD TO CATEGORIES

GROUP
[[Category:Digital]]

STATUS
[[Category:Available]]
[[Category:In progress]]
[[Category:Completed]]
[[Category:Hot]]

TYPE OF WORK
[[Category:Semester Thesis]]
[[Category:Master Thesis]]
[[Category:PhD Thesis]]
[[Category:Research]]

NAMES OF EU/CTI/NT PROJECTS
[[Category:UltrasoundToGo]]
[[Category:IcySoC]]
[[Category:PSocrates]]
[[Category:UlpSoC]]
[[Category:Qcrypt]]

YEAR (IF FINISHED)
[[Category:2010]]
[[Category:2011]]
[[Category:2012]]
[[Category:2013]]
[[Category:2014]]

--->

[[Category:Xiaywang]]
[[Category:Scheremo]]
[[Category:Deep Learning Projects]]
[[Category:Embedded Coding]]
[[Category:Algorithmic]]
[[Category:Digital]][[Category:Semester Thesis]][[Category:2020]][[Category:Hot]]