Towards Online Training of CNNs: Hebbian-Based Deep Learning
At the Integrated Systems Laboratory (IIS) we have been working on techniques for smart data analytics in ultra-low power sensors for the past few years along the entire technological stack, from HW (e.g. the PULP system) to SW running on microcontrollers – in many cases using convolutional neural networks (CNNs) as the algorithmic “tool” to extract semantic information out of raw data streams. Doing that, it is possible to greatly reduce the amount of data that needs to be collected in a ULP sensor node and sent to a higher-level computing device (e.g. a smartphone, the cloud). However, one of the big inherent limitations of current deep-learning based techniques (including CNNs) is that they must be trained offline with huge datasets, with essentially no possibility to use any knowledge acquired on the field to immediately improve classification accuracy. Recent research [Wadhwa16] has tried to connect a well-known learning principle from neuroscience, Hebbian learning [WIKI], with the techniques established for the training of deep neural network. The objective of this project is to implement the proposed Adaptive Hebbian Learning and/or Discriminative Hebbian Learning techniques on a commodity deep learning framework such as Keras (TensorFlow/Theano) or Torch, and evaluate them on a few well-known CNN topologies and datasets.
Goals of the project
The objective of the project is to implement the Adaptive Hebbian Learning and/or Discriminative Hebbian Learning techniques on a commodity deep learning framework such as Keras (TensorFlow/Theano) or Torch. The project will be composed of the following sub-tasks:
1. acquire familiarity with the chosen deep learning framework (Keras or Torch), with the underlying backend and with the principles of deep learning.
2. implement Adaptive Hebbian Learning (AHL) as proposed in Wadhwa and Madhow [Wadhwa16] within the framework, using classification of a simple MNIST dataset as the target application.
3. evaluate the AHL algorithm on a more complex CNN topology, i.e. ResNet-20 for CIFAR-10 classification.
4. (optional): implement and evaluate also the Discriminative Hebbian Learning (DHL) technique.
The main constraint for the CNN topology definition and training will be to keep the overall size of the model low, so that it can effectively be implemented on a highly memory constrained edge node in a IoT device (so PULP or a microcontroller unit). To this end, a few techniques for reduced-precision training developed at the IIS will be accessible.
To work on this project, you will need:
- basic familiarity with a scripting language for deep learning (Python or Lua…)
- a lot of patience!
- Supervision: Francesco Conti
Detailed Task Description
Within the first month of the project you will be asked to prepare a project plan. This plan should identify the tasks to be performed during the project and sets deadlines for those tasks. The prepared plan will be a topic of discussion of the first week’s meeting between you and your advisers. Note that the project plan should be updated constantly depending on the project’s status.
Meetings & Weekly Report
Weekly meetings will be held between the student and the assistants. The exact time and location of these meetings will be determined within the first week of the project in order to fit the student’s and the assistants schedule. These meetings will be used to evaluate the status and progress of the project. Beside these regular meetings, additional meetings can be organized to address urgent issues as well. Moreover, the student will have to send a (short) written report at the end of every week.
Documentation is an important and often overlooked aspect of engineering. One final report has to be completed within this project. Therefore, the final report of the work will have to be written in English, the de facto common language of electrical engineering and information technology. Any form of word processing software is allowed for writing the reports, nevertheless the use of LaTeX with a vector drawing software for block diagrams is strongly encouraged by the IIS staff (e.g. Inkscape, Tgif, Visio, OmniGraffle). The final report has to be presented at the end of the project and a digital copy need to be handed in.
- [Wadhwa16] A. Wadhwa and U. Madhow, “Bottom-up Deep Learning using the Hebbian Principle,” online preprint. []
- The EDA wiki with lots of information on the ETHZ ASIC design flow (internal only) 
- The IIS/DZ coding guidelines