Personal tools

Difference between revisions of "IBM Research"

From iis-projects

Jump to: navigation, search
(In-Memory Computing (IMC))
(Hybrid AI Systems (HAS))
(10 intermediate revisions by the same user not shown)
Line 20: Line 20:
 
The location in Zurich is one of IBM’s 12 global research labs. IBM has maintained a research laboratory in Switzerland since 1956. As the first European branch of IBM Research, the mission of the Zurich Lab, in addition to pursuing cutting-edge research for tomorrow’s information technology, is to cultivate close relationships with academic and industrial partners, be one of the premier places to work for world-class researchers, to promote women in IT and science, and to help drive Europe’s innovation agenda. [https://www.zurich.ibm.com/pdf/ZRL_Fact_Sheet.pdf Download factsheet]
 
The location in Zurich is one of IBM’s 12 global research labs. IBM has maintained a research laboratory in Switzerland since 1956. As the first European branch of IBM Research, the mission of the Zurich Lab, in addition to pursuing cutting-edge research for tomorrow’s information technology, is to cultivate close relationships with academic and industrial partners, be one of the premier places to work for world-class researchers, to promote women in IT and science, and to help drive Europe’s innovation agenda. [https://www.zurich.ibm.com/pdf/ZRL_Fact_Sheet.pdf Download factsheet]
  
==Hyperdimensional Computing (HDC)==
+
==Hybrid AI Systems (HAS)==
  
 
[[File:NatureElectronics20.jpg|thumb|right|200px]]
 
[[File:NatureElectronics20.jpg|thumb|right|200px]]
Hyperdimensional computing (HDC) is a brain-inspired computing paradigm based on representing information with hypervectors with dimensions in the thousands. Hypervectors are holographic and (pseudo)random with independent and identically distributed (i.i.d.) components. Principles of HDC allow implementing efficient machine learning models, where it has shown few-shot learning capabilities by requiring less training data than conventional machine learning models to do accurate predictions. By its very nature, HDC is extremely robust against failures, defects, variations, and noise, all of which are complementary to ultra-low energy computation on nanoscale fabrics such as Phase-Change Memory (PCM) devices.  By eliminating non von-Neumann bottleneck, PCM-based in-memory computing systems enable building systems that can run HDC models at very high energy efficiencies. This opens doors to a vast spectrum of further research in applications, algorithms, architecture and system development.
+
Neither symbolic AI nor neural networks alone has produced the kind of intelligence expressed in human and animal behavior. Why? Each has a long and rich history, but has addressed a relatively narrow aspect of the problem. Symbolic AI focuses on solving cognitive problems, drawing upon the rich framework of symbolic computation to manipulate internal representations in order to perform reasoning and inference. But it suffers from being non-adaptive, lacking the ability to learn from example or by direct observation of the world. Neural networks on the other hand have the ability to learn from data, and derive much of their power from nonlinear function approximation combined with stochastic gradient descent. But intelligence requires more than modeling input-output relationships. Without the richness of symbolic computation, neural nets lack the simple but powerful operations such as variable binding that allow for analogy making and reasoning, which underlie the ability to generalize from few examples.
  
 
+
We approach the problem from a very different perspective, inspired by the brain’s high-dimensional circuits and the unique mathematical properties of high-dimensional spaces. ''It leads us to a novel information processing architecture that combines the strengths of symbolic AI and neural networks, and yet has novel emergent properties of its own.'' By combining a small set of basic operations on high-dimensional vectors, we obtain hybrid AI system (HAS) that makes it possible to represent and manipulate data in ways familiar to us from symbolic AI, and to learn from the statistics of data in ways familiar to us from artificial neural networks and deep learning. Further, principles of such HAS allow few-shot learning capabilities, and extremely robust operations against failures, defects, variations, and noise, all of which are complementary to ultra-low energy computation on nanoscale fabrics such as phase-change memory devices. Exciting further research (listed in below table) awaiting in this direction spans high-level algorithmic exploration all the way to efficient hardware design for emerging computational fabrics.
HDC has also been successfully applied in few-shot meta-learning problems - where a controller is trained to distinguish previously unseen classes by feeding only a few examples. The state-of-the-art accuracy in few-shot learning problems has been achieved by augmenting the controller with an explicit memory and guiding the controller to learning HDC representations. Exciting further research awaiting in this direction includes exploring compact representation of explicit memory and exploring novel cost functions and training algorithms to further improving the classification accuracy.
 
  
 
===Useful Reading===
 
===Useful Reading===
*[https://link.springer.com/article/10.1007/s12559-009-9009-8 Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors]
+
*[https://arxiv.org/pdf/2010.01939.pdf Robust high-dimensional memory-augmented neural networks], arXiv, 2020
*[https://www.nature.com/articles/s41928-020-0410-3 In-memory hyperdimensional computing]
+
*[https://www.nature.com/articles/s41928-020-0410-3 In-memory hyperdimensional computing], Nature Electronics, 2020.
*[https://arxiv.org/pdf/2010.01939.pdf Robust High-dimensional Memory-augmented Neural Networks]
+
*[https://www.nature.com/articles/s41928-020-00510-8 A wearable biosensing system with in-sensor adaptive machine learning for hand gesture recognition], Nature Electronics, 2020.  
 +
*[https://link.springer.com/article/10.1007/s12559-009-9009-8 Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors], Cognitive Computation, 2009.
  
 
===Prerequisites===
 
===Prerequisites===
Line 37: Line 37:
 
*Background in machine learning (''recommended'')
 
*Background in machine learning (''recommended'')
 
*Experience with any deep learning framework such as TensorFlow or PyTorch (''recommended'')
 
*Experience with any deep learning framework such as TensorFlow or PyTorch (''recommended'')
*VLSI I and VLSI II (''recommended'')
+
*VLSI I (''recommended'')
  
 
==In-Memory Computing (IMC)==
 
==In-Memory Computing (IMC)==
Line 65: Line 65:
 
! style="width: 5%;"|Type !! style="width: 20%"|Project !! style="width: 60%"|Description !! style="width: 5%"|Topic !! style="width: 15%"|Workload Type  
 
! style="width: 5%;"|Type !! style="width: 20%"|Project !! style="width: 60%"|Description !! style="width: 5%"|Topic !! style="width: 15%"|Workload Type  
 
|-
 
|-
| MA|| Lifelong learning challenge ||  [http://iis-projects.ee.ethz.ch/images/0/01/IBM_MANN_Y2020.pdf '''Artificial general intelligence (AGI):''' lifelong learning challenge] || HDC || algorithmic design
+
 
 +
 
 +
| MA || Face Recognition at Scale ||  [http://iis-projects.ee.ethz.ch/images/3/3d/IBM_FaceRec_at_Scale.pdf Link to description] || HAS || algorithmic design
 
|-
 
|-
| MA|| Machine learning based on optimal transport using in-memory computing' || [http://iis-projects.ee.ethz.ch/images/3/33/IBM_OT_Y2020.pdf '''Machine learning based on optimal transport using in-memory computing'''] || HDC || algorithmic design
+
 
 +
| MA || Accelerating Transformers with Computational Memory || [http://iis-projects.ee.ethz.ch/images/b/be/IBM_TransfAcc.pdf Link to description] || HAS || Hardware/algorithmic design
 
|-
 
|-
| MA || Developing Efficient Models of Strong AI for Intelligence Quotient (IQ) Test ||  [http://iis-projects.ee.ethz.ch/images/4/4b/IBM_RPM.pdf Developing Efficient Models of '''Strong AI for Intelligence Quotient (IQ) Test''']  || HDC || algorithmic design
+
 
 +
 
 +
 
 +
 
 +
| MA || Developing Efficient Models of Strong AI for Intelligence Quotient (IQ) Test ||  [http://iis-projects.ee.ethz.ch/images/4/4b/IBM_RPM.pdf Link to description]  || HAS || algorithmic design
 +
|-
 +
 
 +
| MA|| Lifelong learning challenge ||  [http://iis-projects.ee.ethz.ch/images/0/01/IBM_MANN_Y2020.pdf Link to description] || HAS || algorithmic/hardware design
 
|-
 
|-
 +
 +
| MA|| Machine learning based on optimal transport using in-memory computing' ||  [http://iis-projects.ee.ethz.ch/images/3/33/IBM_OT_Y2020.pdf Link to description] || HAS || hardware design
 +
|-
 +
 +
 
| MA || Accurate deep learning inference using computational memory ||  [https://www.zurich.ibm.com/careers/2020_027.html Project description and application]  || IMC || algorithmic design
 
| MA || Accurate deep learning inference using computational memory ||  [https://www.zurich.ibm.com/careers/2020_027.html Project description and application]  || IMC || algorithmic design
 
|-
 
|-
Line 82: Line 97:
 
==Contact==
 
==Contact==
 
: Thesis will be at IBM Zurich in Rüschlikon
 
: Thesis will be at IBM Zurich in Rüschlikon
; Hyperdimensional Computing (HDC) projects
+
; Hybrid AI Systems (HAS) projects
 
: Contact (at ETH Zurich): [[:User:kgf | Dr. Frank K. Gurkaynak]] and [[:User:Herschmi | Michael Hersche]]
 
: Contact (at ETH Zurich): [[:User:kgf | Dr. Frank K. Gurkaynak]] and [[:User:Herschmi | Michael Hersche]]
 
: Contact (at IBM): [mailto:ase@zurich.ibm.com Dr. Abu Sebastian]  
 
: Contact (at IBM): [mailto:ase@zurich.ibm.com Dr. Abu Sebastian]  

Revision as of 19:53, 13 January 2021

IBM ZRLab.png

Short Description

Today, we are entering the era of cognitive computing, which holds great promise in deriving intelligence and knowledge from huge volumes of data. In today’s computers based on von Neumann architecture, huge amounts of data need to be shuttled back and forth at high speeds, a task at which this architecture is inefficient.

It is becoming increasingly clear that to build efficient cognitive computers, we need to transition to non-von Neumann architectures in which memory and processing coexist in some form. At IBM Research–Zurich in the Neuromorphic and In-memory Computing Group, we explore various such computing paradigms from in-memory computing to brain-inspired neuromorphic computing. Our research spans from devices and architectures to algorithms and applications.

About the IBM Research–Zurich

The location in Zurich is one of IBM’s 12 global research labs. IBM has maintained a research laboratory in Switzerland since 1956. As the first European branch of IBM Research, the mission of the Zurich Lab, in addition to pursuing cutting-edge research for tomorrow’s information technology, is to cultivate close relationships with academic and industrial partners, be one of the premier places to work for world-class researchers, to promote women in IT and science, and to help drive Europe’s innovation agenda. Download factsheet

Hybrid AI Systems (HAS)

NatureElectronics20.jpg

Neither symbolic AI nor neural networks alone has produced the kind of intelligence expressed in human and animal behavior. Why? Each has a long and rich history, but has addressed a relatively narrow aspect of the problem. Symbolic AI focuses on solving cognitive problems, drawing upon the rich framework of symbolic computation to manipulate internal representations in order to perform reasoning and inference. But it suffers from being non-adaptive, lacking the ability to learn from example or by direct observation of the world. Neural networks on the other hand have the ability to learn from data, and derive much of their power from nonlinear function approximation combined with stochastic gradient descent. But intelligence requires more than modeling input-output relationships. Without the richness of symbolic computation, neural nets lack the simple but powerful operations such as variable binding that allow for analogy making and reasoning, which underlie the ability to generalize from few examples.

We approach the problem from a very different perspective, inspired by the brain’s high-dimensional circuits and the unique mathematical properties of high-dimensional spaces. It leads us to a novel information processing architecture that combines the strengths of symbolic AI and neural networks, and yet has novel emergent properties of its own. By combining a small set of basic operations on high-dimensional vectors, we obtain hybrid AI system (HAS) that makes it possible to represent and manipulate data in ways familiar to us from symbolic AI, and to learn from the statistics of data in ways familiar to us from artificial neural networks and deep learning. Further, principles of such HAS allow few-shot learning capabilities, and extremely robust operations against failures, defects, variations, and noise, all of which are complementary to ultra-low energy computation on nanoscale fabrics such as phase-change memory devices. Exciting further research (listed in below table) awaiting in this direction spans high-level algorithmic exploration all the way to efficient hardware design for emerging computational fabrics.

Useful Reading

Prerequisites

  • Python
  • Background in machine learning (recommended)
  • Experience with any deep learning framework such as TensorFlow or PyTorch (recommended)
  • VLSI I (recommended)

In-Memory Computing (IMC)

NNcover imc.jpg

For decades, conventional computers based on the von Neumann architecture have performed computation by repeatedly transferring data between their processing and their memory units, which are physically separated. As computation becomes increasingly data-centric and as the scalability limits in terms of performance and power are being reached, alternative computing paradigms are searched for in which computation and storage are collocated. A fascinating new approach is that of computational memory where the physics of nanoscale memory devices are used to perform certain computational tasks within the memory unit in a non-von Neumann manner. Computational Memory (CM) is finding application in a variety of areas such as machine learning and signal processing. In addition to novel non-volatile memory technologies like PCM and ReRAM, also conventional SRAM has been proposed as underlying storage technology in Computational Memories.

Useful Reading

Prerequisites

  • General interest in Deep Learning and memory/system design
  • VLSI I and VLSI II (recommended)
  • Analog Circuit Design (recommended)

Specific requirements for the different projects vary and are generally negotiable.

Available Projects

We are inviting applications from students to conduct their master’s thesis work or an internship project at the IBM Research lab in Zurich on this exciting new topic.

Type Project Description Topic Workload Type
MA Face Recognition at Scale Link to description HAS algorithmic design
MA Accelerating Transformers with Computational Memory Link to description HAS Hardware/algorithmic design
MA Developing Efficient Models of Strong AI for Intelligence Quotient (IQ) Test Link to description HAS algorithmic design
MA Lifelong learning challenge Link to description HAS algorithmic/hardware design
MA Machine learning based on optimal transport using in-memory computing' Link to description HAS hardware design
MA Accurate deep learning inference using computational memory Project description and application IMC algorithmic design
MA ADC design for computational memory Digital-to-Analog converters (DACs) and Analog-to-Digital converters (ADCs) are extensively employed in Computational Memory (CM) to handle the crossing between the digital and analog domain, in which computationally expensive tasks, like Matrix-Vector Multiplications (MVM), are carried out with O(1) complexity. Each conversion costs a certain amount of energy and its precision can only be guaranteed up to the Effective Number of Bits (ENOB) of the employed data converter.
The research focus will be on understanding the system level requirements on ADC and DAC for optimal performance of Deep Neural Network inference using CM. Furthermore, the effects of noise, non-linearity and manufacturing tolerances shall be examined and counter measurements, like for example periodic digital ADC recalibration and digital post processing, shall be evaluated with regards to effectivity and energy costs.
IMC analog circuit design
SA Testing of a computational memory chip This project is about building a Microprocessor/FPGA-based test platform around a novel IMC chip. After commissioning the chip, DL workload tests can be run to characterize its throughput and energy-efficiency. IMC PCB Design
and/or
µ-code implementation
MA/SA Neural Network Training on a
computational memory chip
TBA IMC algorithm/system design
and/or
analog circuit design

Contact

Thesis will be at IBM Zurich in Rüschlikon
Hybrid AI Systems (HAS) projects
Contact (at ETH Zurich): Dr. Frank K. Gurkaynak and Michael Hersche
Contact (at IBM): Dr. Abu Sebastian
Contact (at IBM): Dr. Abbas Rahimi
Professor: Luca Benini
In-Memory Computing (IMC) projects
Contact (at IBM/ETH Zurich): Riduan Khaddam-Aljameh
Contact (at IBM): Dr. Abu Sebastian
Professor: Luca Benini