Equivalent-accuracy accelerated neural-network training using analogue memory

Stefano Ambrogio, Pritish Narayanan, Hsinyu Tsai, Robert M. Shelby, Irem Boybat, Carmelo Nolfo, Severin Sidler, Massimo Giordano, Martina Bodini, Nathan C. P. Farinha, Benjamin Killeen, Christina Cheng, Yassine Jaoudi and Geoffrey W. Burr ()
Additional contact information
Stefano Ambrogio: IBM Research–Almaden
Pritish Narayanan: IBM Research–Almaden
Hsinyu Tsai: IBM Research–Almaden
Robert M. Shelby: IBM Research–Almaden
Irem Boybat: IBM Research–Zurich
Carmelo Nolfo: IBM Research–Almaden
Severin Sidler: IBM Research–Almaden
Massimo Giordano: IBM Research–Almaden
Martina Bodini: IBM Research–Almaden
Nathan C. P. Farinha: IBM Research–Almaden
Benjamin Killeen: IBM Research–Almaden
Christina Cheng: IBM Research–Almaden
Yassine Jaoudi: IBM Research–Almaden
Geoffrey W. Burr: IBM Research–Almaden

Nature, 2018, vol. 558, issue 7708, 60-67

Abstract: Abstract Neural-network training can be slow and energy intensive, owing to the need to transfer the weight data for the network between conventional digital memory chips and processor chips. Analogue non-volatile memory can accelerate the neural-network training algorithm known as backpropagation by performing parallelized multiply–accumulate operations in the analogue domain at the location of the weight data. However, the classification accuracies of such in situ training using non-volatile-memory hardware have generally been less than those of software-based training, owing to insufficient dynamic range and excessive weight-update asymmetry. Here we demonstrate mixed hardware–software neural-network implementations that involve up to 204,900 synapses and that combine long-term storage in phase-change memory, near-linear updates of volatile capacitors and weight-data transfer with ‘polarity inversion’ to cancel out inherent device-to-device variations. We achieve generalization accuracies (on previously unseen data) equivalent to those of software-based training on various commonly used machine-learning test datasets (MNIST, MNIST-backrand, CIFAR-10 and CIFAR-100). The computational energy efficiency of 28,065 billion operations per second per watt and throughput per area of 3.6 trillion operations per second per square millimetre that we calculate for our implementation exceed those of today’s graphical processing units by two orders of magnitude. This work provides a path towards hardware accelerators that are both fast and energy efficient, particularly on fully connected neural-network layers.

Date: 2018
References: Add references at CitEc
Citations: View citations in EconPapers (17)

Downloads: (external link)
https://www.nature.com/articles/s41586-018-0180-5 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:nat:nature:v:558:y:2018:i:7708:d:10.1038_s41586-018-0180-5

Ordering information: This journal article can be ordered from
https://www.nature.com/

DOI: 10.1038/s41586-018-0180-5

Access Statistics for this article

Nature is currently edited by Magdalena Skipper

More articles in Nature from Nature
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().