Early Prediction of DNN Activation Using Hierarchical Computations

Bharathwaj Suresh, Kamlesh Pillai, Gurpreet Singh Kalsi, Avishaii Abuhatzera and Sreenivas Subramoney
Additional contact information
Bharathwaj Suresh: Processor Architecture Research (PAR) Lab, Intel Labs, Bangalore 560048, India
Kamlesh Pillai: Processor Architecture Research (PAR) Lab, Intel Labs, Bangalore 560048, India
Gurpreet Singh Kalsi: Processor Architecture Research (PAR) Lab, Intel Labs, Bangalore 560048, India
Avishaii Abuhatzera: Corporate Strategy Office, Intel, Haifa 3508409, Israel
Sreenivas Subramoney: Processor Architecture Research (PAR) Lab, Intel Labs, Bangalore 560048, India

Mathematics, 2021, vol. 9, issue 23, 1-18

Abstract: Deep Neural Networks (DNNs) have set state-of-the-art performance numbers in diverse fields of electronics (computer vision, voice recognition), biology, bioinformatics, etc. However, the process of learning (training) from the data and application of the learnt information (inference) process requires huge computational resources. Approximate computing is a common method to reduce computation cost, but it introduces loss in task accuracy, which limits their application. Using an inherent property of Rectified Linear Unit (ReLU), a popular activation function, we propose a mathematical model to perform MAC operation using reduced precision for predicting negative values early. We also propose a method to perform hierarchical computation to achieve the same results as IEEE754 full precision compute. Applying this method on ResNet50 and VGG16 shows that up to 80% of ReLU zeros (which is 50% of all ReLU outputs) can be predicted and detected early by using just 3 out of 23 mantissa bits. This method is equally applicable to other floating-point representations.

Keywords: DNN; ReLU; floating-point numbers; hardware acceleration (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2021
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/9/23/3130/pdf (application/pdf)
https://www.mdpi.com/2227-7390/9/23/3130/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:9:y:2021:i:23:p:3130-:d:695161

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().