SinLU: Sinu-Sigmoidal Linear Unit

Ashis Paul, Rajarshi Bandyopadhyay, Jin Hee Yoon, Zong Woo Geem and Ram Sarkar
Additional contact information
Ashis Paul: Department of Computer Science and Engineering, Jadavpur University, Kolkata 700032, India
Rajarshi Bandyopadhyay: Department of Computer Science and Engineering, Jadavpur University, Kolkata 700032, India
Jin Hee Yoon: School of Mathematics and Statistics, Sejong University, Seoul 05006, Korea
Zong Woo Geem: College of IT Convergence, Gachon University, Seongnam 13120, Korea
Ram Sarkar: Department of Computer Science and Engineering, Jadavpur University, Kolkata 700032, India

Mathematics, 2022, vol. 10, issue 3, 1-14

Abstract: Non-linear activation functions are integral parts of deep neural architectures. Given the large and complex dataset of a neural network, its computational complexity and approximation capability can differ significantly based on what activation function is used. Parameterizing an activation function with the introduction of learnable parameters generally improves the performance. Herein, a novel activation function called Sinu-sigmoidal Linear Unit (or SinLU) is proposed. SinLU is formulated as S i n L U ( x ) = ( x + a sin b x ) · σ ( x ) , where σ ( x ) is the sigmoid function. The proposed function incorporates the sine wave, allowing new functionalities over traditional linear unit activations. Two trainable parameters of this function control the participation of the sinusoidal nature in the function, and help to achieve an easily trainable, and fast converging function. The performance of the proposed SinLU is compared against widely used activation functions, such as ReLU, GELU and SiLU. We showed the robustness of the proposed activation function by conducting experiments in a wide array of domains, using multiple types of neural network-based models on some standard datasets. The use of sine wave with trainable parameters results in a better performance of SinLU than commonly used activation functions.

Keywords: activation function; trainable parameter; sinusoidal curve; sigmoid function; CNN; deep learning (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2022
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/10/3/337/pdf (application/pdf)
https://www.mdpi.com/2227-7390/10/3/337/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:10:y:2022:i:3:p:337-:d:731244

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().