Heuristics in Design of Deep NeuralNetworks

Ricardo Martins de Abreu Silva () and Andersson Alves da Silva ()
Additional contact information
Ricardo Martins de Abreu Silva: Federal University of Pernambuco
Andersson Alves da Silva: Federal University of Pernambuco

Chapter 35 in Handbook of Heuristics, 2025, pp 1031-1085 from Springer

Abstract: Abstract The complexity of Deep Neural Networks (DNNs) has driven advancements in Neural Architecture Search (NAS), Hyperparameter Optimization (HPO), and Learning Rule Optimization (LRO). This study reviews heuristic methodologies, focusing on Evolutionary Algorithms (EAs) and Swarm Intelligence (SI). We analyze Genetic Algorithms (GAs), Particle Swarm Optimization (PSO), and Multi-Objective Optimization (MOO), emphasizing the Biased Random Key Genetic Algorithm (BRKGA). BRKGA encodes neural architectures and hyperparameters as continuous vectors, enhancing search efficiency in NAS and HPO. We evaluate BRKGA on Feedforward Neural Networks (FNNs), Convolutional Neural Networks (CNNs), and Graph Neural Networks (GNNs), demonstrating their effectiveness in tuning learning rates, dropout rates, and batch sizes. Additionally, we explore its role in LRO, optimizing adaptive weight updates and gradient modulation. Experiments on benchmark datasets show that BRKGA consistently yields promising architectures and hyperparameter configurations, balancing accuracy, efficiency, and adaptability. Our findings highlight BRKGA as a viable alternative for NAS, HPO, and LRO, particularly in complex search spaces where structured exploration is essential. Finally, challenges in heuristic-driven NAS, HPO, and AutoML are examined, along with future research directions in scalable optimization, adaptive learning mechanisms, and neuromorphic computing.

Keywords: Biased random key genetic algorithm; Deep feedforward neural networks; Convolutional neural networks; Graph neural networks; Neural architecture search; Hyperparameter optimization; Learning rule optimization (search for similar items in EconPapers)
Date: 2025
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:sprchp:978-3-032-00385-0_74

Ordering information: This item can be ordered from
http://www.springer.com/9783032003850

DOI: 10.1007/978-3-032-00385-0_74

Access Statistics for this chapter

More chapters in Springer Books from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().