Adaptive Neural Feedback Methods for Bias and Weight Adjustment in Feed Forward Layers of LLMs

Sai Sukesh Reddy Tummuri

International Journal of Scientific Research in Science and Technology, 2024, vol. 11, issue 5, 821-833

Abstract: Feed-forward layers constitute the dominant computational and parametric component of transformer-based Large Language Models (LLMs), yet they are a major source of training instability due to static bias terms, uncontrolled weight scaling, and activation distribution drift. Conventional optimization methods rely solely on global backpropagation signals, which are often insufficient to correct local statistical imbalances that emerge during large-scale, long-horizon training. This work proposes AFB-FFN (Adaptive Feedback Bias and Weight Corrected Feed-Forward Network), a novel feed-forward layer architecture that integrates an internal neural feedback mechanism to dynamically regulate bias and weight behavior during forward propagation. The proposed model introduces lightweight feedback units that generate bias correction vectors and weight gating signals conditioned on intermediate activations, enabling real-time stabilization of hidden representations. The AFB-FFN architecture is embedded within a transformer framework and evaluated on a token-level language modeling task. Extensive experimental analysis demonstrates that the proposed method significantly improves training stability under both nominal and high learning-rate regimes. The model achieves a controlled token-level accuracy of 97.8%, while maintaining smooth convergence, reduced gradient norm variance, lower activation drift, and stable gate entropy compared to conventional FFN baselines. These results validate that adaptive neural feedback driven bias and weight correction within feed-forward layers is an effective and scalable strategy for stabilizing LLM training. The proposed AFB FFN offers a practical architectural advancement toward robust, efficient, and statistically stable large language model optimization.

Keywords: Feed Forward Layers; Large Language Models; Adaptive Feedback Bias; Weight Corrected Feed-Forward Network; Deep Transformer Stacks; High Learning Rate; LLM Training (search for similar items in EconPapers)
Date: 2024
References: Add references at CitEc
Citations:

Downloads: (external link)
https://ijsrst.com/home/article/view/IJSRST52310380 Abstract page (text/html)
https://ijsrst.com/home/article/download/IJSRST52310380/IJSRST52310380 Full text (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:etm:ijsrst:v11:y2024:i5:id:1383

DOI: 10.32628/IJSRST52310380

Access Statistics for this article

More articles in International Journal of Scientific Research in Science and Technology from Technoscience Academy
Bibliographic data for series maintained by Pankaj Sharma ().