Mean-Field-Type Transformers

Hamidou Tembine (), Manzoor Ahmed Khan and Issa Bamia
Additional contact information
Hamidou Tembine: Learning and Game Theory Laboratory, TIMADIE France and Université du Québec à Trois-Rivières, 3351, Boulevard des Forges, Trois-Rivières, QC G9A 5H7, Canada
Manzoor Ahmed Khan: Autonomous Systems Research Department at Nokia Bell Labs, Murray Hill, NJ 07974-0636, USA
Issa Bamia: African Institute of Mathematical Sciences, South West Region, Crystal Garden, Limbe P.O. Box 608, Cameroon

Mathematics, 2024, vol. 12, issue 22, 1-51

Abstract: In this article, we present the mathematical foundations of generative machine intelligence and link them with mean-field-type game theory. The key interaction mechanism is self-attention, which exhibits aggregative properties similar to those found in mean-field-type game theory. It is not necessary to have an infinite number of neural units to handle mean-field-type terms. For instance, the variance reduction in error within generative machine intelligence is a mean-field-type problem and does not involve an infinite number of decision-makers. Based on this insight, we construct mean-field-type transformers that operate on data that are not necessarily identically distributed and evolve over several layers using mean-field-type transition kernels. We demonstrate that the outcomes of these mean-field-type transformers correspond exactly to the mean-field-type equilibria of a hierarchical mean-field-type game. Due to the non-convexity of the operators’ composition, gradient-based methods alone are insufficient. To distinguish a global minimum from other extrema—such as local minima, local maxima, global maxima, and saddle points—alternative methods that exploit hidden convexities of anti-derivatives of activation functions are required. We also discuss the integration of blockchain technologies into machine intelligence, facilitating an incentive design loop for all contributors and enabling blockchain token economics for each system participant. This feature is especially relevant to ensuring the integrity of factual data, legislative information, medical records, and scientifically published references that should remain immutable after the application of generative machine intelligence.

Keywords: game theory; deep learning; generative transformers (search for similar items in EconPapers)
JEL-codes: C (search for similar items in EconPapers)
Date: 2024
References: View references in EconPapers View complete reference list from CitEc
Citations:

Downloads: (external link)
https://www.mdpi.com/2227-7390/12/22/3506/pdf (application/pdf)
https://www.mdpi.com/2227-7390/12/22/3506/ (text/html)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:gam:jmathe:v:12:y:2024:i:22:p:3506-:d:1517567

Access Statistics for this article

Mathematics is currently edited by Ms. Emma He

More articles in Mathematics from MDPI
Bibliographic data for series maintained by MDPI Indexing Manager ().