Generative AI in Safety-Critical Systems: Risks of Sycophancy, Hallucination, and Trust Degradation

Mohammad Yazdi (), Esmaeil Zarei (), Sidum Adumene () and Amin Beheshti ()
Additional contact information
Mohammad Yazdi: Macquarie University, Faculty of Science and Engineering, School of Engineering
Esmaeil Zarei: Embry-Riddle Aeronautical University, Department of Safety Science, College of Aviation
Sidum Adumene: Memorial University of Newfoundland, School of Ocean Technology, Fisheries and Marine Institute
Amin Beheshti: Macquarie University, Centre for Applied Artificial Intelligence

A chapter in Data-Driven Methods for Reliability and Safety Engineering: Applications in Industrial Systems, 2026, pp 557-582 from Springer

Abstract: Abstract The integration of advanced AI language models like AI Co-pilots into workplace health and safety (WHS) and risk management for sociotechnical systems efficiency gains; however, it may raise critical challenges. This paper focuses on key pitfalls of using AI in safety-critical decision support, emphasizing problems of sycophancy (the AI’s tendency to agree with user assumptions), hallucination (generation of inaccurate or non-existent information), and trust erosion (reduced confidence in AI outputs due to inconsistency, opacity, or past errors). We provide context on how AI is being considered for tasks such as hazard identification and risk analysis, then outline a methodology to evaluate AI limitations using expert oversight and performance metrics. The results, drawn from emerging research and case analyses, indicate that AI tools can produce misleading or biased safety guidance without proper checks, undermining trust in high-stakes environments. In discussions, we explore why these issues occur, from the predictive, pattern-based nature of language models to human factors like automation bias and examine implications for safety professionals. Finally, we conclude that AI systems should serve strictly as decision-support tools rather than autonomous decision-makers in WHS. Rigorous human-in-the-loop strategies, critical oversight, and adherence to safety science principles are essential to harness AI’s benefits without compromising workplace safety.

Keywords: Ethical AI deployment; AI transparency; AI-assisted decision-making; Explainable AI (XAI); Safety-critical systems (search for similar items in EconPapers)
Date: 2026
References: Add references at CitEc
Citations:

There are no downloads for this item, see the EconPapers FAQ for hints about obtaining it.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:ssrchp:978-3-032-22873-4_41

Ordering information: This item can be ordered from
http://www.springer.com/9783032228734

DOI: 10.1007/978-3-032-22873-4_41

Access Statistics for this chapter

More chapters in Springer Series in Reliability Engineering from Springer
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().