Out-of-context and out-of-scope: Manipulating large language models through minimal instruction set modifications

Monty-Maximilian Zühlke, Daniel Kudenko and Wolfgang Nejdl

PLOS ONE, 2026, vol. 21, issue 2, 1-36

Abstract: Understanding the emergence of reasoning capabilities in large language models (LLMs) is important for aligning their response behaviour with human intentions, especially as these models become accessible to a broad range of users and begin to operate autonomously without supervision. One interesting capability is out-of-context reasoning, where models seem to infer and adopt specific response behaviours based on descriptive information in a zero-shot fashion, that is, without any concrete examples. However, because a model’s training data is rarely available for inspection, it is difficult to judge whether all possible behavioural patterns that can be inferred in this way are benign in nature. Understanding this mechanism in more detail and its dependency on the data is therefore a crucial step in evaluating emerging reasoning capabilities in LLMs. In this work, we extend current research on out-of-context reasoning by showing that user-defined response behaviour can be embedded into LLMs through fine-tuning on a few short descriptions of the behaviour hidden in a substantially larger set of longer and differently formatted instructions. More precisely, we mathematically motivate and empirically show that models can not only pick up signals diffused in a large body of noisy information during training but can infer and adopt response patterns from it. Conversely, we show that triggering these response patterns can heavily depend on the prompting strategy while tokens, which are assigned fixed sequences of token IDs, can reinforce and facilitate the embedding and triggering. Together, our findings demonstrate that LLMs can be manipulated through minimal instruction set modifications but may only reveal the effect of this manipulation when prompted in a specific way. This highlights that using models, whose training data is not publicly accessible, in environments, where their input is not adequately monitored, may have unforeseen consequences.

Date: 2026
References: Add references at CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0341558 (text/html)
https://journals.plos.org/plosone/article/file?id= ... 41558&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pone00:0341558

DOI: 10.1371/journal.pone.0341558

Access Statistics for this article

More articles in PLOS ONE from Public Library of Science
Bibliographic data for series maintained by plosone ().