Metacognition Should Be the Scientific Framework for Bounded and Effective Self-Governance in Generative AI
Extracted source text ยท arXiv 2605.23981
\documentclass{article}
% NeurIPS 2026 loads natbib by default; pass author-year options before loading the style file.
\PassOptionsToPackage{authoryear,round}{natbib}
\usepackage[preprint, position]{neurips_2026}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{url}
\usepackage{booktabs}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{nicefrac}
\usepackage{microtype}
\usepackage{xcolor}
\usepackage[colorlinks=true,linkcolor=black,citecolor=green!50!black,urlcolor=blue]{hyperref}
\usepackage{graphicx}
\usepackage{tabularx}
\usepackage{array}
\usepackage{enumitem}
\title{Metacognition Should Be the Scientific Framework for Bounded and Effective Self-Governance in Generative AI}
\author{
Eugene Yu Ji \\
University of Waterloo \\
\texttt{eugeneyuji1@gmail.com}
\And
Igor Grossmann \\
University of Waterloo \\
\texttt{igrossma@uwaterloo.ca}
\And
Amir-Hossein Karimi \\
University of Waterloo \\
\texttt{amirh.karimi@uwaterloo.ca}
}
\begin{document}
\maketitle
\begin{abstract}
Generative AI research increasingly confronts a shared problem: systems must sustain yet govern their own generative activity when uncertainty is high, evidence is missing, or context is insufficient. This position paper argues that \textbf{metacognition should become the scientific framework for bounded and effective self-governance in generative AI}, where output generation is properly evaluated together with the capacities through which generative systems navigate and regulate their own activity. We advance this position by showing that bounded and effective AI self-governance requires metacognitive alignment across computational, algorithmic, and ecological levels. At the \textit{computational} level, metacognition specifies the meta-level functions a system is meant to serve, such as monitoring, evaluation, control, and adaptation. At the \textit{algorithmic} level, these functions are realized through procedures such as elicitation, iteration, and modularization. At the \textit{ecological} level, metacognitive signals become meaningful, actionable, and accountable within the interface, workflow, and accountability arrangements. Metacognition thus makes it possible to conceive generative AI as both capable and well-governed, rather than treating capability and governance as competing aims.\end{abstract}
\section{Introduction}
Generative AI increasingly requires systems whose task- and object-level activities can be regulated by ``meta-level'' strategies \citep{johnson2026}. We argue that \textbf{metacognition should be the scientific framework for bounded and effective self-governance in generative AI}, where generation is explained and evaluated together with the capacities through which generative systems manage their own activity.
The term ``metacognition'' comes from psychology, where it refers broadly to knowledge, awareness, monitoring, and control directed toward one's own mental activity \citep{flavell1979,nelson1990}. Computer science provides a related lineage: early ``metareasoning'' studies the cost, value, and control of computation under bounded resources \citep{russell1989,russell1991}, while related contemporary work extends this concern to introspective monitoring, meta-level control, anomaly detection, and failure recovery \citep{cox2005,schmill2011,cox2011}. Although metacognition offers an important shared vocabulary for developing generative AI systems that can assess, update, regulate, and govern the adequacy and effectiveness of their own behavior, the term becomes conceptually and analytically unstable when applied across heterogeneous lines of work. In contemporary AI research, ``metacognitive'' can refer to psychology-inspired prompting \citep{wang2024}, retrieval planning \citep{zhou2024}, learning \citep{liu2025}, confidence calibration and verbalized uncertainty \citep{lin2022,tian2023,xiong2024}, internal-state monitoring \citep{azaria2023,jian2025}, tool-use control \citep{schick2023,yao2023b}, verifier-guided correction \citep{dhuliawala2024,gou2024}, agentic memory and reflection \citep{shinn2023,zhao2024}, or broader aspirations for safer, more governable AI systems \citep{bai2022,johnson2026}.
We contend that these are fragments of a single scientific problem: how generative systems can achieve proper self-governance over their own generative activity. This paper argues that the goal should be \textbf{bounded \textit{and} effective self-governance}, and that this goal is impossible to specify, design, or evaluate without \textbf{cross-level metacognitive alignment}, in which \textit{functions}, \textit{procedures}, and \textit{ecological} conditions are precisely distinguished and systematically related, extending Marr's well-known three-level analysis of intelligent systems \citep{marr1982,krafft2018}.
\section{What Metacognition Can Mean for Self-Governing AI}
Metacognition entered cognitive psychology through models of monitoring and control of task-level processes that perform the primary activity by meta-level processes that observe, evaluate, and guide that activity \citep{flavell1979,nelson1990}. Later research shows that such monitoring is shaped by development, affect, embodiment, social interaction, and context, and can dissociate from task performance \citep{metcalfe1994,veenman2006,efklides2008,efklides2011,dunlosky2009,fleming2012,yeung2012,grossmann2017,grossmann2020,grossmann2021}. This lineage is helpful for conceptualizing AI self-governance, but must be translated carefully: metacognition for self-governance should refer to the functional capacity to monitor, evaluate, control, and adapt one's own activity, not to conscious introspection, subjective awareness, or human-like access to mental states \citep{overgaard2012,proust2013}.
A similar computer-science tradition, from rational metareasoning to computational metacognition, treats intelligent systems as monitoring and controlling their own reasoning \citep{russell1989,russell1991,cox2005,cox2011,schmill2011,cox2022,caro2022}. Recent work extends this idea to generative AI through metacognitive prompting, retrieval monitoring and planning, activation-level self-report, and calibration-based assessment \citep{wang2024,zhou2024,jian2025,liu2025,servajean2026}. Additionally, a broader literature in contemporary computer science also develops procedures related to self-governance without using the language of metacognition. These include meta-level selection over reasoning paths \citep{wang2023,yao2023a}, feedback-based revision and memory \citep{madaan2023,shinn2023,gao2023,gou2024}, and coordination with retrieval, tools, or external action \citep{lewis2020,schick2023,yao2023b,zhou2024}. Work on self-correction makes the same issue explicit: models may assess confidence, limits, and the need for assistance, but such correction remains fragile and often depends on prompts, verifiers, or external scaffolding \citep{lin2022,kadavath2022,ren2023,xiong2024,kapoor2024,huang2024,li2024}.
These developments converge on the target of our position: the insufficiency of a capability-first view of generative AI \citep{johnson2026}. This view prioritizes task-level capabilities, such as fluent generation, answer accuracy, problem-solving performance, planning quality, and successful task completion, while treating meta-level capacities, such as uncertainty monitoring, sufficiency evaluation, self-correction, perspective-conditioned reasoning, as auxiliary or post-hoc additions to model performance. Against this view, we argue that bounded and effective self-governance requires task-level capacities to be properly organized by metacognitive capacities.
\section{Cross-Level Metacognitive Alignment for Self-Governing AI}
Our proposal is that bounded and effective self-governance in generative AI requires cross-level metacognitive alignment: \textit{computational targets}, \textit{algorithmic procedures}, and \textit{ecological arrangements} must be specified together rather than treated as separable design problems. By \textbf{bounded} self-governance, we mean that generative activity is not only treated as capability-driven open-ended production, but proceeds under metacognitive conditions that specify what should be monitored and evaluated, how control and adaptation should be procedurally realized, and where their signals and actions become meaningful in use. By \textbf{effective} self-governance, we mean that these conditions do not merely constrain generation, but make task-level activity more reliable, useful, and accountable.
\begin{quote}
\textbf{Working definition:} Metacognition in a generative system refers to self-governing processes through which the system uses information about its own states, outputs, and trajectories to manage subsequent generative behavior. At the \textbf{computational} level, metacognition specifies what should be \textit{monitored} and \textit{evaluated} and what forms of \textit{control} or \textit{adaptation} these assessments are meant to guide. At the \textbf{algorithmic} level, these functions are realized through procedures such as\textit{ elicitation},\textit{ iteration}, and\textit{ modularization}. At the \textbf{ecological} level, they are situated within \textit{interface}, \textit{workflow}, and \textit{accountability} arrangements that determine whether metacognitive signals and actions become meaningful, actionable, and accountable in use.
\end{quote}
This definition treats bounded and effective self-governance as cross-level metacognitive alignment rather than as a property of single-level or single-model capability (\hyperref[fig:framework]{Figure~\ref*{fig:framework}}). The tripartite-level approach takes loose inspiration from Marr's distinction among computational, algorithmic, and implementational levels for intelligent systems \citep{marr1982}, along with later work extending or adapting levels of analysis in cognitive science, machine learning, and LLMs \citep{griffiths2015,bechtel2015,krafft2018,hamrick2020,ku2025}. Our use of levels is deliberately revisionary. Marr's framework primarily differentiates explanatory levels and often treats the computational level as the privileged site for specifying the problem a system solves. Our framework treats differentiation and interrelations as equally central. In particular, we replace implementation with ecology to capture how metacognitive governance is realized under real sociotechnical conditions, both within and across models, prompts, tools, users, workflows, organizations, and institutions \citep{hutchins1995,hollan2000,selbst2019,suresh2021}.
\begin{figure}[t]
\centering
\includegraphics[width=1.1\linewidth]{figure1.png}
\caption{Cross-level metacognitive alignment as a tripartite framework for bounded and effective self-governance in generative AI. The framework distinguishes computational targets, algorithmic realizations, and ecological conditions as three levels of metacognitive regulation, related through dynamic cross-level feedback. }
\label{fig:framework}
\end{figure}
\subsection{The computational level}
At the computational level, the central question is what metacognition is for: what meta-level problem the system is meant to solve with respect to its own generative activity. We treat \textit{self-monitoring}, \textit{self-evaluation}, \textit{self-control}, and \textit{self-adaptation} as regulatively ordered targets for self-governance: monitoring makes aspects of generation available; evaluation assigns significance to them; control changes the immediate course of generation; and adaptation carries regulatory consequences forward. Existing work already addresses certain important parts of these problems. Yet these contributions usually remain partial or loosely connected.
\textbf{Self-monitoring}. Self-monitoring tracks conditions of the system's own activity that may matter for later regulation. In generative AI, these conditions include analogous monitoring problems in verbalized uncertainty \citep{lin2022,xiong2024}, confidence calibration \citep{tian2023}, semantic uncertainty \citep{kuhn2023}, internal-state signals of truthfulness or epistemic condition \citep{azaria2023,jian2025}, among others. These studies make monitoring visible as a computational target, but their limit is that monitoring is often treated as an endpoint. Monitoring is metacognitive only when the tracked condition is available for subsequent evaluation or control, rather than being largely reported as an isolated diagnostic.
\textbf{Self-evaluation}. Self-evaluation assigns regulatory significance to what has been monitored. A monitored state becomes evaluative when uncertainty is treated as a reliability, risk, or self-improving signal \citep{lin2022,kadavath2022,tian2023,kuhn2023,xiong2024,kapoor2024,liu2025}; when inconsistency across reasoning paths guides selection or search \citep{wang2023,yao2023a}; when retrieval sufficiency is accessed \citep{lewis2020,gao2023,zhou2024}; or when verifier disagreement triggers correction or further validation \citep{dhuliawala2024,gou2024}.
\textbf{Self-control}. Self-control turns evaluative signals into changes in the immediate trajectory of generation. The system may resample, expand, defer, abstain, or prune a reasoning path, retrieve evidence, call a tool, revise an answer. This target connects classical metareasoning on bounded computation and expected-value control \citep{russell1989,russell1991}, with contemporary generative AI methods for inference-time search \citep{yao2023a}, retrieval planning and repair \citep{zhou2024}, tool invocation \citep{schick2023,yao2023b}, and selective answering or abstention under uncertainty \citep{ren2023,kapoor2024}.
\textbf{Self-adaptation}. Self-adaptation carries the consequences of regulation into later behavior. It concerns how systems maintain or adjust their capacities, limits, memories, strategies, and conditions of success over time \citep{cox2005,schmill2011,cox2011,cox2022}. In generative AI, this most often appears in reflective memory \citep{shinn2023,zhao2024}, experiential learning from task histories \citep{cox2022,zhao2024,liu2025}, failure recovery and feedback-guided strategy revision \citep{madaan2023,dhuliawala2024,gou2024}.
Together, these four interconnected targets define metacognition at the computational level. They specify the reflective navigating and regulatory problem that bounded and effective generative activity must solve: how the system tracks its epistemic and operational condition, interprets the significance of that condition, manages the action according to the condition, and reshapes later behavior in light of them.
\subsection{Algorithmic level}
At the algorithmic level, we define a minimal algorithmic template of how metacognitive computational targets become proper self-governing procedures. Let $x$ denote the input task, $c$ the default task context, $h_t$ the relevant system state, $G$ the base generative model, and $y$ the generated output:
\[
y \sim G(\cdot \mid x,c,h_t).
\]
Metacognition-relevant algorithms introduce intermediate structure $u$, such as rationales, decompositions, sampled trajectories, uncertainty estimates, retrieved evidence, verifier scores, memory traces, perspective taking, and so on:
\[
u = G_{\mathrm{int}}(x,c,h_t).
\]
Here $u$ as $G_{\mathrm{int}}$ becomes metacognitively relevant only when it is interpreted as a regulatory signal $z$, mapped by a control policy $\pi$ into an action $a$, and, in stronger cases, carried forward through adaptation $A$:
\[
u \xrightarrow{M,V} z \xrightarrow{\pi} a \xrightarrow{A} h_{t+1}.
\]
A self-regulatory signal $z$ is an interpreted assessment of the state of the generation under monitoring $M$ and evaluation $V$. A control policy $\pi$ maps such signals to regulatory actions $a$. Adaptation $A$ carries the consequences of regulation into the later system state $h_{t+1}$. The algorithmic question is therefore whether $z$ is diagnostically valid and $\pi$ routes it to actions that improve reliability, usefulness, or accountability (effectiveness) and how the action $a$ regulates the later system state $h_{t+1}$ specifies (boundedness). The algorithmic question is to determine a proper \textbf{self-governing horizon }$m^\ast$ such that \[
m^\ast = \inf \{ k \geq 0 : y_{t+k} \text{ is sufficient for } g(x,c) \text{ under } L(c,h_{t+k}) \}.
\]
Here $g(x,c)$ is the task-level goal, and $L(c,h_{t+k})$ denotes the relevant metacognitive constraints in task context $c$. In other words, the system seeks an output $y_{t+m^\ast}$ that is \textit{effective} for achieving a task goal $g$ in context $c$, while regulating the conditions under which that output remains \textit{bounded}, with the latter conditions characterized through the actual ecological characteristics of context $c$.
\textbf{Reflective elicitation}. Reflective elicitation makes intermediate structure $u=G_{\mathrm{int}}(x,c,h_t)$ available as a possible object of monitoring and evaluation, through which the system can estimate whether it is close to, far from, or wrongly oriented toward $m^\ast$:
\[
u \xrightarrow{M,V} z.
\]
The form of $u$ differs across methods. Chain-of-thought prompting makes $u$ a reasoning trace \citep{wei2022}; self-consistency makes $u$ a set of sampled reasoning paths whose agreement can be compared \citep{wang2023}; Tree of Thoughts makes $u$ a search structure of partial states \citep{yao2023a}; and related methods elicit structure by reorganizing the task or context before answering \citep{press2023,zhou2023, weston2023}. Elicitation thus supports self-governance by making reasoning, uncertainty, evidence, or task structure inspectable.
\textbf{Reflective iteration}. Reflective iteration uses prior generation to update the path toward $m^\ast$. An earlier output or trajectory becomes the object of later monitoring, evaluation, control, and adaptation:
\[
y_t \rightarrow u_t \xrightarrow{M,V} z_t \xrightarrow{\pi} a_t \xrightarrow{A} h_{t+1}.
\]
The reflective procedure unfolds through regulatory actions from $a_t$ to $a_{t+k}$. If a revision is selected at step $k_0$, where $k_0$ is determined to be $m^\ast$, then $a_{t+k_0}$ is a terminal action, and the revised output $y_{t+k_0}$ = $h_{t+k_0}$ marks the endpoint of the procedure, where $y$ satisfies the task goal $g$ under the metacognitive enabler and constraint $L(c,h_{t+k_0})$ within the relevant ecological context $c$. Hence, iteration contributes to effective self-governance when each pass changes the system's estimate of what remains to be done before an effective and bounded output can be reached. Otherwise, repeated generation maintains as resampling or rewriting. Self-Refine \citep{madaan2023}, Reflexion \citep{shinn2023}. Constitutional AI \citep{bai2022}. Chain-of-Verification \citep{dhuliawala2024}, retrieval-or tool-based revision\citep{gou2024,gao2023}, and dynamical system accounts of temporally evolving self-regulation \citep{ji2026,chemnitz2025} instantiate this logic in different ways. Their shared risk is that poor critique, unstable verification, or weak or problematic ecological feedback can steer the system away from $m^\ast$ rather than toward it.
\textbf{Reflective modularization}. Reflective modularization distributes the search for $m^\ast$ across the model's internal or auxiliary components such as retrievers, verifiers, memory systems, action loops, tools, and agent harnesses. Modularization maintains or changes how intermediate structure is produced, thereby where monitoring, evaluation, control, and adaptation are located:
\[
u_i = \mathcal{D}_i(x,c,h_t),
\]where $\mathcal{D}_i$ denotes a module, such as a representation, retriever, verifier, memory component, or tool interface. Modules become metacognitively meaningful only when the broader reflective procedure interprets and routes them:
\[
\{u_i\}_{i=1}^{n} \xrightarrow{M,V} z \xrightarrow{\pi} a \xrightarrow{A} h_{t+1}.
\]
Modularization helps determine $m^\ast$ by deciding when the system has enough evidence or should retrieve more evidence in retrieval-augmented generation \citep{lewis2020, zhou2024}, needs external support through tools or action interfaces \citep{schick2023, yao2023b}, should verify a claim through separate evaluative procedures \citep{dhuliawala2024, gou2024}, should store or retrieve memory for later attempts \citep{shinn2023, zhao2024}, or should escalate or stop through runtime harness control \citep{yang2024, pan2026, he2026}. This is where boundedness and effectiveness meet most directly: modules can impose or ease evidence requirements, verification gates, access rules, cost constraints, and stopping criteria, but they improve self-governance only when uncertainty, insufficiency, or failure is routed toward the right support and under the proper feedback at the ecological level, rather than committed to procedural complexity alone.
\subsection{Ecological level}
At the ecological level, the central question is when metacognitive regulation becomes meaningful in use, where context $c$ acquires its full meaning for self-governance. It specifies the conditions under which an output $y_{t+m^\ast}$ is not only task-effective, but also bounded by evidence, uncertainty, cost, authority, value, institution of its setting.
Let
\[
\mathcal{E}_c=(I,W,Q)
\]
denote the ecological specification of context $c$, where $I$ is the \textit{reflective interface}, $W$ the \textit{reflective workflow}, and $Q$ the \textit{reflective accountability} regime. They are the conditions of uptake through which the metacognitive algorithmic procedure above obtains practical status: $I$ determines whether $z_t$ can be understood as a reason for self-governing judgment; $W$ determines whether $a_t$ can be carried into a course of self-governing action; and $Q$ determines whether $h_t \rightarrow h_{t+1}$ can be justified as an accountable self-governing transition. Together, they subject the task goal $g$ to the contextual limit $L(c,h_{t+k})$ that make the self-governing horizon $m^\ast$ ecologically proper and achievable in context \textit{c}.
We can therefore treat the algorithmic procedure as producing a \textbf{metacognitive decision event }$e_t$:
\[
e_t = (z_t, a_t, h_t \rightarrow h_{t+1}),
\]
whose practical status depends on the contextual ecology in which it occurs:
\[
\operatorname{status}(e_t \mid \mathcal{E}_c).
\]
The same event may support self-governance, be ignored, induce overreliance, create friction, or become institutionally unusable depending on $\mathcal{E}_c$. Ecology therefore determines whether metacognitive regulation can become effective and bounded in practice.
\textbf{Reflective interface}. Reflective interface $I$ concerns the ecological standing of the regulatory signal $z_t$:
\[
I: z_t \mapsto \text{\textit{legible signal}}.
\]
Metacognitive signals reach users through interface forms, and these forms shape how users trust, ignore, verify, or over-rely on AI outputs. Research on automation bias, trust calibration, and human-AI decision-making shows that their effects crucially depend on task structure, cognitive effort, verification costs, user incentives, prior beliefs, among others \citep{parasuraman1997,lee2004,lai2019,green2019,yin2019,bansal2021}. Cognitive forcing and verification-oriented interface designs can reduce overreliance by changing how users engage with AI recommendations, although they may also introduce costs in effort, usability, or workflow efficiency \citep{bucinca2021,vasconcelos2023}. Without reflective interface, $z_t$ may exist algorithmically while failing ecologically: it may be invisible, misread, over-trusted, or disconnected from what should be checked, revised, withheld, or escalated.
\textbf{Reflective workflow}. Reflective workflow $W$ concerns the ecological standing of the control action $a_t$:
\[W: a_t \mapsto \text{\textit{distributed action.}}\]
It concerns the conditions under which $a_t$ becomes embedded in a coordinated course of action across models, users, tools, documents, organizational routines, and institutional procedures. This view extends work on distributed cognition, extended cognition, and situated action, where cognitive control is organized within and across agents, artifacts, environments, and practices \citep{suchman1987,hutchins1995,clark1998,hollan2000,dourish2001}. Recent AI work makes this point increasingly concrete for generative systems by locating distributed reliability particularly in retrieval and evaluation pipelines \citep{gao2024,fan2024}, RAG failure diagnosis \citep{barnett2024}, deployed governance workflows \citep{raji2020,technology2024}.
\textbf{Reflective accountability}. Reflective accountability $Q$ concerns the governance standing of the transition from $h_t$ to $h_{t+1}$: \[Q: h_t \rightarrow h_{t+1} \mapsto \text{\textit{accountable governing}}\] Its function is to make self-regulatory procedure ultimately answerable towards proper regulatory horizon $m^\ast$: recorded, interpretable, contestable, auditable, and connected to responsible roles. Sociotechnical AI research shows that model behavior must be evaluated in relation to deployment context, stakeholder roles, organizational incentives, and downstream harms \citep{selbst2019,suresh2021}. Practitioner-oriented work makes the same point operationally by emphasizing debugging, documentation, auditing, checklists, and accountability processes \citep{holstein2019,mitchell2019,raji2020,madaio2020,gebru2021}. Work on contestable AI further shows that accountable and responsible systems must support challenge, questioning, and redress \citep{alfrink2023}, while governance frameworks treat trustworthy AI as a contextual, organizational achievement rather than a model property alone \citep{kuehnert2025,ojewale2025}. From a metacognitive perspective, reflective governance specifies the accountability conditions under which a system's self-regulation and adaptation can guide action under unavoidable internal and environmental complexity and uncertainty \citep{grossmann2025,smith2025,ji2026,johnson2026}.
\section{Metacognitive alignment towards self-governance requires dynamic, cross-level feedback}
Metacognitive alignment requires feedback because the adequacy of a regulatory target, procedure, or contextual limit cannot be settled at one level alone. A computational target may be well specified but poorly realized; an algorithmic procedure may regulate generation but fail in use; an ecological constraint may be responsible in principle but lack the signal or action needed to make it operative. Feedback is therefore the design through which the three levels test and revise one another.
Let
\[
\Delta_t = F_{\mathcal{E}_c}(e_t)
\]
denote \textbf{cross-level feedback} generated when a metacognitive decision event $e_t$ enters its ecological context $c$ at time $t$ (either as updating an immediate structure $u$ or moving towards producing an output $y$). It returns to the computational and algorithmic levels:
\[
\Delta_t \Rightarrow (M,V,\pi,A,L)_{t+1}.
\]
The cross-level feedback $\Delta_t$ may identify what the regulation monitors, how signals were routed, when the procedure stopped, and whether the selected action could be interpreted, enacted, contested, or assigned to responsible roles (\hyperref[tab:mismatches]{Table~\ref*{tab:mismatches}}). In this sense, $\Delta_t$ tests both whether the ultimate $y_{t+m^\ast}$ is effective and properly bounded in ecological context. Recent work echoes this ecological point: confidence cues and system recommendations can support appropriate reliance or intensify overreliance depending on how they are presented and embedded in use \citep{parasuraman1997,lee2004,bansal2021,bucinca2021,vasconcelos2023}, and governance and contestability work makes a similar point in the institutional context \citep{raji2020,alfrink2023}.
On the other hand, the inclusion of $L$ renders explicit the contextual conditions that both enable and constrain metacognitive regulation (see Section 3.2). Hence, feedback revises $L$ when use reveals that these conditions were mis-specified in context, thereby also reshaping the ecology in which future metacogntive decision events $e_t$ s occur. Contemporary work on human decisions and machine predictions makes this point increasingly visible: algorithmic outputs can reshape the environments in which decision-makers interpret evidence, allocate trust, act, and update knowledge \citep{kleinberg2018,ludwig2021,mullainathan2025,acemoglu2026a,acemoglu2026b}.
In short, feedback $\Delta_t$ navigates and regulates across the three levels: ecology may revise what counts as adequate self-governance, while computational and algorithmic changes may reorganize the ecological conditions under which future metacognitive decision events $e_t$s obtain proper practical status. Together, they move towards stabilizing or changing the self-governing horizon $m^\ast$ within proper temporary, ecological context. Cross-level metacognitive alignment is therefore dynamic across the three levels, concerning what forms of agentive relations the metacognitive decision-making could achieve for bounded and effective self-governance.
\begin{table}[t]
\caption{Diagnosis of examples of cross-level metacognitive misalignment, based on the tripartite framework. The table summarizes how AI governing design may fail when metacognitive targets, procedures, and ecological conditions are misaligned.}
\label{tab:mismatches}
\centering
\scriptsize
\setlength{\tabcolsep}{2.5pt}
\renewcommand{\arraystretch}{1.15}
\begin{tabularx}{\textwidth}{p{0.19\textwidth} p{0.12\textwidth} p{0.12\textwidth} X p{0.18\textwidth}}
\toprule
\textbf{Potential cross-level mismatch} & \textbf{Strength} & \textbf{Weakness} & \textbf{Example} & \textbf{What the framework can add} \\
\midrule
Computationally vague, algorithmically elaborate & Reflective procedure & Meta-level function & Chain-of-thought elicitation \citep{wei2022}; rationale unfaithfulness \citep{turpin2023}; generic reflection prompting \citep{wang2024} & Metacognitive functional specification \\
Computationally valid, algorithmically weak & Valid target & Reliable signal & Verbalized uncertainty \citep{lin2022,xiong2024}; self-knowledge limits \citep{kadavath2022}; confidence calibration instability \citep{tian2023}; generation-verification mismatch \citep{song2024} & Metacognitive signal validation \\
Algorithmically strong, ecologically brittle & Effective procedure & Situated use & Trust calibration failure \citep{lee2004}; human-AI complementarity breakdown \citep{bansal2021}; cognitive forcing need \citep{bucinca2021}; overreliance under AI advice \citep{vasconcelos2023} & Use-context reflectivity \\
Ecologically responsible, computationally underspecified & Governance aim & Triggering signal & Ethics checklist without runtime criteria \citep{madaio2020}; audit process without escalation signals \citep{raji2020}; documentation without intervention thresholds \citep{mitchell2019,gebru2021}; risk workflow without operational monitors \citep{technology2024} & Metacognitively triggered signal-to-action alignment \\
\bottomrule
\end{tabularx}
\end{table}
\section{A Case Study}
The case of algorithmic recourse makes the cross-level metacognitive alignment for self-governance concrete. Consider a loan applicant denied by an automated classifier. A counterfactual explanation may identify a model-flipping change, such as increasing income by a specified amount \citep{wachter2017, wachter2018}. Actionable and causal recourse then ask whether such a change is feasible, interventionally warranted, and connected to a real path from the applicant's current state to the desired outcome \citep{ustun2019, karimi2020, karimi2021}. Yet even a technically valid recommendation is not self-authorizing: it may depend on uncertain causal knowledge, unequal access to remedy, model or distributional change, institutional contingency, and sociocultural recognition, even when the system provisionally treats its evaluation as fair or justified.
In this setting, the metacognitive decision event $e_t = (z_t,a_t,h_t \rightarrow h_{t+1})$ is the point at which the system assesses a candidate recourse recommendation and decides whether it should be presented, qualified, revised, routed, or withheld. Here $z_t$ summarizes the recommendation's regulatory status, including its feasibility, causal warrant, uncertainty under various agentive contexts \citep{khotanlou2025}, while $a_t$ selects and performs the appropriate treatment of that recommendation. Cross-level metacognitive feedback $\Delta_t = F_{E_c}(e_t)$ then diagnoses potential cross-level misalignment: a recommendation may satisfy the classifier but fail as ecologically actionable recourse, appear procedurally fair while lacking causal warrant, or be causally coherent while lacking institutional uptake or contestable status. In such cases, $\Delta_t$ should revise not only the recommendation itself, but also the metacognitive constraint $L(c,h_{t+k})$ under which future recourse outputs are treated as warranted, usable, and accountable. The self-governing horizon $m^\ast$ is reached only when the system has produced a recourse output $y_{t+m^\ast}$ that is sufficient for the object-level goal, helping the applicant understand, contest, or change the decision, while remaining bounded by the metacognitive constraints $L(c,h_{t+k})$ determined by both the algorithm and ecology. Those limits concern not merely causal assumptions or cost as task-level features, but the system's authority to present advice as warranted, actionable, fair, and institutionally usable.
This case therefore clarifies bounded and effective self-governance: recourse is effective when it improves an agent's practical ability to act on or challenge a decision, and bounded when the system regulates the authority, uncertainty, and ecological contingency of its own guidance. Algorithmic recourse thus shows why metacognition should not be an extra, post-hoc layer after generation and prediction. It is the self-governing process through which a system decides why a recommendation is warranted, how it should be qualified, and under what ecological conditions it can become responsibly actionable.
\section{Research Agenda}
A research agenda for bounded and effective self-governance in generative AI should move in four directions. The first is to \textbf{treat capability and governance as mutually enabling }rather than competing aims. Generative systems should and can be evaluated by whether their competence includes the capacity to regulate and govern their own exercise. The second is \textbf{humble, perspectival self-governance} against a purely maximalist view of AI capability \citep{johnson2026}. The field should not define advanced AI only by ever greater task-level behavioral reach, but also by the ability to recognize the authority, uncertainty, and perspective-limits of its own outputs, and to revise, qualify, or defer accordingly. The first and second directions together require the development of new datasets and benchmarks that test \textit{both} task performance \textit{and} whether systems can manage and regulate the exercise of that performance. The third is \textbf{cross-level diagnosis through dynamic feedback}. Metacognitive goals and procedures should be studied as self-governing pathways whose failures may become visible only when use feeds back on what was regulated, how it was regulated, and whether the contextual limits were properly specified (\hyperref[tab:mismatches]{Table~\ref*{tab:mismatches}}). The fourth is \textbf{distributed and higher-order metacognition}. As self-governance increasingly depends on external and distributed supports, the capacity to decide why, how, and when to rely on those supports should be treated as intrinsic to self-governance itself. In particular, self-governing systems must be able not only to regulate task- level activity in the first-order manner, but also to manage their own regulatory behavior through higher-order metacognitive control \citep{langlois2020}.
\section{Alternative Views and Objections}
Our position is sharpened by distinguishing it from three narrower views of metacognition. First, \textbf{metacognition may seem to anthropomorphize AI by importing a human mental model}. Although the term is inspired by human cognition, it does not locate metacognition inside one person's mind or one model's inner state. We use metacognition functionally: It concerns whether information about a system's own states, outputs, and trajectories is organized to regulate later generative behavior, and the locus of such regulation may be internal, scaffolded, or distributed across models, tools, users, organizations, and so on, consistent with computational accounts of metacognition and metareasoning as control over a system's own reasoning processes \citep{russell1991,cox2011}.
Second, \textbf{metacognition may seem to mean deliberative, expensive, and ``slow'' rationality}. This view appears in parts of metacognition and System-2-inspired AI work \citep{wei2022,madaan2023,dhuliawala2024}, often through psychology-inspired dual-process language \citep{kahneman2011,evans2013}. But deliberation does not always improve judgment or performance, as work on bounded and ecological rationality has long emphasized \citep{gigerenzer1996,kruglanski2011,gigerenzer2025a,gigerenzer2025b}. Metacognition is therefore not simply more resource-intensive reflection for better outputs \citep{ji2026,katsikopoulos2026}, and can support resource-saving regulation when the appropriate self-governing horizon is determined in certain contexts.
Third, \textbf{distributed and higher-order metacognition may seem to make self-governance too expansive and costly}. Once metacognition extends to interfaces, workflows, and regulatory supports, self-governance may seem unlimitedly open-ended. But distributed and higher-order metacognition is not unconstrained regulation. In particular, higher-order, meta- metacognition is one crucial capacity to bound regulation itself \citep{langlois2020}. Distributed and higher-order metacognition therefore strengthens rather than weakens the position: it explains how self-governance can remain effective without becoming either isolated inside the model or expanded without limit.
\section{Conclusion}
This paper has argued that generative AI needs metacognition to become bounded and effectively self-governing. By specifying metacognition across computational targets, algorithmic procedures, and ecological arrangements, the account developed here makes self-governance a concrete object of design and evaluation. Metacognition thereby makes it possible to conceive generative AI as both capable and well-governed, rather than treating capability and governance as competing aims.
\bibliographystyle{plainnat}
\bibliography{references}
\end{document}