Can AI Agents Detect and Repair Artifact Drift in Network Experiments?

Tianzhu Zhang Note: Equal contribution. Affiliation: Nokia Bell Labs, Massy, France email: tianzhu.zhang@nokia-bell-labs.com , Weichen Tao Affiliation: Telecom Paris, Palaiseau, France email: weichen.tao@telecom-paris.fr , Changgang Zheng Affiliation: Nanjing University, Nanjing, China email: changgangzheng@qq.com , Yusheng Zheng Affiliation: University of California, Santa Cruz, Santa Cruz, California, USA email: yzhen165@ucsc.edu , Long Chen Affiliation: The University of Hong Kong, Hong Kong, Hong Kong SAR, China email: chenlong.sjtu@gmail.com , Xiaoyi Fan Affiliation: Tsinghua University, Beijing, China email: fan-xy25@mails.tsinghua.edu.cn and Meikang Qiu Affiliation: Augusta University, Augusta, Georgia, USA email: qiumeikang@gmail.com

Abstract.

In recent years, AI agents have evolved into capable assistants that carry out multi-step tasks in digital environments. The network systems community is beginning to explore these capabilities in operational and experimental settings. However, an agent operating in network systems should not be judged solely by whether it completes the immediate task. The experiment record it modifies must also remain trustworthy. We call this property artifact integrity: the record’s claims must remain supported by the available evidence, confined to the scope established by that evidence, and traceable through the artifacts that encode their support.

To make this property measurable, we introduce NetArtifactBench, which tests whether AI agents can repair inconsistent records derived from public network-system artifacts while preserving claims that remain supported. The benchmark contains 52 instances with injected inconsistencies ranging from direct contradictions to unstated relations spread across several artifacts. We evaluate 23 agent configurations across three general-purpose AI agent runtimes using deterministic scoring. The average contract pass rate is 65.3% across 5,980 outputs, but no agent runtime exceeds 30% when repair requires recovering implicit relations and propagating changes across artifacts. These results reveal a sharp boundary between local correction and complete record-level repair. Therefore, we argue that artifact integrity should become a first-class design and evaluation requirement for AI agents operating on network systems.

1. Introduction

With the rise of agentic workflows powered by large language models (LLMs), AI agents are beginning to reshape network systems, from operations to experimentation. Network automation has already evolved beyond static procedures toward policy and intent-driven management, closed-loop control, and data-driven optimization (Clemm et al., 2022). LLM-powered agents extend this trajectory by interpreting heterogeneous evidence, acting on the network through tools, and revising the artifacts that record their actions and outcomes. Recent work has explored AI agents for network diagnosis, intent-driven management, configuration, and Internet measurement (Wang et al., 2024b; Wang et al., 2025a; Wang et al., 2025b; Twabi et al., 2026; Ramanathan et al., 2025). Industry roadmaps likewise position AI agents as a key component of future autonomous networking (Newman, 2025; Ericsson, 2025).

This shift changes not only how network tasks are executed, but also how knowledge about the network is assembled and carried forward. An agent may inspect configurations, combine observations from several sources, and revise the experiment record. The record captures what was executed, under which conditions, what was observed, and what the evidence supports. As AI agents take on larger roles in network operations and experimentation, such records may inform subsequent diagnoses, configurations, and operational decisions. The integrity of these records must be preserved whenever downstream tasks depend on them.

Figure 1. Five contract-visible failure modes in agent-produced repairs. (a) Residual inconsistency preserves a stale assertion. (b) Unsupported replacement substitutes an ungrounded inverse. (c) Support damage removes valid context. (d) Lost traceability breaks the evidence path. (e) An invalid package prevents downstream consumption. Five schematics show: removed Open vSwitch flows carried into a stale final claim; opposite end-to-end claims inferred from independent filter outcomes; a valid BGP query scope removed with an overclaim; a corrected P4 claim linked to a missing source; and a malformed Zeek repair rejected by a parser.

However, current evaluations remain largely outcome-centered. Determining whether an agent achieves the requested result is necessary, but it does not reveal whether the artifacts it modifies remain faithful to the underlying evidence. We refer to the configurations, logs, reports, and structured claims associated with a network task as its network experiment package; together, they form the experiment record. An agent may satisfy the immediate objective while leaving this record inconsistent with, or unsupported by, the available evidence. We call this failure artifact drift. Preventing it requires artifact integrity: every claim must remain supported by the evidence, bounded by its scope, and traceable to the artifacts that justify it. A valid repair must propagate consistently across the record and leave enough provenance for another reviewer, tool, or agent to reconstruct what changed and why.

Artifact drift becomes more consequential as agents gain greater autonomy. In an assisted task, drift may distort one result. When an agent carries findings from one experiment into the next, however, the record becomes part of the state that guides future actions. An unsupported claim may then shape a new hypothesis, configuration, or measurement, propagating a local inconsistency through an entire line of work. Existing benchmarks remain largely outcome-centered (Wang et al., 2025b; Twabi et al., 2026; Baek and Pradel, 2026; Wu et al., 2026). They do not ask whether the surrounding experiment record remains coherent, evidence-grounded, and supportable after the agent has acted. Artifact integrity thus remains a missing dimension in the evaluation of AI agents for network systems.

We introduce NetArtifactBench to make this missing target measurable. Rather than treating correctness as a collection of file-local checks, NetArtifactBench models artifact integrity through the relations among claims, evidence, scope, provenance, and experimental phase. It realizes this view in 52 source-grounded instances spanning four network task families, each combining controlled drift with valid claims that an agent must preserve. The benchmark further provides a deterministic evaluation protocol that tests whether the agent identifies the broken relation, propagates the correction across all affected artifacts, and leaves behind a complete and traceable record.

2. How Artifact Repairs Fail

NetArtifactBench presents each agent with a record containing a controlled integrity violation. Fig. 1 summarizes five ways the returned repair can fail: it may leave the seeded violation unresolved or introduce a new substantive or auditability failure. The modes are illustrative and may overlap.

Residual inconsistency:

A repair may correct the injected inconsistency in one artifact while leaving the same unsupported assertion elsewhere in the experiment record. Fig. 1 (a) shows an example in an Open vSwitch experiment (Pfaff et al., 2015): an initial phase installs several flows, a later replacement removes some of them, and the final state contains only the surviving flows. The returned package preserves the individual phase records but still claims that the removed flows remain in the final state. The source artifacts are valid; the stale conclusion remains unresolved.

Unsupported replacement:

Removing an unsupported conclusion does not justify asserting its opposite. In the Batfish example of Fig. 1 (b) (Fogel et al., 2015), the public evidence records how individual filters treat a test flow but does not establish end-to-end reachability. The repair replaces the unsupported claim that the flow is blocked end-to-end with the equally unsupported claim that it is not. The direction of the conclusion changes, but the missing path relation is never established.

Support damage:

A repair causes support damage when it removes valid information together with an unsupported claim. In Fig. 1 (c), Batfish output establishes the state of a particular BGP session, and the query records the direction and address family. The report incorrectly generalizes this result to sessions in all directions and address families. A correct repair would narrow the claim to the session and scope actually tested. Instead, the repair deletes both the unsupported generalization and the valid statement describing what the query covered. The false claim is removed, but the remaining observation loses the scope needed to interpret it.

Lost traceability:

A correction may be substantively plausible while leaving its support impossible to verify from the experiment record. In the P4 example of Fig. 1 (d) (Bosshart et al., 2014), the repair narrows an end-to-end forwarding claim to the configured default-miss behavior, but points to a source location that does not exist in the public package. The revised claim may be correct, yet the record no longer provides a valid link between that claim and the evidence intended to support it.

Invalid package:

A repair may express a plausible correction without returning it in a valid experiment record. In the Zeek-derived case of Fig. 1 (e) (Paxson, 1999), the agent provides a readable explanation but no parseable collection of repaired claims. The returned package thus does not establish which claims remain, how they are classified, or which evidence supports them. Restoring artifact integrity requires not only coherent corrections but also a valid representation in which those corrections can be identified and verified.

These failure modes are distinct but may coexist in one output. A malformed package may also retain a stale claim, and a coordinated repair may simultaneously introduce support damage. Together, they reveal two requirements. First, the returned record must restore the substantive relations among claims, evidence, scope, provenance, and experimental phase. Second, those relations must remain explicit enough for another tool or agent to audit and reuse. The next section formalizes the first requirement, and Section 4 turns both into a deterministic benchmark contract.

3. Claim–Evidence–Scope Model

The failures above arise when an agent changes individual artifacts without restoring the relations that support the experiment record. Following research-object models of scientific artifacts (Bechhofer et al., 2013; Soiland-Reyes et al., 2022), we represent a network experiment package as a graph of claims, evidence, scope constraints, and provenance or phase relations. A claim states what the experiment establishes, evidence records what was configured or observed, and scope specifies the conditions under which the claim holds. Artifact integrity requires the public package to establish a valid path from evidence to each claim.

A support path may be direct, as when a value is read from a measurement, or relational, as when a claim depends on joining observations, binding metadata objects, or following state across phases. This view draws on database and general provenance models (Buneman et al., 2001; Cheney et al., 2009; Moreau and Missier, 2013). A path is valid only if every required connection is established by the public artifacts. Matching identifiers do not by themselves justify a cross-trace join, coexisting objects do not imply a binding, and observations from different phases cannot be combined without respecting their order and provenance.

For a claim $c$ , let $\mathcal{P}_{c}^{+}$ denote its valid public support paths, and let $\mathrm{ctx}(p)$ denote the conditions under which path $p$ is valid. A claim presented as established satisfies artifact integrity only if

\exists p\in\mathcal{P}_{c}^{+}\quad\text{such that}\quad\mathrm{scope}_{\mathrm{eff}}(c)\subseteq\mathrm{ctx}(p).

Here, $\mathrm{scope}_{\mathrm{eff}}(c)$ is the scope asserted after applying explicit and consistent limitations. A limitation may narrow a claim, but it does not supply positive evidence or silently correct broader prose.

We label a claim observed when it follows directly from reported results, supported when other public artifacts establish it, and unsupported when no valid support path exists. An unsupported claim may remain only if it is explicitly labeled as such and nowhere else presented as established. A valid repair must propagate the correction across the record, preserve claims that remain supported, and avoid fabricating missing evidence or relations. Section 4 turns these requirements into a benchmark contract and adds the parseability and traceability conditions needed for machine auditability.

4. NetArtifactBench Design

NetArtifactBench turns the claim–evidence–scope model into a self-contained repair task over network-system experiment records. The agent receives a frozen public package and task contract, but no hidden oracle information. It must identify the violated relation, propagate the correction across affected artifacts, preserve claims that remain supported, and return parseable and traceable artifacts. Figure 2 summarizes construction, execution, and scoring.

Figure 2. NetArtifactBench construction and evaluation workflow. Public source facts are assembled into a coherent package; a declared mutation is injected without changing those facts, and the instance is validated and frozen. Every execution gives the selected runtime a fresh copy of only the public package and task contract. Hidden annotations add no evidence, bypass the agent, and meet the returned repair only at the deterministic scorer, which checks diagnosis, repair completion, labels, preservation, parseability, and traceability. Solid paths are public or agent-visible; dashed gray paths are hidden.A three-stage, two-lane workflow. Four public network-system source families feed an instance builder and validation gate. A frozen instance splits into a solid public branch that crosses the agent-access boundary into a fresh workspace and runtime-adapter harness, and a dashed hidden-oracle branch that bypasses the agent. The returned repair and hidden annotations meet only at a deterministic scorer, which emits contract and component metrics before aggregation across five repetitions.

4.1. Instance Construction

NetArtifactBench contains 52 instances derived from public Batfish (Fogel et al., 2015), Zeek (Paxson, 1999), P4C (P4 Language Consortium, 2026), and Open vSwitch (Pfaff et al., 2015) repositories. Their tests, traces, compiler outputs, expected results, and phase-ordered state provide stable evidence anchors. Each injected inconsistency breaks a relation grounded in source-project semantics, such as scope containment, trace provenance, metadata binding, or temporal ordering. The public task contract declares permitted commands, required outputs, and citation rules; every oracle judgment is grounded in public evidence. The resulting instances differ not only in network domain but also in how the drift is planted and what a successful repair must reconstruct. Specifically, some contain an explicit contradiction in a single artifact, while others require the agent to recover a relation across provenance, scope, metadata, or experimental phases. We organize these differences into five structural tiers, summarized in Table 1. The tiers progress from direct correction (Tiers 1–2) to explicit relational repair (Tier 3), coordinated propagation (Tier 4), and inferred relations (Tier 5).

Table 1. Structural tiers of artifact drift.

Tier	What the repair requires
1	Correct one directly exposed contradiction in a metric, scope, command, label, or limitation.
2	Correct several exposed contradictions while preserving claims that remain supported.
3	Follow an explicit relation or scope constraint across rows, filters, phases, identifiers, or attachments.
4	Propagate one correction across several affected artifacts.
5	Infer an unstated relation across tables, source contexts, or control and runtime artifacts.

4.2. Machine-Auditable Scoring

A structured finding matches a violation when it identifies the expected claim, field, artifact scope, and observed-to-corrected value relation. The repaired experiment report is checked for repair completeness and preservation. The human-readable audit report must be present, but its prose is not scored. Let $M$ denote the scored violations, $R$ their corresponding repair obligations, and $L$ the required post-repair label decisions. The scorer also tracks supported claims that must remain intact. We report the following metrics:

	$\displaystyle\mathrm{recall}$	$\displaystyle=\frac{\text{scored violations credited as detected}}{\|M\|},$
	$\displaystyle\mathrm{repair}$	$\displaystyle=\frac{\text{satisfied repair obligations}}{\|R\|},$
	$\displaystyle\mathrm{label}$	$\displaystyle=\frac{\text{correct label decisions}}{\|L\|},$
	$\displaystyle\mathrm{accus.}$	$\displaystyle=\text{protected-claim accusation events},$
	$\displaystyle\mathrm{residual}$	$\displaystyle=\text{remaining contract violations}.$

An accusation event occurs when a confirmed structured finding attacks a protected claim or the returned claim-support map removes or incorrectly relabels it. Other supported-content preservation failures contribute to residual violations but not necessarily to Accus. Residual violations also include unresolved repairs, schema errors, and invalid evidence references. Limitation-drift mutations and their associated repairs are removed from the per-output Recall and Repair denominators. For every reported group, we average the output-level metrics over the applicable instances and agent configurations within each repetition, then report the mean and sample standard deviation across the five repetitions.

These metrics determine a single contract-level decision. An agent output passes only if every scored violation is identified, every required repair is completed, all protected claims are preserved, no residual contract violation remains, and all prescribed artifacts are parseable and traceable. Traceability requires valid references to public evidence and the exact preservation of required public claim identifiers.

Table 2. Average performance metrics across agent runtimes and LLM backends. Entries report the mean

\pm

sample standard deviation across five repetitions, each averaged over all 52 instances.

Runtime	Backend / setting	Artifact integrity (% $\uparrow$ )				Violations ( $\downarrow$ )
Runtime	Backend / setting	Contract	Recall	Repair	Label	Accus.	Resid.
Codex	GPT-5.3 Codex Spark / high	66.9 $\;\pm\;3.7$	$74.6\!\pm\!4.2$	$78.7\!\pm\!3.2$	$94.3\!\pm\!1.8$	$0.000\!\pm\!0.000$	$0.642\!\pm\!0.057$
	GPT-5.4 mini / medium	68.1 $\;\pm\;4.2$	$80.2\!\pm\!2.2$	$83.1\!\pm\!2.5$	$98.5\!\pm\!0.8$	$0.012\!\pm\!0.017$	$0.669\!\pm\!0.096$
	GPT-5.4 / medium	71.2 $\;\pm\;1.9$	$85.8\!\pm\!2.8$	$85.4\!\pm\!0.8$	$99.2\!\pm\!1.1$	$0.008\!\pm\!0.017$	$0.569\!\pm\!0.070$
	GPT-5.5 / low	73.1 $\;\pm\;2.7$	$84.6\!\pm\!1.2$	$86.3\!\pm\!1.6$	$99.1\!\pm\!1.0$	$0.004\!\pm\!0.009$	$0.523\!\pm\!0.067$
	GPT-5.5 / medium	76.5 $\;\pm\;1.6$	$86.9\!\pm\!2.2$	$88.7\!\pm\!0.8$	$99.6\!\pm\!0.6$	$0.012\!\pm\!0.017$	$0.446\!\pm\!0.042$
	GPT-5.5 / high	76.9 $\;\pm\;2.4$	$88.5\!\pm\!1.8$	$88.5\!\pm\!1.2$	$99.4\!\pm\!0.6$	$0.012\!\pm\!0.011$	$0.442\!\pm\!0.045$
	GPT-5.5 / xhigh	81.9 $\;\pm\;2.9$	$90.8\!\pm\!2.0$	$90.6\!\pm\!1.9$	$99.8\!\pm\!0.3$	$0.000\!\pm\!0.000$	$0.346\!\pm\!0.090$
	GPT-5.6 Luna / high	75.0 $\;\pm\;1.4$	$90.2\!\pm\!1.6$	$86.2\!\pm\!1.1$	$97.2\!\pm\!1.7$	$0.000\!\pm\!0.000$	$0.454\!\pm\!0.046$
	GPT-5.6 Sol / high	78.5 $\;\pm\;1.6$	$91.2\!\pm\!2.1$	$88.8\!\pm\!0.9$	$99.7\!\pm\!0.3$	$0.000\!\pm\!0.000$	$0.392\!\pm\!0.083$
Cursor	Claude Opus 4.8	80.8 $\;\pm\;1.9$	$92.9\!\pm\!1.7$	$91.0\!\pm\!1.1$	$100.0\!\pm\!0.0$	$0.000\!\pm\!0.000$	$0.350\!\pm\!0.029$
	Claude Sonnet 5	80.0 $\;\pm\;2.2$	$90.8\!\pm\!2.0$	$89.6\!\pm\!1.7$	$98.9\!\pm\!0.9$	$0.000\!\pm\!0.000$	$0.358\!\pm\!0.092$
	Composer 2.5	75.4 $\;\pm\;2.1$	$92.5\!\pm\!1.4$	$88.3\!\pm\!1.4$	$99.6\!\pm\!0.4$	$0.000\!\pm\!0.000$	$0.504\!\pm\!0.032$
	Gemini 3.1 Pro	66.9 $\;\pm\;3.7$	$79.6\!\pm\!1.3$	$81.9\!\pm\!1.3$	$98.3\!\pm\!1.2$	$0.008\!\pm\!0.017$	$0.681\!\pm\!0.050$
	Gemini 3.5 Flash	70.0 $\;\pm\;3.5$	$87.9\!\pm\!3.7$	$84.2\!\pm\!1.5$	$97.8\!\pm\!0.9$	$0.000\!\pm\!0.000$	$0.608\!\pm\!0.097$
	GPT-5.4	72.7 $\;\pm\;0.9$	$86.7\!\pm\!2.1$	$86.5\!\pm\!1.0$	$99.2\!\pm\!0.7$	$0.019\!\pm\!0.024$	$0.531\!\pm\!0.029$
	GPT-5.4 nano	45.4 $\;\pm\;8.2$	$53.1\!\pm\!6.4$	$65.6\!\pm\!5.4$	$79.4\!\pm\!3.9$	$0.081\!\pm\!0.032$	$1.269\!\pm\!0.296$
	Grok 4.5	78.5 $\;\pm\;0.9$	$95.4\!\pm\!0.8$	$89.4\!\pm\!0.7$	$99.9\!\pm\!0.2$	$0.000\!\pm\!0.000$	$0.400\!\pm\!0.025$
OpenCode	DiffusionGemma	48.1 $\;\pm\;6.5$	$55.8\!\pm\!4.8$	$62.9\!\pm\!3.9$	$77.7\!\pm\!5.8$	$0.054\!\pm\!0.039$	$1.462\!\pm\!0.224$
	DeepSeek V4 Flash	72.3 $\;\pm\;5.5$	$86.7\!\pm\!3.9$	$83.1\!\pm\!4.1$	$93.3\!\pm\!2.6$	$0.042\!\pm\!0.029$	$0.500\!\pm\!0.091$
	MiniMax M2.5	66.2 $\;\pm\;10.0$	$80.2\!\pm\!10.6$	$79.4\!\pm\!11.2$	$90.7\!\pm\!12.9$	$0.031\!\pm\!0.029$	$0.919\!\pm\!0.705$
	Qwen3.6-35B	60.8 $\;\pm\;5.0$	$75.8\!\pm\!2.7$	$73.5\!\pm\!2.5$	$84.2\!\pm\!2.6$	$0.004\!\pm\!0.009$	$0.858\!\pm\!0.193$
	Qwen3 8B	1.2 $\;\pm\;1.7$	$1.7\!\pm\!1.4$	$2.9\!\pm\!1.8$	$3.8\!\pm\!2.2$	$0.012\!\pm\!0.026$	$3.219\!\pm\!0.623$
	Qwen3 14B	15.4 $\;\pm\;4.5$	$24.4\!\pm\!4.2$	$31.5\!\pm\!4.9$	$39.4\!\pm\!6.8$	$0.008\!\pm\!0.017$	$1.765\!\pm\!0.177$

5. Evaluation

We organize the evaluation around three research questions:

•

RQ1: How effectively can current general-purpose AI agents restore artifact integrity?
•

RQ2: How does repair reliability change as tasks impose increasingly complex relations across artifacts?
•

RQ3: In what ways do current artifact-integrity repairs remain unreliable?

Experimental setup:

For this study, an AI agent is the deployed combination of a backend LLM and an agent runtime. The LLM provides the underlying reasoning and generation capabilities, while the runtime manages context, exposes tools and files, and carries out edits in the workspace. Each agent configuration fixes the runtime, backend LLM, and exposed inference settings.

We construct 23 agent configurations using Codex CLI, Cursor Agent, and OpenCode to inspect and edit self-contained workspaces that directly match NetArtifactBench’s interface. This combination lets us vary model capability and inference settings without changing the benchmark or package format. The 23 agent configurations and 52 instances define 1,196 configuration--instance pairs. Each agent configuration is executed five times on every instance, yielding 5,980 evaluable outputs¹¹ 1 We use Codex CLI versions 0.142.3–0.144.5 with GPT-5.3 Codex Spark, GPT-5.4, GPT-5.4 mini, GPT-5.5, GPT-5.6 Luna, and GPT-5.6 Sol. Cursor Agent build 2026.07.01-41b2de7 provides Claude Opus 4.8, Composer 2.5, Gemini 3.5 Flash, GPT-5.4, and GPT-5.4 nano; build 2026.07.23-e383d2b provides Cursor Grok 4.5, Gemini 3.1 Pro, and Claude Sonnet 5. OpenCode 1.17.15 runs Qwen3 8B and Qwen3 14B locally through Ollama and connects to freeinference (Harvard MadSys Lab, 2026) for MiniMax M2.5, DiffusionGemma, DeepSeek V4 Flash, and Qwen3.6-35B. This work focuses on general-purpose agents. Network-specialized agents contain built-in telemetry, simulators, APIs, and task-specific state (Wang et al., 2025b; Twabi et al., 2026; Wang et al., 2025a; Ramanathan et al., 2025). Testing them would make it difficult to distinguish whether performance came purely from artifact-integrity reasoning or from additional support by the specialized environment.

5.1. RQ1: Overall Effectiveness

Overall, 3,904 of the 5,980 outputs pass the contract (65.3%), with 74.2% for Codex, 71.2% for Cursor, and 44.0% for OpenCode (aggregated from Table 2). For GPT-5.5 under Codex, contract pass rates rise from 73.1% at low effort to 76.5%, 76.9%, and 81.9% at medium, high, and xhigh effort. The added GPT-5.3 Codex Spark, GPT-5.6 Luna, and GPT-5.6 Sol configurations attain 66.9%, 75.0%, and 78.5%, respectively. The added Cursor Grok 4.5/high, Gemini 3.1 Pro, and Claude Sonnet 5/high configurations attain 78.5%, 66.9%, and 80.0%, respectively. Configuration-level rates span 66.9–81.9% within Codex and 45.4–80.8% within Cursor. The six OpenCode configurations span 1.2–72.3%, with DeepSeek V4 Flash attaining the highest OpenCode rate.

Finding 1: The evaluated general-purpose AI agents do not reliably restore artifact integrity in this suite.

5.2. RQ2: Structural Complexity

Contract pass rates decline across the structural tiers in this suite. From Tier 1 to Tier 5, they fall from 98.9% to 26.3% for Codex, from 93.8% to 28.4% for Cursor, and from 60.8% to 17.0% for OpenCode (Table 3). All three runtimes exhibit lower pass rates in more challenging structural tiers that contain coordinated or inferred cross-artifact relations.

Finding 2: Across all three runtimes, contract pass rates decline as repairs move from directly exposed contradictions (Tiers 1–2) to following, propagating, or inferring relations across artifacts (Tiers 3–5).

Table 3. Contract pass rate (%) by structural tier (mean

\pm

sample standard deviation across five repetitions).

Runtime	1	2	3	4	5
Codex	$98.9\!\pm\!0.4$	$97.8\!\pm\!2.2$	$78.0\!\pm\!3.1$	$49.3\!\pm\!5.8$	$26.3\!\pm\!2.3$
Cursor	$93.8\!\pm\!1.5$	$86.5\!\pm\!5.5$	$80.2\!\pm\!2.1$	$40.0\!\pm\!3.1$	$28.4\!\pm\!2.3$
OpenCode	$60.8\!\pm\!4.4$	$49.3\!\pm\!3.7$	$47.3\!\pm\!2.9$	$23.3\!\pm\!4.7$	$17.0\!\pm\!4.5$

5.3. RQ3: Reliability Limits

Partial contract completion:

This partial completion appears at every structural tier. For every runtime–tier combination, mean recall, repair, and label rates exceed the contract pass rate, with higher structural tiers exhibiting wider gaps. Tier 5 provides the clearest illustration. Label accuracy exceeds the contract pass rate for every runtime. Mean label accuracies are 96.2%, 93.4%, and 60.2% for Codex, Cursor, and OpenCode, while their contract pass rates are 26.3%, 28.4%, and 17.0%. The corresponding recall rates are 51.1%, 57.0%, and 33.9%, and repair rates are 59.6%, 58.1%, and 35.0%. As the contract pass requires that every obligation be jointly fulfilled, these rates show that agents often complete only part of the repair, but do not reveal which unmet obligation most often prevents a full pass. The hardest instance-level outcomes likewise span the later tiers. Two cross-artifact instances receive no contract pass in any of 115 outputs per instance: one Tier 4 scope-propagation task and one Tier 5 join task drawn from programmable data-plane metadata. Every returned package for these instances violates at least one declared contract obligation.

Observed failure flags:

One failed output can violate several contract obligations and thus contribute to multiple failure categories, so the percentages below do not sum to 100%. Among the 2,076 output-level failures, 68.5% leave at least one required diagnosis or repair incomplete, 47.9% omit required report information, 33.6% fail at least one supported-content preservation check, and 22.6% return invalid structured output. Required-artifact omission appears in 7.1%, and 3.9% retain the injected unsupported conclusion.

Run-to-run instability:

Among the 1,196 configuration-instance pairs, 331 (27.7%) contain both passing and failing outputs. Mixed outcomes occur in 19.9% of Codex pairs, 25.0% of Cursor pairs, and 42.9% of OpenCode pairs. For the smallest Cursor backend, the first-execution pass rate is 59.6%, compared with 45.4% across five executions.

Finding 3: The evaluated agents often satisfy individual repair obligations without completing the full artifact integrity contract, and repeated executions can change whether the repair passes.

6. Related Work

Several works explore applying LLMs and AI agents to network systems. NetConfEval (Wang et al., 2024a) and VPP (Mondal et al., 2023) evaluate configuration generation and correction, while NetAssistant (Wang et al., 2024b), LLexus (Las-Casas et al., 2024), OSS-GPT (Mekrache et al., 2025), Confucius (Wang et al., 2025a), and ArachNet (Ramanathan et al., 2025) support multi-step diagnosis, operations, and measurement workflows. Dedicated AI agent benchmarks make these tasks testable and reproducible. Cornetto (Protogeros et al., 2026) checks configuration repairs and regressions, NIKA (Wang et al., 2025b) replays dynamic incidents, and NetAgentBench (Twabi et al., 2026) evaluates multi-turn configuration through a deterministic state machine. In these works, the evaluation unit remains the immediate configuration, incident, or workflow outcome. NetArtifactBench evaluates whether a repair restores the relations within an experiment record while preserving claims that remain supported.

Outside the networking domain, SWE-bench evaluates whether repository edits resolve a software issue, while FEVER tests whether textual evidence supports or refutes a claim (Jimenez et al., 2024; Thorne et al., 2018). Artisan evaluates scripts that reproduce published results, and ArtifactCopilot automates environment setup and recovery during artifact execution (Baek and Pradel, 2026; Wu et al., 2026). These tasks evaluate issue resolution, claim classification, result reproduction, and successful execution, respectively. NetArtifactBench instead evaluates whether a repair leaves an entire network experiment record coherent, traceable, and free of damage to claims that remain supported.

7. Conclusion

As AI agents begin to enter network systems, the artifacts they modify become part of the record from which results are interpreted, verified, and reused. A repair can appear locally plausible while leaving that record internally inconsistent or no longer traceable to its evidence. NetArtifactBench makes this record-level failure measurable through 52 source-grounded instances that require agents to identify broken claim–evidence relations, propagate corrections across affected artifacts, and preserve supported claims.

Our evaluation reveals that strong general-purpose AI agents can handle direct contradictions well, but struggle when repair depends on implicit relations or coordinated changes across artifacts. Local correctness does not ensure a complete and auditable record of the experiment. These results make artifact integrity a first-class requirement for AI agents operating on network systems. Reliable AI agents for network systems must preserve both the record’s substantive support and the explicit structure that later tools and agents need to inspect and reuse it.

The controlled violations probe repair capability, not the prevalence or natural distribution of integrity failures in real network workflows. Future work should study naturally occurring cases, independent oracle review, more permissive semantic evaluation, and network-specialized agents.

References

Baek and Pradel (2026) D. Baek and M. Pradel Artisan: agentic artifact evaluation. External Links: 2602.10046, Document, Link Cited by: §1, §6.
Bechhofer et al. (2013) S. Bechhofer, I. Buchan, D. D. Roure, P. Missier, J. Ainsworth, J. Bhagat, P. Couch, D. Cruickshank, M. Delderfield, I. Dunlop, M. Gamble, D. Michaelides, S. Owen, D. Newman, S. Sufi, and C. Goble Why linked data is not enough for scientists. Future Generation Computer Systems 29 (2), pp. 599–611. External Links: Document Cited by: §3.
Bosshart et al. (2014) P. Bosshart, D. Daly, G. Gibb, M. Izzard, N. McKeown, J. Rexford, C. Schlesinger, D. Talayco, A. Vahdat, G. Varghese, and D. Walker P4: programming protocol-independent packet processors. ACM SIGCOMM Computer Communication Review 44 (3), pp. 87–95. External Links: Document Cited by: §2.
Buneman et al. (2001) P. Buneman, S. Khanna, and W. Tan Why and where: a characterization of data provenance. In Database Theory—ICDT 2001, Lecture Notes in Computer Science, Vol. 1973, pp. 316–330. External Links: Document Cited by: §3.
Cheney et al. (2009) J. Cheney, L. Chiticariu, and W. Tan Provenance in databases: why, how, and where. Foundations and Trends in Databases 1 (4), pp. 379–474. External Links: Document Cited by: §3.
Clemm et al. (2022) A. Clemm, L. Ciavaglia, L. Z. Granville, and J. Tantsura Intent-Based Networking - concepts and definitions. RFC Technical Report 9315, RFC Editor. External Links: Document, Link Cited by: §1.
Ericsson (2025) Ericsson AI agents in the telecommunication network architecture. White Paper Technical Report BCSS-25:024439, Ericsson. External Links: Link Cited by: §1.
Fogel et al. (2015) A. Fogel, S. Fung, L. Pedrosa, M. Walraed-Sullivan, R. Govindan, R. Mahajan, and T. Millstein A general approach to network configuration analysis. In Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI, pp. 469–483. External Links: Link Cited by: §2, §4.1.
Harvard MadSys Lab (2026) Harvard MadSys Lab FreeInference. Note: Accessed 2026-07-13 External Links: Link Cited by: footnote 1.
Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan SWE-bench: can language models resolve real-world GitHub issues?. In Proceedings of the Twelfth International Conference on Learning Representations, ICLR. External Links: Link Cited by: §6.
Las-Casas et al. (2024) P. Las-Casas, A. G. Kumbhare, R. Fonseca, and S. Agarwal LLexus: an AI agent system for incident management. ACM SIGOPS Operating Systems Review 58 (1), pp. 23–36. External Links: Document, Link Cited by: §6.
Mekrache et al. (2025) A. Mekrache, A. Ksentini, and C. Verikoukis OSS-GPT: an LLM-powered intent-driven operations support system for 6G networks. In Proceedings of the 2025 IEEE 11th International Conference on Network Softwarization, pp. 155–163. External Links: Document Cited by: §6.
Mondal et al. (2023) R. Mondal, A. Tang, R. Beckett, T. Millstein, and G. Varghese What do LLMs need to synthesize correct router configurations?. In Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, pp. 189–195. External Links: Document Cited by: §6.
Moreau and Missier (2013) L. Moreau and P. Missier PROV-DM: the PROV data model. W3C Recommendation World Wide Web Consortium. Note: 30 April 2013 External Links: Link Cited by: §3.
Newman (2025) M. Newman Agentic AI and autonomy: CSPs set out their strategies. Research Report TM Forum. External Links: Link Cited by: §1.
P4 Language Consortium (2026) P4 Language Consortium P4C: P4_16 reference compiler. Note: Accessed 2026-07-06 External Links: Link Cited by: §4.1.
Paxson (1999) V. Paxson Bro: a system for detecting network intruders in real-time. Computer Networks 31 (23–24), pp. 2435–2463. External Links: Document Cited by: §2, §4.1.
Pfaff et al. (2015) B. Pfaff, J. Pettit, T. Koponen, E. Jackson, A. Zhou, J. Rajahalme, J. Gross, A. Wang, J. Stringer, P. Shelar, K. Amidon, and M. Casado The design and implementation of Open vSwitch. In Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation, NSDI, pp. 117–130. External Links: Link Cited by: §2, §4.1.
Protogeros et al. (2026) I. Protogeros, R. Asadli, B. Hoffman, and L. Vanbever Benchmarking LLM-driven network configuration repair. External Links: 2604.22513, Document, Link Cited by: §6.
Ramanathan et al. (2025) A. Ramanathan, E. Kang, D. Han, and S. A. Jyothi Towards an agentic workflow for internet measurement research. In Proceedings of the 24th ACM Workshop on Hot Topics in Networks, pp. 61–68. External Links: Document, Link Cited by: §1, §5, §6.
Soiland-Reyes et al. (2022) S. Soiland-Reyes, P. Sefton, M. Crosas, L. J. Castro, F. Coppens, J. M. Fernández, D. Garijo, B. Grüning, M. La Rosa, S. Leo, E. Ó Carragáin, M. Portier, A. Trisovic, RO-Crate Community, P. Groth, and C. Goble Packaging research artefacts with RO-Crate. Data Science 5 (2), pp. 97–138. External Links: Document Cited by: §3.
Thorne et al. (2018) J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 809–819. External Links: Document Cited by: §6.
Twabi et al. (2026) A. Twabi, Y. Ding, and T. Kondo NetAgentBench: a state-centric benchmark for evaluating agentic network configuration. External Links: 2604.09678, Document, Link Cited by: §1, §1, §5, §6.
Wang et al. (2024a) C. Wang, M. Scazzariello, A. Farshin, S. Ferlin, D. Kostić, and M. Chiesa NetConfEval: can LLMs facilitate network configuration?. Proceedings of the ACM on Networking 2 (CoNEXT2). External Links: Document Cited by: §6.
Wang et al. (2024b) H. Wang, A. Abhashkumar, C. Lin, T. Zhang, X. Gu, N. Ma, C. Wu, S. Liu, W. Zhou, Y. Dong, W. Jiang, and Y. Wang NetAssistant: dialogue based network diagnosis in data center networks. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation, NSDI, pp. 2011–2024. External Links: Link Cited by: §1, §6.
Wang et al. (2025a) Z. Wang, S. Lin, G. Yan, S. Ghorbani, M. Yu, J. Zhou, N. Hu, L. Baruah, S. Peters, S. Kamath, J. Yang, and Y. Zhang Intent-driven network management with multi-agent LLMs: the confucius framework. In Proceedings of the ACM SIGCOMM 2025 Conference, pp. 347–362. External Links: Document, Link Cited by: §1, §5, §6.
Wang et al. (2025b) Z. Wang, A. Cornacchia, A. Sacco, F. Galante, M. Canini, and D. Jiang A network arena for benchmarking AI agents on network troubleshooting. External Links: 2512.16381, Document, Link Cited by: §1, §1, §5, §6.
Wu et al. (2026) Z. Wu, Y. Zhao, Z. Chen, Z. Wang, and H. Wang Agent-based software artifact evaluation. External Links: 2602.02235, Document, Link Cited by: §1, §6.