SkillEval: Decomposing Agent Skill Quality into Interpretable Signals

Jiahui Han^1,2,*, Qinuo Li^2,5,*, Ziheng Peng^2,4,*, Haotian Wu³, Haoze Liu²,
Danfeng Shan¹, Guanchu Wang², Huiqi Deng^1,2,†, Ninghao Liu^3,†

Abstract

Agent skills provide reusable procedural knowledge that helps agents solve specialized tasks. As their use expands, evaluating skill quality becomes increasingly important. Existing evaluations often measure skill quality by testing whether a skill improves performance on specific downstream tasks. However, a reusable skill may apply to multiple task scenarios. Downstream evaluation mainly reflects the compatibility between a skill and the evaluated task, provides only a partial view of skill quality, and does not identify which aspect of the skill should be improved. We find that general properties of the SKILL.md document play an important role in skill quality. To evaluate these properties, we propose SkillEval, an interpretable framework for document-level skill evaluation. SkillEval evaluates each property using a fixed and inspectable scoring direction, producing interpretable scores. It further measures and reduces the influence of unrelated document features, such as length and formatting, so that each score captures its intended semantic property more specifically. Specifically, SkillEval learns an interpretable direction for each quality property from controlled positive–negative skill pairs in the hidden representation space of the model, and scores a new skill by projecting its representation onto these fixed directions. We use SkillEval to evaluate skills in controlled quality tests and show that SkillEval reliably distinguishes skills of different quality. In addition, SkillEval scores closely reflect downstream task performance, providing an early indication of whether a skill is likely to help an agent complete a task. We further explore SkillEval for diagnosing weaknesses in skill documents and guiding targeted revisions. The revised skills improve the targeted properties and achieve higher pass rates on downstream tasks.

^†^†footnotetext: ^* Equal contribution. ^† Corresponding authors.

Introduction

Agent skills increasingly serve as reusable procedural documents that provide agents with task-specific knowledge, tool-use guidance, and procedural workflows at inference time (Jiang et al. 2026; Ling et al. 2026). A skill packages the recurring procedure for a class of tasks into a form that can be stored, retrieved, and reused without retraining the underlying model (Agent Skills n.d.). This makes skills a practical mechanism for accumulating experience in modern agent systems (Liang et al. 2026b; Lei et al. 2026). As agents are deployed in increasingly specialized tasks, skills are becoming a common way to extend agents beyond their base capabilities (Jiang et al. 2026; Ling et al. 2026; Li et al. 2026b).

As skills are increasingly used, the evaluation of skill quality becomes increasingly important. The quality of a skill directly affects how well an agent can understand the task and complete the required workflow. Recent work has studied the evaluation of skills through the outcomes of downstream tasks, such as task pass rates, or through analyses of agent execution trajectories (Li et al. 2026b; Han et al. 2026; Zhong et al. 2026). However, existing evaluations still test skills in specific downstream task environments. This makes the observed performance highly dependent on the compatibility between a skill and the evaluated task (Liu et al. 2026). In addition, the pass rate alone cannot identify which aspect of a skill causes poor performance or how the skill should be improved. For a reusable skill, quality is not determined only by its performance on one task, but also by its ability to support different task scenarios consistently. Therefore, single-task downstream performance provides a useful but partial view of skill quality.

We find that the general properties of a SKILL.md document also play an important role in determining quality of skills. To evaluate these document-level properties, we propose SkillEval, an interpretable framework that evaluates the skill document. SkillEval decomposes skill quality into four complementary dimensions: applicability, content quality, execution guidance, and robustness, as illustrated in Table 1. These dimensions examine when a skill should be used, whether its content is clear and specific, how it guides execution, and how it handles failures and edge cases. In addition, for each learned metric, SkillEval derives an explicit direction from controlled positive–negative skill pairs and scores a document by projecting its representation onto that direction. This fixed scoring procedure makes individual quality signals inspectable and reproducible. It also enables potential biases introduced by document length or formatting to be quantified and corrected by orthogonalizing the metric direction against the corresponding bias direction in hidden space.

Dimension	Metric Name	Definition
Applicability	A1 Format Validity	Whether the skill is a Markdown file with valid YAML frontmatter and the required name and description fields.
Applicability	A2 Trigger Clarity	Whether the description states what the skill does, when it should be used, and its main applicability conditions.
Content Quality	B1 Content Consistency	Whether the capabilities stated in the description are supported by the instructions and resources in the body.
Content Quality	B2 Technical Specificity	Whether the skill provides concrete, executable details, including tools, commands, parameters, fields, units, constraints, and file paths.
Execution Guidance	C1 Workflow Integrity	Whether the skill specifies the required input, processing, and output stages.
Execution Guidance	C2 I/O Explicitness	Whether the expected inputs and outputs, formats, fields, schemas, units, and constraints are explicitly defined.
Robustness	D1 Failure Awareness	Whether the skill covers common failures, boundary conditions, diagnostic signals, and recovery strategies.

Table 1: Intrinsic skill-quality properties used in our evaluation framework.

We use SkillEval to evaluate skills and analyze their downstream utility on SkillsBench (Li et al. 2026b), leading to three key findings: (1) The learned metric directions consistently distinguish positive skills from negative skills across A2, B1, B2, C1, C2, and D1 on the validation set of 518 skills, including 259 positive and 259 negative examples. (2) The predictions of pass-rate uplift, defined as the difference between the pass rate with and without a skill, are strongly correlated with the observed uplift on SkillsBench (Li et al. 2026b), with Pearson $r$ ranging from 0.779 to 0.787. (3) SkillEval identifies weaknesses in skill documents and guides LLMs to address them. These targeted revisions improve the relevant metric scores and increase the mean downstream pass rate by 29.5 percentage points. These findings demonstrate that SkillEval provides interpretable evaluation signals that reflect both document-level quality and downstream skill utility.

Our main contributions are as follows:

•

We propose SkillEval, an interpretable framework that decomposes skill quality into four complementary dimensions. The framework evaluates each dimension through an auditable and reproducible scoring process, producing an interpretable quality profile for each skill.
•

We show that SkillEval effectively distinguishes skills of different quality. In addition, pass-rate uplift predicted from its scores is strongly correlated with the uplift observed on downstream tasks, with Pearson $r$ ranging from 0.779 to 0.787. This correlation provides an early indication of whether a skill is likely to help an agent complete a downstream task successfully.
•

We explore SkillEval as a tool for diagnosing and improving skills. These metric-level scores identify specific weaknesses and guide targeted revisions of skill documents. Our experiments show that the revised skills improve the targeted properties and improve pass rates on downstream tasks.

Related Work

Agent skills package instructions, workflows, and resources for reuse (Liang et al. 2026a). Recent surveys organize this emerging field and discuss systematic evaluation (Jiang et al. 2026; Ding et al. 2026). We group prior work by its source of evaluation evidence. Task-based methods infer skill quality from downstream execution, whereas intrinsic methods assess the skill document without executing a downstream task. They differ in cost, environment dependence, and the evidence they provide.

Task-Based Skill Evaluation

Task-based evaluation measures whether a skill improves an agent on downstream tasks. SkillsBench (Li et al. 2026b) compares matched runs with and without a curated skill. It uses containerized environments and deterministic verifiers across several domains (Li et al. 2026b). SWE-Skills-Bench uses a similar design for software engineering. It pairs public skills with pinned repositories and executable acceptance tests (Han et al. 2026). These benchmarks provide direct evidence of practical utility.

Other work broadens the scope of task-based evaluation (Zhou et al. 2026; Ying et al. 2026). SkillLearnBench separates skill-text quality, trajectory alignment, and final task outcomes (Zhong et al. 2026). Some work generate tasks and scoring rubrics from each skill (Shaposhnikov et al. 2026). AgentSkillOS evaluates skill retrieval and composition over large libraries, with pairwise LLM evaluation of task outputs (Li et al. 2026a). These systems cover more stages of skill use than a single success score (Zhu et al. 2026).

Refer to caption — Figure 1: Overview of SkillEval. The figure shows (a) the dataset configuration, where 1,647 SKILL.md documents from 87 tasks are organized into four quality dimensions; (b) the methodology, where strong and weak skills are contrasted in a frozen LLM representation space to learn metric-specific scoring directions; (c) the skill quality metrics used to evaluate reusable skill documents; and (d) the downstream applications, where the learned scores are used for pass-rate correlation analysis and metric-guided skill revision.

Despite these differences, task-based methods require agent execution. They need task environments and outcome verification. Many also require tool access. Their results can also depend on the base model, agent harness, and resource budget (Li et al. 2026b; Xu et al. 2026). Task-level rewards may mix skill quality with agent capability (Ding et al. 2026). Therefore, this approach is costly for frequent screening. A failed run may also reflect the skill, model, tool, or agent setup.

Intrinsic Skill Quality Evaluation

Agent skills exhibit observable document-level properties, and these properties can affect how skills are discovered, selected, and governed in practice (Ling et al. 2026; Saha et al. 2026). Existing work has begun to assess such intrinsic properties directly. Hong et al. (2026) treat SKILL.md as a software artifact, define skill smells such as weak guidance, missing safeguards, and context bloat, and develop methods to detect them. SkillLearnBench evaluates skill-text quality alongside trajectory alignment and task outcomes, including text-level coverage, executability, and safety (Zhong et al. 2026).

LLM judges are useful when explicit rules are hard to write. MT-Bench and G-Eval show that strong judges can align well with human judgments in dialogue and text evaluation (Zheng et al. 2023; Liu et al. 2023). However, their reliability depends on the evaluation setting. Candidate order can change pairwise judgments (Shi et al. 2025), and non-target information can affect decisions (Chen et al. 2024). Reliability also varies across models, tasks, and evaluated properties (Bavaresco et al. 2025). Code judges can change scores after surface edits that preserve program behavior (Moon et al. 2026). Therefore, repeated generative judgments may change with the judge or the presentation of the input.

Internal Representations for Evaluation

Prior work suggests that high-level concepts can be represented as approximately linear directions in a model’s internal representation space (Zou et al. 2023; Kim et al. 2018; Park et al. 2023). Such directions can be extracted from contrastive examples and used to analyze or influence model behavior (Rimsky et al. 2024; Li et al. 2023). Recent studies further show that internal representations can serve as evaluation signals through learned probes or projections onto interpretable semantic directions (Li et al. 2026c; Ma et al. 2026). To our knowledge, however, prior work has not applied this paradigm to the fine-grained quality evaluation of agent skill documents.

Motivated by this gap, we develop an interpretable framework for evaluating the intrinsic quality of agent skills. SkillEval learns one representation direction for each metric. At evaluation time, the directions remain fixed, and the framework produces dimension-level scores without querying a generative judge.

Proposed Methodology

We propose an interpretable framework for evaluating the general properties of skills across four complementary dimensions: applicability, content quality, execution guidance, and robustness. As shown in Figure 1, these dimensions are represented by seven metrics, comprising one structural-format metric and six semantic metrics. Structural Format Validity (A1) is evaluated using deterministic rules. For each semantic metric, we construct positive–negative skill pairs that differ primarily in the target property. We encode the paired skills with a frozen language model and aggregate their representation differences to derive a metric-specific quality direction. At evaluation time, we project a new skill onto each quality direction and standardize the resulting scores using training-split statistics. Together with the structural check, these scores form an interpretable quality profile for each skill.

Metric Selection and Definitions

Agent skills support diverse tasks, domains, and tool environments (Jiang et al. 2026; Li et al. 2026b). Therefore, task-specific evaluation criteria may not apply uniformly across skills (Ding et al. 2026). We focus instead on intrinsic properties that can be evaluated directly from the skill document. We select metrics that are broadly applicable, observable from SKILL.md, and related to common sources of skill failure. Table 1 summarizes the selected metrics.

Structural Format Validity is checked with deterministic rules. The remaining metrics require semantic judgments and are learned from controlled skill pairs as described below.

Positive–Negative Skill Pair Construction

A reliable metric direction requires positive and negative examples that differ mainly in the target quality dimension. If the two skills also differ in topic, task content, writing style, or length, the learned direction may capture these unrelated changes. Therefore, we use multiple controlled skill pairs to reduce noise from individual examples (Rimsky et al. 2024).

For each learned semantic metric $F_{i}$ , we first select a positive skill $d_{i,j}^{+}$ that meets the metric criteria. We then select its negative version with a metric-specific selection function $T_{i}$ :

d_{i,j}^{-}=T_{i}(d_{i,j}^{+}).

The selected counterpart weakens the target property while keeping the task goal, topic, tools, and other content as unchanged as possible. This design limits unrelated differences and helps the learned direction focus on the target metric.

Hidden Representation and Direction Extraction

We use a frozen Qwen3.5-9B model and extract the hidden-state tensor at index 16. For metric $F_{i}$ and skill $d$ , let

\mathbf{H}^{(16)}_{i}(d)\in\mathbb{R}^{T_{d}\times d_{h}}

denote the token-level hidden states. We pool the region most relevant to each metric:

\mathbf{h}_{i}(d)=\mathrm{Pool}_{i}\!\left(\mathbf{H}^{(16)}_{i}(d)\right)\in\mathbb{R}^{d_{h}}.

A2 uses the frontmatter description. B1 averages the description and body representations. B2, C1, and C2 use mean pooling over the body. D1 uses the final token of the full model input. The model, hidden-state index, prompt, and pooling rule are fixed before validation. The maximum input length is 20,000 tokens, and no document in our dataset is truncated.

For the $j$ -th training pair, we compute

\Delta_{i,j}=\mathbf{h}_{i}\!\left(d_{i,j}^{+}\right)-\mathbf{h}_{i}\!\left(d_{i,j}^{-}\right).

A confounding factor may introduce an unintended direction in the hidden space. Following prior work on removing linearly represented attributes through projection (Ravfogel et al. 2020; Haghighatkhah et al. 2022; Belrose et al. 2023), we use a precomputed unit confounding direction $\mathbf{u}_{\mathrm{conf}}$ , fixed before metric training and shared across semantic metrics. We remove the component aligned with this direction from each training difference:

\Delta_{i,j}^{\perp}=\Delta_{i,j}-\left(\Delta_{i,j}^{\top}\mathbf{u}_{\mathrm{conf}}\right)\mathbf{u}_{\mathrm{conf}}.

The metric direction is the normalized mean of the orthogonalized training differences:

\mathbf{b}_{i}=\frac{\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\Delta_{i,j}^{\perp}}{\left\|\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\Delta_{i,j}^{\perp}\right\|_{2}}.

Algorithm 1 SkillEval Direction Learning and Scoring

0: Paired training sets

\mathcal{P}_{i}=\{(d_{i,j}^{+},d_{i,j}^{-})\}_{j=1}^{N_{i}}

i=1,\ldots,K

, Pooled representation functions

\{\mathbf{h}_{i}(\cdot)\}_{i=1}^{K}

, unit confounding direction

\mathbf{u}_{\mathrm{conf}}

, unseen skill

d

0: Metric directions

\{\mathbf{b}_{i}\}_{i=1}^{K}

and standardized scores

\{z_{i}(d)\}_{i=1}^{K}

1: for

i=1,\ldots,K

\mathcal{D}_{i}^{\mathrm{train}}\leftarrow\bigcup_{j=1}^{N_{i}}\{d_{i,j}^{+},d_{i,j}^{-}\}

3: for

j=1,\ldots,N_{i}

\Delta_{i,j}\leftarrow\mathbf{h}_{i}(d_{i,j}^{+})-\mathbf{h}_{i}(d_{i,j}^{-})

\Delta_{i,j}^{\perp}\leftarrow\Delta_{i,j}-(\Delta_{i,j}^{\top}\mathbf{u}_{\mathrm{conf}})\mathbf{u}_{\mathrm{conf}}

6: end for

\bar{\Delta}_{i}\leftarrow\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\Delta_{i,j}^{\perp}

\mathbf{b}_{i}\leftarrow\bar{\Delta}_{i}/\lVert\bar{\Delta}_{i}\rVert_{2}

\mathcal{S}_{i}^{\mathrm{train}}\leftarrow\left\{\mathbf{h}_{i}(\tilde{d})^{\top}\mathbf{b}_{i}\mid\tilde{d}\in\mathcal{D}_{i}^{\mathrm{train}}\right\}

10:

(\mu_{i}^{\mathrm{train}},\sigma_{i}^{\mathrm{train}})\leftarrow\operatorname{MeanStd}(\mathcal{S}_{i}^{\mathrm{train}})

11:

s_{i}(d)\leftarrow\mathbf{h}_{i}(d)^{\top}\mathbf{b}_{i}

12:

z_{i}(d)\leftarrow\dfrac{s_{i}(d)-\mu_{i}^{\mathrm{train}}}{\sigma_{i}^{\mathrm{train}}}

13: end for

14: return

\{\mathbf{b}_{i}\}_{i=1}^{K},\{z_{i}(d)\}_{i=1}^{K}

Metric Scoring and Normalization

For a new skill $d$ , we compute the metric score by projection:

s_{i}(d)=\mathbf{h}_{i}(d)^{\top}\mathbf{b}_{i}.

We convert the score to a $z$ -score using the mean and standard deviation over all positive and negative training skills:

z_{i}(d)=\frac{s_{i}(d)-\mu_{i}^{\mathrm{train}}}{\sigma_{i}^{\mathrm{train}}}.

Higher values indicate stronger alignment with the positive side of metric $F_{i}$ .

Experiments

Experimental Setup

Dataset Setting. Our dataset contains 1,647 skills from open-source datasets, public online sources, manual construction, and LLM generation. For each semantic metric, we construct controlled positive–negative pairs and split them into training and validation sets, with no overlap in original documents, tasks, sources, or generation templates. All metric directions and normalization statistics are learned exclusively from the training split, while the validation split is used only for final evaluation.

Model Setting. In our main experiments, we use a frozen Qwen3.5-9B model and extract representations at hidden-state index 16. A2 applies mean pooling to the description, B1 combines the mean-pooled representations of the description and body, B2, C1, and C2 apply mean pooling to the body, and D1 uses the final-token representation. The maximum input length is 20,000 tokens. For model ablations, we further use Qwen3.5-27B at index 32, Llama3.1-8B-Instruct at index 16, and Llama3.2-3B-Instruct at index 14.

Downstream Evaluation. We use SkillsBench (Li et al. 2026b) for downstream evaluation and run each task with Codex using GPT-5.5 at the xhigh reasoning effort. For each task, the with-skill and no-skill conditions use the same task environment and executor configuration. Each condition is evaluated over three runs, and the pass rate is calculated as the proportion of successful runs. We define pass-rate uplift as the difference between the pass rates under the with-skill and no-skill conditions.

Main Result

Validation Score Distribution. We first examine whether the learned metric directions preserve the intended quality ordering on unseen skill documents. Each direction is constructed using only the training split and remains fixed during validation. We evaluate 518 held-out documents across A2, B1, B2, C1, C2, and D1, including 259 positive and 259 negative skills. A1 is excluded because it is evaluated using deterministic rules. As shown in Figure 3, positive skills obtain mean scores above zero for all six metrics, whereas negative skills consistently receive scores below zero. Specifically, A2 shows the largest separation, with mean scores of $0.98$ and $-0.96$ for positive and negative skills, respectively. B1 and B2 also exhibit clear gaps of $1.36$ and $1.28$ , followed by D1 with a gap of $1.10$ . Although the separations are smaller for C2 and C1, their positive skills still outperform the corresponding negative skills by $0.86$ and $0.75$ , respectively. These results show that the learned directions recover the intended positive–negative ordering on held-out documents rather than merely memorizing individual training pairs. The variation in separation also provides insight into the properties captured by different directions. Explicit properties, such as trigger-condition clarity in A2, produce a sharper distinction, whereas workflow completeness and input–output clarity require information to be integrated across multiple parts of a document and therefore exhibit smaller gaps. This variation supports decomposing skill quality into metric-specific directions instead of representing it with a single holistic score. The resulting quality profile can distinguish skills of different quality and reveal which properties contribute most strongly to their differences.

Metric Correlation with Downstream Task Pass Rate. We use the SkillsBench (Li et al. 2026b) examples with pass-rate uplift, defined as the difference between the pass rate with skills and the pass rate without skills, fit one linear model per metric to map the metric score to uplift. As shown in Figure 4, all five metric-specific signals show positive correlations with downstream uplift. We do not include A2 in this analysis because A2 only evaluates the clarity of trigger conditions and is less directly related to the execution quality measured by downstream task performance. The correlations are consistently high across B1, B2, C1, C2, and D1, with Pearson $r$ ranging from 0.779 to 0.787, while the all-metric setting obtains $r=0.716$ . The colored regions provide an additional distributional check: negative skills are concentrated more on the low-uplift side, whereas positive skills shift toward regions with larger uplift. This pattern indicates that the fitted linear functions preserve the expected positive–negative ordering of skills in the downstream task space. Together with the validation results in Figure 3, this provides evidence that the learned metric scores are directionally aligned with downstream pass-rate variation.

Skill Diagnosis and Targeted Revision

We further explore whether SkillEval can diagnose weaknesses in skill documents and guide their revision. We select continuous-score tasks from SkillsBench for evaluation and compare three settings: skills before revision (D), LLM-only revisions that use D without access to SkillEval scores (G), and LLM revisions guided by the dimension-level diagnostics of SkillEval (M). As shown in Figure 5, LLM-only revision increases the mean downstream pass rate from 18.6% to 38.3%, while M further improves it to 48.1%, corresponding to gains of 29.5 percentage points over D and 9.8 percentage points over G. The metric profiles show a similar pattern. Compared with D, M improves A2 from 0.914 to 1.019, B1 from 0.896 to 1.022, C1 from $-0.834$ to $-0.546$ , C2 from $-0.631$ to $-0.393$ , and D1 from 0.913 to 1.422, while maintaining a comparable B2 score. M also achieves higher scores than G on five of the six metrics, with C2 remaining nearly unchanged. These results indicate that SkillEval provides useful diagnostic feedback for skill revision. Guided by its metric-level scores, the LLM can focus revisions on weaker properties while retaining existing strengths, producing skills with improved quality profiles and higher downstream pass rates.

Additional Analyses

Bias Analysis. We examine whether SkillEval scores are affected by the length of a skill document. For each semantic metric, we collect the set of skills scored by the frozen Qwen3.5-9B evaluator, represent each skill using its token length and standardized metric score. As shown in Figure 6, most metrics show only weak correlations with length, including A2 ( $r=0.017$ ), B1 ( $r=-0.056$ ), B2( $r=-0.355$ ), C1 ( $r=0.045$ ), C2 ( $r=0.149$ ), and D1 ( $r=0.119$ ), indicating that the learned scores are not mainly determined by document length. Based on this observation, we reduce the influence of document length during scoring. Since the bias introduced by document length can be represented as a direction in the same hidden space as the metric directions, we orthogonalize each metric direction against a precomputed length direction before scoring. This reduces the direct influence of document length while preserving the intended semantic meaning of each metric. More bias removal experiments are provided in Appendix.

Cross-Backbone Robustness of the Metrics. We evaluate scorer robustness by comparing four backbones, Qwen3.5-9B (Q9), Qwen3.5-27B (Q27), Llama3.1-8B (L31), and Llama3.2-3B-Instruct (L32), on six metric scopes (A2, B1, B2, C1, C2, and D1), over 1647 skills. For each scope, we compute the pairwise Pearson correlation matrix over z-scored metric outputs, and then average the six matrices with equal weight. As shown in Figure 7, the scorer outputs are highly consistent across backbones. The two Qwen models reach a Pearson correlation of 0.948, the two Llama models reach 0.913, and the cross-family pairs remain in the 0.811–0.845 range. This suggests that the scoring signal is largely preserved under backbone replacement, indicating a reasonable level of robustness in the scorer choice.

Task Dependence of Downstream Skill Utility We analyze whether the downstream utility of a skill is stable across different task scenarios. In this experiment, we focus on reused human-written skills that appear in multiple downstream tasks with identical skill content. To avoid confounding from skill composition, we only consider tasks where the reused skill is the only available skill. For each skill–task pair, we compute the pass-rate uplift as the difference between the agent performance with the skill and without the skill. Figure 8 presents the task-level uplift of reused skills, where tasks are sorted by uplift within each skill. We observe that the same skill can lead to substantially different improvements across tasks. For example, the reused xlsx skill produces almost no improvement on some tasks, but achieves a much larger uplift on tasks such as weighted-gdp-calc and protein-expression-analysis, with a maximum gap of 66.7 percentage points. In contrast, pddl-skills shows a smaller but still visible variation across its reused tasks, with a gap of 7.4 percentage points. These results suggest that downstream performance is strongly affected by the match between a skill and the evaluated task. A single downstream task can show whether a skill is useful in that specific environment, but it cannot fully represent the overall quality of the skill across broader scenarios. This supports the need for complementary skill-level evaluation signals that assess reusable properties of the skill document.

Conclusion

We introduced SkillEval, an interpretable framework for evaluating the document-level quality of agent skills. By decomposing skill quality into four complementary dimensions, SkillEval produces fine-grained quality profiles through an auditable and reproducible scoring process, while allowing potential biases to be identified and controlled. Our experiments show that these quality signals reliably distinguish skills of different quality and are strongly associated with their utility on downstream tasks.

More broadly, our findings suggest that skill quality involves two complementary aspects: compatibility with a particular task and general properties that support reuse across different scenarios. SkillEval connects these two aspects by providing interpretable document-level signals that are informative of downstream utility. We hope this work provides a foundation for more systematic skill evaluation and supports the development of skill ecosystems in which skills can be efficiently assessed, diagnosed, and refined.

References

Agent Skills (n.d.) Agent Skills Specification. Note: https://agentskills.io/specificationAccessed: 2026-07-26 Cited by: Introduction.
A. Bavaresco, R. Bernardi, L. Bertolazzi, D. Elliott, R. Fernández, A. Gatt, E. Ghaleb, M. Giulianelli, M. Hanna, A. Koller, et al. (2025) Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 238–255. Cited by: Intrinsic Skill Quality Evaluation.
N. Belrose, D. Schneider-Joseph, S. Ravfogel, R. Cotterell, E. Raff, and S. Biderman (2023) Leace: perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems 36, pp. 66044–66063. Cited by: Hidden Representation and Direction Extraction.
G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang (2024) Humans or llms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8301–8327. Cited by: Intrinsic Skill Quality Evaluation.
K. Ding, Y. Zhou, C. Jin, F. Tong, M. Zhou, and D. N. Metaxas (2026) Agent skill evaluation and evolution: frameworks and benchmarks. External Links: 2606.11435, Link Cited by: Task-Based Skill Evaluation, Related Work, Metric Selection and Definitions.
P. Haghighatkhah, A. Fokkens, P. Sommerauer, B. Speckmann, and K. Verbeek (2022) Better hit the nail on the head than beat around the bush: removing protected attributes with a single projection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8395–8416. Cited by: Hidden Representation and Direction Extraction.
T. Han, Y. Zhang, W. Song, C. Fang, Z. Chen, Y. Sun, and L. Hu (2026) SWE-skills-bench: do agent skills actually help in real-world software engineering?. arXiv preprint arXiv:2603.15401. Cited by: Introduction, Task-Based Skill Evaluation.
D. B. Hong, A. Imani, and I. Ahmed (2026) From anatomy to smells: an empirical study of skill.md in agent skills. External Links: 2607.01456, Link Cited by: Intrinsic Skill Quality Evaluation.
Y. Jiang, D. Li, H. Deng, B. Ma, X. Wang, Q. Wang, and G. Yu (2026) SoK: agentic skills – beyond tool use in llm agents. External Links: 2602.20867, Link Cited by: Introduction, Related Work, Metric Selection and Definitions.
B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. (2018) Interpretability beyond feature attribution: quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668–2677. Cited by: Internal Representations for Evaluation.
Y. Lei, Z. Wan, J. Zhang, S. Alam, Z. Zhong, P. Huang, X. Wang, J. Zhang, D. Zhou, Y. Hsieh, et al. (2026) SkillEvolBench: benchmarking the evolution from episodic experience to procedural skills. arXiv preprint arXiv:2605.24117. Cited by: Introduction.
H. Li, C. Mu, J. Chen, S. Ren, Z. Cui, Y. Zhang, L. Bai, and S. Hu (2026a) Organizing, orchestrating, and benchmarking agent skills at ecosystem scale. External Links: 2603.02176, Link Cited by: Task-Based Skill Evaluation.
K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg (2023) Inference-time intervention: eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, pp. 41451–41530. External Links: Link Cited by: Internal Representations for Evaluation.
X. Li, Y. Liu, W. Chen, B. You, Z. Di, Y. He, S. Zheng, K. W. Choe, J. Sun, S. Wang, C. Tao, B. Li, X. Zhao, H. Geng, X. Wu, J. Zhou, X. Chen, H. Xing, Y. Li, Q. Zeng, D. Wang, Y. Wang, R. B. Chaim, P. Jiang, H. Shen, L. Kong, X. Liu, R. Wang, X. Liu, J. Li, X. Lan, Y. Lin, W. Ye, J. He, S. Li, Y. Zhang, Y. Gao, Y. Li, Z. Ma, L. Jing, T. Wang, K. Li, Y. Xue, H. Lyu, Y. He, Y. Tian, S. Wu, B. Wang, Y. Gao, B. Chen, L. Liu, S. Cheng, J. Bao, S. Tong, S. Xu, T. Y. Zhuo, T. Ye, Q. Qi, M. Li, L. Liao, Z. Tan, C. Shi, X. Tang, S. Tankasala, B. Yuan, Y. Qian, J. Tu, C. Wang, Y. Sun, W. Wang, A. Taylor, Z. Yang, C. Guan, Z. Dong, X. Zhang, S. Dillmann, H. Lee, and D. Song (2026b) SkillsBench: benchmarking how well agent skills work across diverse tasks. External Links: 2602.12670, Link Cited by: Introduction, Introduction, Introduction, Introduction, Task-Based Skill Evaluation, Task-Based Skill Evaluation, Metric Selection and Definitions, Experimental Setup, Main Result.
Z. Li, Y. Zhang, M. Li, Y. Ji, Y. Zeng, N. Cheng, Y. Zhu, Y. Wang, S. Wang, J. Xiao, et al. (2026c) Rethinking llm-as-a-judge: representation-as-a-judge with small language models via semantic capacity asymmetry. arXiv preprint arXiv:2601.22588. Cited by: Internal Representations for Evaluation.
Q. Liang, H. Wang, Z. Liang, and Y. Liu (2026a) From skill text to skill structure: the scheduling-structural-logical representation for agent skills. External Links: 2604.24026, Link Cited by: Related Work.
Y. Liang, R. Zhong, H. Xu, C. Jiang, Y. Zhong, R. Fang, J. Gu, S. Deng, Y. Yao, M. Wang, et al. (2026b) Skillnet: create, evaluate, and connect ai skills. arXiv preprint arXiv:2603.04448. Cited by: Introduction.
G. Ling, S. Zhong, and R. Huang (2026) Agent skills: a data-driven analysis of claude skills for extending large language model functionality. External Links: 2602.08004, Link Cited by: Introduction, Intrinsic Skill Quality Evaluation.
Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 2511–2522. External Links: Link, Document Cited by: Intrinsic Skill Quality Evaluation.
Y. Liu, J. Ji, L. An, T. Jaakkola, Y. Zhang, and S. Chang (2026) How well do agentic skills work in the wild: benchmarking llm skill usage in realistic settings. External Links: 2604.04323, Link Cited by: Introduction.
X. Ma, X. Zhang, and Z. Weng (2026) Stable and explainable personality trait evaluation in large language models with internal activations. In Findings of the Association for Computational Linguistics: ACL 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), San Diego, California, United States, pp. 16322–16340. External Links: Link, Document, ISBN 979-8-89176-395-1 Cited by: Internal Representations for Evaluation.
J. Moon, Y. Hwang, D. Lee, T. Kang, Y. Kim, and K. Jung (2026) Don’t judge code by its cover: exploring biases in llm judges for code evaluation. In Findings of the Association for Computational Linguistics: EACL 2026, pp. 1364–1389. Cited by: Intrinsic Skill Quality Evaluation.
K. Park, Y. J. Choe, and V. Veitch (2023) The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658. Cited by: Internal Representations for Evaluation.
S. Ravfogel, Y. Elazar, H. Gonen, M. Twiton, and Y. Goldberg (2020) Null it out: guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 7237–7256. Cited by: Hidden Representation and Direction Extraction.
N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner (2024) Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 15504–15522. External Links: Link, Document Cited by: Internal Representations for Evaluation, Positive–Negative Skill Pair Construction.
S. Saha, K. Faghih, and S. Feizi (2026) Under the hood of skill. md: semantic supply-chain attacks on ai agent skill registry. arXiv preprint arXiv:2605.11418. Cited by: Intrinsic Skill Quality Evaluation.
M. Shaposhnikov, N. Fortuin, S. Stipcich, M. I. Gorinova, A. Heineike, and R. Willoughby (2026) A framework for evaluating agentic skills at scale. External Links: 2606.17819, Link Cited by: Task-Based Skill Evaluation.
L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. Vosoughi (2025) Judging the judges: a systematic study of position bias in llm-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pp. 292–314. Cited by: Intrinsic Skill Quality Evaluation.
Y. Xu, L. Li, L. Sleem, N. Gentile, Y. Song, Y. Wang, S. Ji, W. Wu, and R. State (2026) Agent skill framework: perspectives on the potential of small to medium language models in industrial environments. arXiv preprint arXiv:2602.16653. Cited by: Task-Based Skill Evaluation.
J. Ying, B. Ai, W. Tang, S. Liu, and Y. Cao (2026) OpenSkillEval: automatically auditing the open skill ecosystem for llm agents. External Links: 2605.23657, Link Cited by: Task-Based Skill Evaluation.
L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Cited by: Intrinsic Skill Quality Evaluation.
S. Zhong, Y. Lu, J. Ning, Y. Wan, L. Feng, Y. Ao, L. F. R. Ribeiro, M. Dreyer, S. Ammirati, and C. Xiong (2026) SkillLearnBench: benchmarking continual learning methods for agent skill generation on real-world tasks. External Links: 2604.20087, Link Cited by: Introduction, Task-Based Skill Evaluation, Intrinsic Skill Quality Evaluation.
Y. Zhou, Z. Zhang, Z. Cheng, S. Zhang, Q. Lan, Z. Chen, Z. Yang, QianyuXu, R. Chen, H. Wang, and S. Hu (2026) SkillGenBench: benchmarking skill generation pipelines for llm agents. External Links: 2605.18693, Link Cited by: Task-Based Skill Evaluation.
J. Zhu, K. Mao, Y. Guo, D. He, S. Xu, S. Gu, and Y. Yue (2026) SkillCoach: self-evolving rubrics for evaluating and enhancing agentic skill-use. External Links: 2607.01874, Link Cited by: Task-Based Skill Evaluation.
A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, et al. (2023) Representation engineering: a top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. Cited by: Internal Representations for Evaluation.

Appendix A Bias Diagnostic.

We further examine whether SkillEval scores are affected by common biases that are not part of the intended semantic meaning of each metric. In this experiment, we select representative bias variables for each metric and compute their Pearson correlations with the corresponding z-scored metric outputs.

Specifically, for A2, we test description length, defined as the token length of the trigger description, since A2 is evaluated from the trigger description. For B1, we test document length, code block count, and identifier density, where document length is measured by the token length of the skill body, code block count measures the number of Markdown code blocks, and identifier density measures the density of technical identifiers such as API names, file names, field names, and numeric expressions. These variables may reflect document scale or formatting rather than description-body alignment.

For B2, we test model token length and code block count, since specificity scores may be affected by longer or more code-heavy skill documents. For C1, we test numbered step count, sequential marker count, and stage coverage. Numbered step count measures explicit ordered steps, sequential marker count measures temporal markers such as “first”, “then”, and “finally”, and stage coverage measures whether the skill covers input, processing, and output stages. These variables correspond to common workflow-format cues.

For C2, we test model token length, schema-line count, and unit/type count, where schema-line count measures lines describing input fields, columns, keys, units, types, or constraints, and unit/type count measures explicit unit or data-type expressions. These variables may affect input-output explicitness. For D1, we test model token length to examine whether failure-awareness scores are mainly driven by skill length.

As shown in Figure 9, most selected bias variables show weak correlations with the corresponding metric scores. A2 has only a weak correlation with description length ( $r=0.120$ ). B1 is weakly correlated with document length ( $r=-0.056$ ), code block count ( $r=-0.042$ ), and identifier density ( $r=-0.024$ ). B2 shows a moderate negative correlation with model token length ( $r=-0.355$ ) and a weak correlation with code block count ( $r=-0.118$ ). C1 is weakly correlated with numbered step count ( $r=0.152$ ), sequential marker count ( $r=-0.059$ ), and stage coverage ( $r=0.094$ ). C2 also shows weak correlations with model token length ( $r=0.149$ ), schema-line count ( $r=0.211$ ), and unit/type count ( $r=0.015$ ). For D1, the correlation between model token length and the metric score is also weak ( $r=0.119$ ).

These results suggest that most SkillEval scores are not mainly determined by the selected surface-level biases. At the same time, the diagnostic still reveals metric-specific sensitivities, such as the moderate length effect observed for B2. This provides a practical audit mechanism for SkillEval: after a metric direction is learned, we can test whether its scores align with intended semantic properties or with unintended document cues. When a bias shows a visible correlation, it can be further controlled during metric construction or removed by orthogonalizing the metric direction against the corresponding bias direction in hidden space.

Appendix B Cross-Model Robustness of Skill Utility.

We further evaluate whether the downstream utility of skills is stable across different execution models. The evaluation data are provided by SkillsBench, which contains downstream task results under both with-skill and no-skill conditions. Following the SkillsBench setting, we use results from 87 tasks evaluated with 18 downstream executor models.

For each task and executor model, we compute the raw uplift as the pass-rate difference between the with-skill and no-skill conditions. Since each condition is evaluated with three trials, the smallest pass-rate resolution is $1/3$ . We therefore use a tolerance-aware pairwise concordance metric with margin $0.333$ : for each pair of executor models, we compare task pairs only when the uplift gap is at least this margin for both models, and count whether the two models rank the task pair in the same order.

As shown in Figure 10, the average concordance is $0.724$ and the median concordance is $0.739$ . In total, 103 out of 153 model pairs have concordance above $0.7$ , and 146 out of 153 model pairs are above $0.5$ . Several model pairs show particularly strong agreement, such as Codex/GPT-5.5 and OpenHands/Opus 4.8 ( $0.956$ ), and Claude Code/Opus 4.7 and OpenHands/Opus 4.7 ( $0.931$ ). At the same time, a small number of model pairs have lower agreement, with the minimum concordance observed between OpenHands/GPT-5.4 Mini and OpenHands/Grok 4.3 ( $0.385$ ).

These results indicate that skill-induced uplift is largely consistent across different downstream executor models, suggesting that the measured utility of skills is not tied to a single execution model. However, the lower-concordance cases also show that executor choice can still affect the relative ordering of skill benefits. Therefore, downstream skill evaluation is reasonably robust across models, but cross-model analysis remains useful for identifying model-specific sensitivity.

•

Appendix C Hidden-State Layer Analysis

Experimental Setup. We examine how the choice of hidden-state layer affects the metric directions learned by SkillEval. Using the frozen Qwen3.5-9B model, we sweep hidden-state indices from 4 to 32 at intervals of four. For each metric, we use the same metric-specific training pairs across layers and retain the pooling rule used in the main experiments.

For metric $i$ and hidden-state index $\ell$ , we project the pooled representation of a skill $d$ onto the corresponding metric direction and standardize the resulting score using the training-split statistics:

z_{i}^{(\ell)}(d)=\frac{s_{i}^{(\ell)}(d)-\mu_{i}^{(\ell)}}{\sigma_{i}^{(\ell)}}.

We evaluate each layer using two complementary measures. Validation AUROC measures how well the standardized scores distinguish positive skills from negative skills. Specifically, we fit a one-dimensional logistic classifier to the standardized training scores and compute AUROC from its predicted probabilities on the corresponding validation split. The mean paired margin measures the average standardized score difference within matched positive–negative validation pairs:

M_{i}^{(\ell)}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\left[z_{i}^{(\ell)}(d_{i,j}^{+})-z_{i}^{(\ell)}(d_{i,j}^{-})\right],

where $N_{i}$ is the number of validation pairs for metric $i$ . A larger AUROC indicates better discrimination between positive and negative skills, while a larger paired margin indicates stronger separation within matched pairs.

Validation AUROC

Figure 11 compares the validation AUROC obtained from different hidden-state indices. A2 achieves an AUROC of 1.000 at every sampled layer, while B2 remains stable between 0.970 and 0.984. B1 is more sensitive to layer choice: its AUROC increases from 0.508 at layer 4 to 0.913 at layer 16 and reaches its maximum of 0.943 at layer 20. At layer 16, C1 and C2 achieve AUROCs of 0.701 and 0.721, respectively, compared with their sampled maxima of 0.719 at layer 8 and 0.735 at layer 20. D1 obtains an AUROC of 0.889 at layer 16, while its highest value of 0.992 occurs at layers 4 and 8.

For A2, B1, B2, C1, and C2, the AUROC at layer 16 is within 0.031 of the best sampled result. D1 exhibits greater variation across layers, although its AUROC at layer 16 remains substantially above the chance level of 0.5. These results indicate that the layer producing the strongest discrimination varies across metrics, while layer 16 retains reliable discrimination for all six metrics.

Mean Paired Margin

We further evaluate each layer using the mean paired margin. As shown in Figure 12, all six metrics retain positive margins at layer 16, indicating that positive skills receive higher scores than their matched negative counterparts on average. The margins at layer 16 are 1.939 for A2, 1.356 for B1, 1.288 for B2, 0.754 for C1, 0.861 for C2, and 1.105 for D1.

Layer 16 produces the largest sampled margins for A2, B1, and C1. For C2, its margin at layer 16 differs from the sampled maximum at layer 4 by less than 0.001. B2 and D1 obtain their largest margins at layer 32, reaching 1.340 and 1.628, respectively. The layers that maximize the paired margin therefore do not always coincide with those that maximize AUROC, showing that score separation and ranking performance provide complementary views of each metric direction.

Taken together, these results show that no single hidden-state layer is optimal for every metric.

Appendix D Pooling Strategy Analysis

Experimental Setup. We further examine how the pooling strategy affects the metric directions learned by SkillEval. At hidden-state index 16, we compare two general pooling strategies while keeping the model, prompts, skill pairs, and evaluation procedure fixed. Full-mean pooling averages the hidden representations of all tokens in the complete input sequence, whereas last-token pooling uses only the representation of the final token. For each pooling strategy, we reconstruct the metric direction from the same metric-specific training pairs and evaluate it on the corresponding fixed validation split.

The mean paired margin is the average difference between the standardized scores of matched positive and negative validation skills. Validation AUROC is computed from the probabilities produced by a one-dimensional logistic classifier fitted to the standardized training scores. Figure 13 presents the results for all six learned metrics.

Mean Paired Margin. As shown in Figure 13(a), the pooling strategy affects the separation between matched positive and negative skills differently across metrics. Full-mean pooling produces larger margins for A2 and B1, increasing the margin from 0.414 to 0.857 for A2 and from 0.397 to 1.459 for B1. It also gives a slightly larger margin for C2, with 0.816 compared with 0.768 under last-token pooling. In contrast, last-token pooling produces larger margins for B2, C1, and D1. Their margins increase from 1.262 to 1.889, from 0.804 to 1.400, and from 0.328 to 1.162, respectively. Averaged equally across the six metrics, the mean paired margin is 0.921 for full-mean pooling and 1.005 for last-token pooling.

Validation AUROC. Figure 13(b) shows a similar metric-dependent pattern in ranking performance. Full-mean pooling performs better for A2 and B1, achieving AUROCs of 0.731 and 0.926, compared with 0.621 and 0.610 under last-token pooling. Last-token pooling achieves higher AUROC for the remaining four metrics. Specifically, it improves B2 from 0.973 to 1.000, C1 from 0.703 to 0.991, C2 from 0.722 to 0.749, and D1 from 0.632 to 0.881. Its macro-average AUROC is 0.809, compared with 0.781 for full-mean pooling.

These results indicate that pooling preference is metric dependent. Full-mean pooling is substantially stronger for A2 and B1, whereas last-token pooling provides better ranking performance for B2, C1, C2, and D1. C2 further shows that the strategy with the larger paired margin does not necessarily achieve the higher AUROC, confirming that paired separation and ranking performance capture complementary properties. Therefore, the results do not support a single pooling strategy for all metrics and instead motivate selecting the pooling scope according to the semantic property evaluated by each metric.

Appendix E Metric-Specific Dataset Construction

Data Sources and Common Procedure.

We construct a separate dataset for each of the six learned semantic metrics using three complementary sources. First, LLMs generate complete positive and negative SKILL.md documents according to metric-specific instructions. Second, we collect publicly available skill documents from online sources and screen them as positive or negative examples based on the corresponding metric criteria. Therefore, the negative examples are not limited to controlled rewrites of positive skills: they also include LLM-generated documents and collected documents that naturally exhibit the weakness measured by a metric. Third, we construct controlled positive–negative pairs using metric-specific operations when a clearer comparison is needed. The same criteria are used to filter generated, collected, and constructed examples. During controlled construction, we preserve the task goal, topic, tools, document structure, and non-target properties as much as possible, so that the main difference within each pair is the property evaluated by the corresponding metric. All examples retain the data split of their source task to prevent related skills from appearing in both training and validation sets. A1 is excluded because Structural Format Validity is evaluated directly using deterministic rules.

A2: Trigger Clarity

The A2 dataset contains LLM-generated and online-collected skills whose descriptions either clearly specify or fail to specify when the skill should be used. Positive examples state an explicit trigger condition and connect it to a concrete task scenario, whereas negative examples provide vague conditions or only summarize the tool’s general capability. For controlled pairs, an LLM rewrites only the frontmatter description, replacing a clear trigger condition with a general tool or capability description. The skill body and all other frontmatter fields remain unchanged. We also keep the positive and negative descriptions similar in length. Generated, collected, and rewritten examples are screened using the same trigger-expression and task-scenario criteria.

B1: Content Consistency

The B1 dataset combines generated and collected skills with either consistent or inconsistent descriptions and bodies. Positive examples describe the same task in the frontmatter and body, whereas negative examples contain a description that does not match the procedural content. To construct additional controlled negatives, we preserve the original frontmatter while replacing the body with one taken from a different skill. Replacement bodies are matched to the original body in length where possible. This operation changes description–body consistency while preserving the description, metadata format, and approximate document scale. We record the source of each replacement body and check that the frontmatter remains unchanged and that the paired bodies are different.

B2: Technical Specificity

The B2 dataset includes generated and collected skills with different levels of technical specificity. Positive examples contain task-related technical details, such as API names, function calls, parameters, configuration values, file names, schema fields, numerical constants, units, and usable code or command examples. Negative examples include LLM-generated or collected skills that provide only general technical guidance. For controlled pairs, an LLM rewrites a specific skill by replacing concrete technical details with more general descriptions while preserving the frontmatter, task topic, main workflow, input–output setting, and overall structure. The rewritten document is kept close to the original in length. All candidates are screened using the amount and density of task-related technical evidence.

C1: Workflow Integrity

The C1 dataset contains generated and collected skills with complete or incomplete workflows. Positive examples provide an ordered procedure that connects the input, intermediate operations, and final output. Negative examples include skills with missing stages, disordered steps, weak transitions, or disconnected recommendations. We construct additional controlled negatives by asking an LLM to reorder dependent steps, remove an important transition, replace an ordered procedure with unordered suggestions, or make the path from input to output incomplete. The task, tools, input–output setting, and most technical details are retained. The resulting examples are screened to ensure that workflow integrity is reduced without substantially changing the task content or other quality properties.

C2: I/O Explicitness

The C2 dataset combines generated and collected skills with different levels of input–output explicitness. Positive examples specify inputs and outputs using concrete formats, fields, columns, keys, schemas, units, data types, constraints, or output labels. Negative examples include generated and collected documents in which this information is missing or expressed only in general terms. For controlled construction, we preserve the frontmatter, task, tools, APIs, and main workflow while removing or generalizing input–output details. Explicit descriptions such as CSV fields, JSON keys, units, types, and output schemas are replaced with general references to input data or result summaries. Content unrelated to the input–output contract is kept unchanged, and large deletions are avoided to control document length.

D1: Failure Awareness

The D1 dataset contains LLM-generated and online-collected skills with different levels of failure awareness. Positive examples describe potential failures, boundary conditions, incorrect practices, diagnostic signals, or recovery strategies. Negative examples include generated and collected skills that omit such guidance or mention it only superficially. For controlled pairs, rule-based processing first removes complete failure-related sections and standalone warnings. An LLM then performs minimal edits to failure-related clauses embedded in otherwise useful sentences. The frontmatter and unrelated procedural content are preserved. The resulting examples are screened for failure-related sections, explicit prohibitions, descriptions of common mistakes, explanations of failure causes, and corrective guidance.