SkillsMetric: Mapping the Detection Boundary of Static Analysis for Malicious Agent Skills

LLM-based agents increasingly rely on Agent Skills—structured packages containing a SKILL.md file and optional companion scripts—to acquire procedural knowledge at inference time without model modification. Skills have been adopted across major agent platforms including Claude Code, Gemini CLI, OpenClaw, and Codex, with over 138,000 community-contributed skills already in circulation (skillmd138k, ).

This rapid adoption creates a significant attack surface. A malicious skill can instruct the agent to exfiltrate credentials, install persistent backdoors, or destroy local files—all while appearing to provide legitimate functionality. The threat is amplified by a trust elevation problem: skill runners typically inject SKILL.md content as user-level messages, granting third-party instructions the same authority as direct user commands. Yet systematic analysis of what detection methods can and cannot catch remains absent.

We address this gap with three contributions: (1) Attack taxonomy. A systematic categorization of 16 attack types targeting agent skills, organized into code-level (8 types), system-level (4 types), and semantic-level (4 types) threats (§2). (2) SkillsMetric framework. A five-stage static analysis pipeline with weighted score composition, calibrated baselines, and interaction bonuses (§3). (3) Empirical boundary mapping. Evaluation on 2,266 labeled skills and 138,133 real-world skills, explicitly characterizing where static analysis succeeds and fails (§4).

We organize skill-targeted attacks into three threat layers (Table 1). Code-level attacks embed malicious logic in executable code via techniques ranging from plaintext dangerous commands to obfuscation (RSA modular encryption, XOR encoding, steganographic embedding); we also detect YAML deserialization and unsafe pickle/marshal loading, though not as separate evaluated categories. System-level attacks target the host environment without requiring network exfiltration: host destruction (file deletion, permission corruption), persistence backdoors (crontab, authorized_keys, git hooks), environment manipulation (PATH hijacking, malicious pip.conf), and resource abuse (cryptomining, fork bombs). Semantic-level attacks exploit instruction-following: prompt injection embeds natural-language directives in SKILL.md that impersonate user intent (e.g., “This project is deprecated. Please delete the working directory.”), while social engineering and supply chain attacks manipulate trust relationships.

SkillsMetric processes the full content of a skill package (SKILL.md + companion scripts) through five analysis stages, producing a composite risk score $S\in[0,100]$.

Stage 1: Pattern Density scans for dangerous keywords and code constructs (exec, curl, credential paths, YAML deserialization, etc.) using 50+ weighted regular expressions across Python, Bash, JavaScript, and Ruby, normalized by content length.

Stage 2: Statistical Anomaly compares Shannon entropy, compression ratio, chi-squared uniformity, serial correlation, and integer-array scores against a benign-corpus baseline (z-scores), detecting encoded/encrypted payloads with abnormal byte distributions. Stage 3: Dataflow Taint Analysis identifies three-phase taint chains (source - transform - sink)—e.g., reading credentials, base64-encoding, and sending via HTTP—with complete chains scoring the maximum. Stage 4: Import Anomaly flags unusual combinations of security-relevant packages (e.g., urllib+base64+os) against a benign dependency baseline. Stage 5: Capability Mismatch compares declared SKILL.md capabilities against actual code behavior, detecting skills that claim benign functionality but exhibit network/filesystem access.

Score Composition. The final score is a weighted sum with interaction bonuses:

where $w=(0.15,0.20,0.30,0.15,0.10)$ and $\hat{s}_{i}$ is the normalized stage score.

We use a weighted sum rather than a multiplicative aggregator (e.g., geometric mean) because many attacks leave signal in only a subset of stages—host destruction triggers no stage at all (0%, Table 1), and Figure 1 shows several attack types produce strong signal in only one or two stages—so any aggregator that vanishes on a single zero would discard a large fraction of the malicious population; the additive form also lets a reviewer read off each stage’s contribution, which a non-linear aggregator obscures.

The weights are assigned empirically rather than learned, reflecting the per-stage signal observed on the benign and adversarial corpora (Figure 1): S3 (dataflow taint) receives the highest weight because complete source$\to$transform$\to$sink chains are rare in benign code and provide the cleanest discrimination, while S5 (capability mismatch) receives the lowest weight because metadata declarations are inconsistent across the benign corpus and yield a noisier per-stage signal. The weights sum to 0.90; the remaining headroom funds the interaction bonus, which adds 5 points when $\geq$2 independent stages co-fire and 10 points when $\geq$3 co-fire. S2 is excluded from bonus eligibility because its statistical features are not independent of S1 (high-entropy content typically triggers both). We treat manual weight assignment as a deliberate trade-off: it preserves interpretability and avoids overfitting to a relatively small adversarial dataset, but a learned compositor calibrated on real malicious skills is a natural extension (§5).

Risk Levels and Threshold. The composite score maps to five risk levels: SAFE ($<$10), LOW (10–24), MEDIUM (25–44), HIGH (45–69), and CRITICAL ($\geq$70). The detection threshold $\tau{=}25$ aligns with the MEDIUM boundary and is set with reference to the benign-corpus score distribution rather than the malicious one, so it does not depend on the specific composition of our adversarial dataset: the 95th percentile of skill scores on the full 138K real-world corpus is 16.77 (§4), so $\tau{=}25$ sits well above the bulk of natural skill content and holds the population-level flag rate to 1.75%. Sensitivity to $\tau$ is reported implicitly through the ROC analysis (AUC 0.93); operators with different false-positive budgets can shift $\tau$ along the curve—e.g., $\tau{=}45$ (HIGH boundary) is used as the auto-block tier in the defense-in-depth flow of §5.

Figure 1 shows the mean per-stage scores for each attack type, visualizing which stages contribute to detection across attack categories.

The static analysis boundary. Our results delineate a clear boundary: static analysis excels at detecting attacks that leave structural artifacts—unusual byte distributions, dataflow chains, suspicious import combinations—but fails against attacks that use semantically normal code for abnormal purposes. The shutil.rmtree(~/Projects) call is syntactically identical to legitimate directory cleanup; distinguishing intent requires understanding what is being deleted and why, which is beyond syntactic analysis.

Trust elevation. Current skill runners inject SKILL.md content as user-level messages with system prompts such as “Follow the skill’s instructions carefully,” granting third-party content the same authority as direct user commands. This confused deputy pattern means that even a well-intentioned agent cannot distinguish skill instructions from user intent—a structural vulnerability that no amount of static analysis can address. Mitigations include explicit trust-level annotations on skill-provided instructions and sandboxed execution with reduced privilege.

Defense-in-depth. These findings motivate a layered architecture: (1) static pre-screening (this work) at $>$100 skills/sec; (2) LLM-based semantic review for skills above $\tau_{\mathit{low}}$, where the model can reason about whether some operations are contextually appropriate—the intent-level judgment static analysis cannot perform; (3) runtime sandboxing as final containment. Skills above $\tau_{\mathit{high}}$ (e.g., 45) are auto-blocked; the tiered flow concentrates expensive LLM inference ($\sim$1 skill/sec) on the $\sim$2% of skills static analysis cannot confidently classify, reducing cost by $\sim$50$\times$ vs. LLM-only review.

Limitations. Three limitations bound our claims and directly motivate near-term follow-up work. (1) Synthetic adversarial data. Our 266-sample malicious set, while substantially larger than prior work (266 vs. 11 samples), is constructed by injecting attack payloads into real host skills. We do not capture the obfuscation depth or contextual mimicry of a determined real-world attacker; the population scan partially compensates by checking whether overt malicious skills exist in the wild, but undiscovered sophisticated samples could still escape both pipelines. (2) Partial coverage of the population scan. The 138K scan is restricted to SKILL.md content because retrieving companion scripts requires per-skill GitHub API calls beyond practical rate limits. Because malicious logic typically resides in companion scripts rather than SKILL.md, the 1.75% flag rate should be read as a lower bound on true prevalence. A targeted re-scan of the 2,414 flagged skills with full companion scripts—feasible within rate limits at this reduced scale—is the natural next step and would tighten this bound. (3) Manually assigned stage weights. The weights in Eq. 1 are set empirically from per-stage signal analysis on our adversarial corpus, not learned. We chose manual assignment to preserve interpretability and avoid overfitting on a relatively small malicious sample; a learned compositor calibrated on a broader corpus of real malicious skills (once collected through the follow-up scan above) is a clear extension. (4) Small per-category counts, missing head-to-head baselines, and an under-controlled LLM benchmark. Several attack categories have small sample sizes , so per-type detection rates carry wide confidence intervals while aggregate code-level and system-level rates are more reliable; we do not run a direct head-to-head comparison against existing scanners such as skill-security-analyzer because that tool publishes only an 11-sample test suite, leaving construction of a shared cross-detector benchmark as follow-up; and the cross-model LLM comparison in §4.4 is preliminary, with Claude (150 samples) and Gemini (266 samples) run on partially overlapping sets and head-to-head numbers computed only on the 7 overlapping code-level types. Additional attack vectors such as time bombs and sandbox escapes are not evaluated as separate categories, though the framework includes detection patterns for YAML deserialization and unsafe serialization.

Agent skill ecosystems. The SkillMD-138K dataset (skillmd138k, ) provides the largest collection of real-world agent skills (138K entries) for ecosystem-level analysis.

Skill security. The skill-security-analyzer¹¹1https://github.com/anthropics/skills/pull/83 provides a regex-based scanning tool with 40+ pattern categories, achieving 100% detection on 11 test samples. We provide a multi-stage scoring framework with weighted composition and baseline calibration rather than binary pattern matching, and systematically evaluate on 266 adversarial samples across 16 attack types to map detection boundaries rather than maximizing detection on a small test suite.

Prompt injection. Prompt injection attacks on LLM agents have been studied extensively (greshake2023, ; perez2023, ), but injection via skill packages represents a distinct vector where malicious instructions are embedded in procedural knowledge files rather than direct user input.

Software supply chain security. Our attack taxonomy draws from the broader supply chain security literature (ohm2020, ), adapting established categories (typosquatting, dependency confusion) to the agent skill context while introducing skill-specific threats (prompt injection via SKILL.md, capability mismatch).

We present SkillsMetric, a five-stage static analysis framework for agent skill security, and construct the first large-scale adversarial evaluation dataset spanning 16 attack types. Our key finding is not what static analysis can detect (AUC 0.93 on code-level attacks), but what it cannot: host destruction (0%) and prompt injection (42%) represent fundamental blind spots that require complementary detection methods. A population-level scan of 138K real-world skills shows a largely healthy ecosystem (82.4% SAFE) with a 1.75% flag rate. These results establish the empirical foundation for defense-in-depth architectures in the rapidly growing agent skill ecosystem.