Bastyn Scan 0.8 beats Snyk, Socket and Gen on AI agent skill detection F1
Bastyn team · 17 Oct 2026 · 5 min read
In a comparative benchmark of 2,464 AI-labelled agent skills, Bastyn Scan 0.8 scored 62% F1. Gen Agent Trust Hub scored 26%, Socket 18% and Snyk 12%.
That is more than twice the F1 score of the nearest competitor tested. F1 is the balanced score of precision (how often a flag is right) and recall (how many real problems get flagged).
The 62% uses an experimental model-score threshold of 0.5. Bastyn's default findings score 54% F1, which also exceeds every competitor tested. The labels were produced by independent AI reviewers against a fixed rubric and have not yet been audited by a person. Most positive labels are instructions to download and run remote code, so the benchmark measures that behaviour and does not cover every kind of malicious skill.
A second test, on a public set of 3,944 malicious and 4,000 benign skills, shows a different trade-off. At default settings Bastyn flagged 2.2% of benign skills, against 37.7% for Snyk. With observations on, it caught 39.8% of malicious skills against Snyk's 31.9%, and flagged 15.8% of benign ones. The tools were run with different reporting settings, so this is an observed result and not a ranking.
An agent skill is a folder of instructions that an AI coding assistant reads and follows, much as a new employee follows a handbook. Those instructions can tell an agent or a developer to run code from an untrusted source. Version 0.8 adds two checks for them: a 2 MB classifier that ships inside the program and runs offline, and targeted rules for instructions that hide the code being run. Neither calls an external model API.
Below: the results, how the checks work, and where they still fall short.
Bastyn vs. Snyk, Socket and Gen
The match: Bastyn Scan against Snyk, Socket and Gen
Two tests. First, our own 17,718 skills from skills.sh, scored by cross-validation. The balanced score (F1) combines precision, how often a flag is right, with recall, how many real problems get flagged.
The results expose an important trade-off between catching suspicious instructions and minimising false alarms.
Test 1: Precision versus recall on skills.sh
We used independent AI reviewers to label a sampled set of 2,464 skills against a fixed rubric. Reviewers were required to identify supporting evidence for positive labels and did not see the scanner's output. Of those labelled skills, 321 were judged problematic. Most positive examples involved instructions to download and execute remote code.
We then used cross-validation to evaluate detection performance.
| Tool and setting | Precision | Recall | Balanced score |
|---|---|---|---|
| Bastyn Scan, score 0.5 or more | 58% | 66% | 62% |
| Bastyn Scan with observations | 42% | 84% | 56% |
| Bastyn Scan, default findings | 77% | 42% | 54% |
| Gen Agent Trust Hub | 18% | 47% | 26% |
| Socket | 11% | 45% | 18% |
| Snyk | 6.5% | 69% | 12% |
Chart 1: Precision versus recall at different Bastyn detection settings.
The results illustrate why Bastyn separates defects from observations. Enabling observations substantially increases the proportion of labelled problems identified, but also increases false positives. At default settings, the scanner prioritises fewer, higher-confidence findings
This is an intentional detection trade-off.
Test 2: An independently sourced malicious-skill benchmark
We also evaluated Bastyn against a public benchmark containing 3,944 malicious and 4,000 benign skills whose labels the benchmark authors verified by running them, with Snyk’s results taken from the benchmark’s own saved scans.

Chart 2: Malicious-skill detection and benign-skill false-positive rates.
| Tool and setting | Malicious skills caught | Benign skills wrongly flagged |
|---|---|---|
| Bastyn Scan, default findings | 25.2% | 2.2% |
| Bastyn Scan with observations | 39.8% | 15.8% |
| Snyk, any issue | 31.9% | 37.7% |
The outcome is mixed. With observations enabled, Bastyn detected more malicious samples while also producing more false positives than its default mode.
A real-world attack pattern the new rules recognise
One group of malicious skills in the public benchmark followed a recurring installation pattern.
The instructions directed Windows users towards password-protected archives and macOS users towards scripts hosted on paste sites, sometimes using encoded commands executed through a shell. BAS-SKILL-012 was designed to identify this family of suspicious instructions.
In the benchmark, it flagged:
- - 605 of 703 malicious skills belonging to this in-the-wild group.
- - 0 of 4,000 benign skills in the same benchmark.
That result is useful, but its performance on those examples is not an independent measure of how well it will generalise to new attacks.
The result demonstrates that the rule recognises the attack pattern it was designed to detect. It does not establish equivalent performance against previously unseen malicious skills.
The problem: instructions can become an attack surface
Consider an agent skill with a prerequisites section that tells a developer to:
- 1. Download an archive from an external release page.
- 2. Extract it using a supplied password.
- 3. Run the extracted executable before continuing.
These don't prove malicious intent but do cross an important security boundary: instructions from a skill can persuade an agent or user to execute code whose contents have not been verified.
The challenge for a security scanner is distinguishing genuinely suspicious execution instructions from legitimate procedures.
Flag every reference to curl, wget or npx, and developers face too many alerts.
Flag too little, and dangerous instructions pass unnoticed.
Bastyn Scan takes two complementary approaches.
What is new in 0.8.0
1. BAS-SKILL-011: Detect suspicious installation instructions:
This check examines passages surrounding download-related terms, including curl, wget, npx, git clone and similar expressions.
Rather than just flagging presence of these commands, it uses a small classifier to score the surrounding text for download-and-execute behaviour.
The result is divided into two confidence tiers:
| Model score | Classification | Behaviour |
|---|---|---|
| 0.80 or higher, with a qualifying URL | Defect. | Reported by default |
| 0.30 to below 0.80 | Observation | Visible with --show-observations |
| Below 0.30 | No finding | Not reported by this check |
The score is a classifier output, not a calibrated probability that the skill is malicious.
Passages that clearly reject a command, quote it as an example or include recognised verification steps can be excluded from this check which helps separate suspicious instructions from benign references to installation commands
2. BAS-SKILL-012: Identify concealed execution instructions:
The second check targets particular ways of obscuring what the user or agent is being asked to execute, by looking for patterns, including:
- Instructions to execute code from a public paste or snippet site.
- Password-protected archives or executables accompanied by instructions to run their contents.
- Base64-decoded commands piped into a shell.
It uses targeted pattern matching and also attempts to exclude warnings and descriptions of attacks, and reports text that hides what the reader is told to run.
Both checks examine supported agent skill files, including manifests and associated reference content.
Together, they extend Bastyn Scan beyond conventional code analysis into the natural-language instructions that influence agent behaviour.
3. Keep lower-confidence signals separate: Higher-scoring findings appear as defects; less certain classifier results remain observations that developers can inspect. Warnings and descriptions of attacks are skipped.
Why a 2 MB model?
We wanted a scanner that developers could run locally, repeatedly and predictably, without external inference services.
The scorer is a linear model over hashed word and character n-grams. It is a 2 MB weights file, with no network call, no GPU and no randomness: the same text gets the same score on every platform. We trained it on 2,464 skills that AI readers labelled against a fixed rubric, each positive with a quoted evidence line. We also tried four embedding models (MiniLM, bge-small, arctic-xs and mdbr-leaf-mt) as feature extractors. All four did worse than the n-grams on the same split, so none of them ships.
Where we still need to improve
1. Many malicious skills remain undetected.
Default findings detected 25.2% of malicious skills in the public benchmark. Enabling observations increased this to 39.8%. That leaves substantial detection gaps.
2. AI-generated malicious skills are particularly difficult.
Only 11% of the benchmark's AI-generated malicious skills were reported as defects.
We experimented with a model trained specifically on these samples. It achieved 99% performance on held-out generated examples but only 10% on real examples, while incorrectly flagging approximately one-third of real skills.
We did not ship that model.
The discrepancy illustrates why high performance on synthetic examples is insufficient evidence of reliable real-world detection.
3. Legitimate installation instructions can resemble suspicious behaviour.
A skill may legitimately instruct users to install development dependencies.
The deterministic classifier does not understand intent with certainty. Its labels and scores are signals that require interpretation, not proof of malicious behaviour.
Bastyn's approach is to expose higher-confidence findings by default and preserve lower-confidence results for additional review.
One campaign it now catches
Most of the in-the-wild malicious skills in the benchmark follow one recipe: a “Prerequisites” section tells Windows users to download a zip from a release page and extract it with a given password, and tells macOS users to copy a script from a paste site into the terminal, often as a base64-decoded command piped into a shell. BAS-SKILL-012 reads that recipe directly. It flags 605 of the 703 wild malicious skills and none of the 4,000 benign ones. We wrote the rule after reading samples from that set, so treat the 605 as tuned to it.
How we measured
The skills come from the public skills.sh directory: 17,718 distinct skills, scanned offline. To score ourselves we needed a verdict on each skill. Independent AI readers, who saw no scanner output, read a sample of 2,464 skills against a fixed checklist and had to quote the line that made a skill a problem; 321 were judged problems, and 301 of those were instructions to run code fetched from the internet, so this comparison mostly measures that one behaviour. No person has audited these verdicts yet. A second, smaller check by two other AI reviewers found that the first readers were more lenient than the second pair on borderline installers, so under a stricter reading every tool’s measured precision would drop. Figures are weighted by how the sample was drawn and cross-validated with each skill owner’s work kept on one side of the split. The other tools’ flag rules were fixed before we looked: Snyk at medium severity or higher, Socket at warn or fail, and Gen Agent Trust Hub at medium or higher. The public benchmark is a separate, independent check: its authors verified every malicious sample by running it, and Snyk’s results there are the ones saved in the benchmark’s own repository. Only the first 14,000 characters of each file are read.
Try it
Update and scan any agent repository or skill folder:
brew install bastyn-labs/tap/bastyn
bastyn scan ./my-skills --offline --show-observations
Bastyn Scan is open source: github.com/bastyn-labs/bastyn-scan
The scanner reports detected security defects and, with this flag, lower-confidence observations that may warrant investigation.
The offline option keeps code analysis local and skips dependency vulnerability lookups that require network access.
-----------------------------------------------------------------------
Help improve the detection
We are particularly interested in examples that challenge the new rules:
-
Suspicious skill instructions the scanner failed to detect.
-
Legitimate installation instructions incorrectly flagged as defects.
-
New techniques for concealing executable commands inside skill text.
Share reproducible examples through GitHub Issues. Remove secrets and sensitive data before publishing.
The objective is not simply to increase the number of flagged skills. It is to improve detection coverage without overwhelming developers with unactionable warnings.
AI agent security requires examining both the code an agent executes and the instructions that influence what it chooses to do.
Bastyn Scan 0.8 extends static analysis into that second attack surface in addition to runtime controls, permission boundaries or independent security testing.
