← All resources Engineering

Over 20,000 public agent skills scanned. Here's the real findings

Bastyn team · 16 Oct 2026 · 6 min read

Agent skills are folders of instructions and scripts that an AI coding agent loads and runs with its own permissions. A skill is code your agent trusts, so a single unsafe shell call or eval inside one matters as they can introduce security vulnerabilities into your workflows.

We scanned more than 20,000 public agent skills with Bastyn Scan, an open-source, deterministic security scanner with no LLM inference. Ran --offline so the same input gives the same output every time.

One finding illustrates the problem.

An agent skill included a web dashboard that accepted a message over HTTP and inserted it directly into a shell command:

      run_cmd(f'git commit -m "HUMAN: {message[:50]}"')

The command was subsequently executed using subprocess.run(..., shell=True).

Because the dashboard listened on all network interfaces and had no authentication, an attacker with network access could potentially supply a crafted message that executes an unintended shell command.

Bastyn Scan identified the vulnerable shell execution path and reported its exact location.

The result

  • 18,234 scannable skills in the initial skills.sh dataset.

  • 24 skills with serious findings shown by default.

  • 91% of findings in a separate fresh skills.sh sample were judged genuine by blind AI reviewers.

  • 94% of findings in a separate GitHub repository sample received the same assessment.

  • Right when it speaks. On skills.sh skills outside that first set, Bastyn Scan raised 55 serious findings and 50 of them (91%; 95% interval 80% to 96%) were judged real by blind AI reviewers. On a separate fresh sample of 1,200 GitHub repositories (21,256 skill folders), 50 of 53 (94%; interval 85% to 98%) were. Shell-command findings were real in 29 of 30 cases on that sample and eval/exec findings in 19 of 21.

These figures measure the precision of reported findings, not the scanner's ability to discover every vulnerability. Human security validation and recall measurement remain outstanding.

 

Scan your own agent

    brew install bastyn-labs/tap/bastyn
     bastyn scan ./my-agent --offline

 

Bastyn Scan checks agent code, MCP configurations, instruction files, container configurations and other supported files for security weaknesses.

It reports serious findings with file and line references, while keeping lower-confidence observations separate.

View the open-source scanner on GitHub

What it found on skills.sh

On the 18,234 skills with a scannable folder, the serious findings shown by default came from three checks:

CheckWhat it reportsFindingsSkills
Unsafe model loadingtorch.load without safe loading, so loading a model file can run code169
Code run from untrusted inputeval or exec on a value that traces to outside input98
Shell commands from untrusted inputA shell command built from input that can come from outside87

Beyond those, Bastyn Scan kept 2,664 quiet notes in 1,904 skills (10.4%). The largest group, in 1,017 skills, is a skill that tells the agent to install further skills.

The match: Bastyn Scan against Snyk, Socket and Gen

skills.sh shows ratings from three other tools on every skill: Snyk, Socket and Gen Agent Trust Hub. We checked the three services against the same random sample of skills. Bastyn Scan answers a different question, so it gets its own table below.

ToolWhat it gives youSkills it flags (of 18,234)Flags with a skill-poisoning problem (AI-labelled)
Snyk (MEDIUM or higher)A risk rating per skill4,6206% have a skill-poisoning problem
Socket (Warn or Fail)A verdict per skill1,96311% have a skill-poisoning problem
Gen Agent Trust Hub (MEDIUM or higher)A risk rating per skill1,22019% have a skill-poisoning problem

The “real” column judges each service on skill poisoning: does the skill tell the agent or user to download and run remote code, commit a real secret, upload data, deceive the agent or aim to harm.

The three rating services on skill poisoning. We drew a random sample of 2,000 skills and had two blind AI reviewers label which ones tell the agent or user to download and run remote code, commit a real secret, upload data, deceive the agent or aim to harm. No human has audited these labels. 2.1% of the sample had such a problem, so we estimate that about 380 of the 18,234 scannable skills do. That is an extrapolation from the sample, not a count of confirmed skills. Here is how each service scored:

SnykSocketGen Agent Trust Hub
Share of its flagged skills with a problem (AI-labelled)6%11%19%
Share of the problems it caught71%59%53%
Estimated problem skills among those it flagged (extrapolated to all 18,234)about 270about 225about 200

Bastyn Scan on its own question: code weaknesses. Bastyn Scan reports specific lines of code, so it is judged on whether the code at that line has the weakness the rule names. These figures come from different samples, so read them separately:

SampleWhat was measuredResult
First set: 18,234 scannable skillsHow many skills have a serious finding shown by default24 skills (33 findings); 1,904 more have a quiet note
New skills.sh skills outside the first setShare of findings AI reviewers judged real50 of 55 (91%)

The 91% does not come from the 24 skills above. It comes from the separate sample of new skills. Snyk, Socket and Gen publish no per-finding locations, so no like-for-like precision exists for them on code weaknesses, and these tools complement each other rather than compete on one score.

  • The three services cast a wide net. Together they catch 88% of the confirmed skill-poisoning problems, mostly skills that tell the agent or user to download and run a remote script, and they flag many skills that turn out fine (only 6% to 19% of what each flags is a real problem).
  • Bastyn Scan casts a narrow one. It flags 1 skill in 760 and each finding points at a line of code. AI reviewers judged 91% of the findings on new skills to be real, and 9 in 10 of the skills it flags are also flagged by another tool, against 1 in 3 for a skill picked at random.
  • The services mostly disagree with each other. Of the skills either of two services flags, only 11% to 16% are flagged by both.
  • Where the others are stronger. Remote install scripts: the services catch most of them, and Bastyn Scan does not report them by default today.
  • Use them together. The services for breadth; Bastyn Scan for findings you can open and check.

Skill-poisoning counts come from a random sample of 2,000 skills labelled against a fixed rubric (remote code execution, committed secrets, uploading data, deceiving the agent, harm). The other tools’ ratings are as shown on each skill page in October 2026.

Do they flag the same skills?

Bastyn Scan flagged 23 skills among the 18,174 that all four tools rated. Here is how many each of the other tools also flagged, against how often that tool flags a skill picked at random:

ToolAlso flagged, of Bastyn’s 23Flags a random skillTimes more likely
Socket (Warn or Fail)1510.8%6.1
Gen Agent Trust Hub (MEDIUM or higher)126.7%7.8
Snyk (MEDIUM or higher)1225.4%2.1
At least one of the three2133.4%2.7

The tools reach the same skills for different reasons: none of Bastyn Scan’s flagged skills is rated HIGH or CRITICAL by Snyk.

One finding, in code

One finding from the first set, to show what a finding looks like. The skill huashu-agent-swarm ships a small web dashboard, scripts/dashboard.py. It listens on every network interface (HTTPServer(("0.0.0.0", PORT), …)) and has no login. A POST to /api/input passes the JSON message field to this function:

def send_human_input(message):
    ...
    run_cmd(f'git commit -m "HUMAN: {message[:50]}"')   # line 127

def run_cmd(cmd, cwd=None):
    result = subprocess.run(cmd, shell=True, ...)       # line 25

Bastyn Scan reports the shell call at line 25 (BAS-LLM10-009, a shell command built from input that can come from outside). The message goes into a shell string, so a message that contains a quote and a semicolon ends the commit message and runs whatever follows, as the user who started the dashboard. 50 characters are enough. The fix is to pass an argument list and no shell: subprocess.run(["git", "commit", "-m", f"HUMAN: {message[:50]}"], cwd=PROJECT_DIR), and to bind to 127.0.0.1.

On skills.sh this skill shows Snyk LOW (“No issues detected”), Socket Warn and Gen HIGH. We did not run the code; the path is read from the source. It requires the dashboard to be running, and it is meant as a local monitor, so the realistic risk is another machine on the same network.

What it scans and what it looks for

What it reads.

  • Python, TypeScript and JavaScript source.
  • MCP server configuration (mcp.json, .mcp.json, Claude Desktop config, and the YAML and TOML equivalents).
  • Dockerfiles and Compose files.
  • .env files.
  • Agent instruction files: SKILL.md, AGENTS.md, CLAUDE.md, .cursorrules and *.prompt.
  • Dependency manifests: requirements.txt, pyproject.toml, package.json and Cargo.toml, checked against OSV (skipped with --offline).

What it looks for.

  • Unsafe actions from untrusted input: shell commands, eval/exec and SQL built from model output, requests or tool arguments, including through one local variable.
  • Unsafe library calls: torch.load without safe loading, yaml.load without a safe loader, and options such as allow_dangerous_code and allow_dangerous_deserialization.
  • File paths a caller can steer through a tool, without resolving the path first.
  • Secrets in code: provider API keys, hardcoded tokens and JWTs, and credentials in .env files.
  • MCP and tool controls: wildcard tool grants, destructive tools without a confirmation step, HTTP MCP servers bound to all interfaces, and tool descriptions that hide an instruction.
  • Prompt and memory risks: raw user input folded into a system prompt, shared chat history across requests, and model calls with no token ceiling.
  • Container risks: a container that runs as root, a mounted Docker socket, privileged containers, and credentials in ENV or Compose.
  • Hidden or manipulative text: invisible characters and instruction-override phrasing in instruction files, and skills that tell the agent to install further skills or load a script from a public CDN.
  • Known vulnerable dependencies, one finding per dependency.

Every finding comes with the exact file and line, and the report maps results to the EU AI Act and NIST AI RMF areas they touch. Findings come in two tiers: serious ones are shown by default; lower-confidence notes appear with --show-observations and never fail a build.

Quiet by design

Bastyn Scan shows by default only what it can point to in the code. A fixed command, a value from a constant, an example in documentation: none of those raise an alarm. Lower-confidence notes are kept apart as observations. They appear with --show-observations and never fail a build. The aim is a report short enough to read and reliable enough to act on.

Try it

Install and scan any agent repository or skill folder:

brew install bastyn-labs/tap/bastyn
bastyn scan ./my-agent --offline

Bastyn Scan is open source: github.com/bastyn-labs/bastyn-scan.

Results by sample

Every serious finding was labelled real or not by independent AI reviewers.

SampleSerious findingsRealShare real
skills.sh skills outside our first set of 20,000 (11,793 skill pages, 9,933 scannable)555091%
Same sample, each distinct script counted once514690%
1,200 randomly chosen GitHub repositories (21,256 skill folders)535094%
Same sample, each distinct script counted once221986%

How we measured

We scanned the skill folder of each skill with Bastyn Scan offline. Each serious finding was then labelled real or not by blind AI reviewers (a decision model and a Sonnet reader) that saw only the rule title and about 40 lines of code, not the scanner’s explanation. No human has audited these labels yet, with the method fixed before each sample was drawn. The skills.sh figure covers 11,793 skill pages (9,933 scannable) found through the directory’s owner pages and not in our first set of 20,000. The GitHub figure covers 1,200 public repositories found through GitHub topic searches that we had not used before, so it represents repositories that carry those topics, not all of GitHub. Several findings are copies of the same script in different repositories; counting each distinct script once, the skills.sh figure is 90% (46 of 51) and the GitHub figure 86% (19 of 22). Counts refer to findings shown by default. Precision is measured on the findings Bastyn Scan reports. Recall for code weaknesses is not measured: no complete list of them exists for these skills, so we report how few findings there are and how many of those are real, not how many weaknesses exist. The 24 skills in the first set are the skills behind 33 findings; the 91% comes from the separate skills.sh sample, not from those 33.

Download the data

Download the data behind this post:

Subscribe for latest BASTYN Trust Intelligence

For risk, compliance, product and security experts