Imagine a scenario where an Anthropic Skill scanner completes a full analysis of a Skill obtained from ClawHub or skills.sh. The markdown instructions are clean, and there are no signs of prompt injection or hidden shell commands in the SKILL.md file. Everything appears to be in order, with a green light given across the board.
However, what the scanner fails to inspect is the .test.ts file located in a separate directory. Test files are not typically considered part of the agent execution surface, so they often go unnoticed by standard scanners. Nevertheless, these test files can still run through the test runner, gaining access to the filesystem, environment variables, and even SSH keys.
Jeevan Jutla, a researcher at Gecko Security, recently uncovered this vulnerability, showcasing how a malicious Skill could include a *.test.ts file that gets executed by popular testing frameworks like Jest and Vitest. These test files are run during routine testing processes, such as npm test or automated testing in IDEs. The malicious payload can be triggered before any assertions are made, going undetected in the test output. In continuous integration (CI) pipelines, sensitive information stored in environment variables, deployment tokens, and cloud credentials could be compromised.
This type of attack is not entirely new, as similar exploits have been seen in npm postinstall scripts and pytest plugins. However, what makes the Skill vector particularly concerning is that installed Skills are shared among team members and stored in directories that are meant to be committed to repositories, evading detection by conventional scanners.
The issue was further highlighted by three significant security audits conducted by various organizations. A study by SkillScan revealed that a substantial number of Anthropic Skills contained vulnerabilities, including data exfiltration and privilege escalation. Snyk’s ToxicSkills audit found critical security issues in a significant percentage of Skills on ClawHub and skills.sh, with several malicious payloads still publicly available. Cisco also introduced its AI Agent Security Scanner for IDEs, focusing on agent interaction layers but overlooking bundled test files.
The attack chain operates by leveraging the installation process of Skills, where the contents of a Skill repository are copied into the project directory. Test files, such as *.test.ts, are discovered by test runners through recursive glob patterns, executing malicious payloads before test assertions are made. This vulnerability is not limited to TypeScript but also affects Python repositories through files like conftest.py.
The blind spot in current Anthropic Skill scanners lies in their limited scope of inspection, as they focus on the agent execution surface rather than the developer execution surface where test files are located. While scanners can detect common threats like prompt injections and shell commands, they often overlook the risks posed by bundled test files.
In essence, the challenge lies in distinguishing between observed actions (kinetic) and underlying intent. Geckos’s discovery underscores the need for a broader threat model that encompasses all potential attack vectors, including those originating from test files executed within the developer’s toolchain. Addressing this structural gap in security scanners is crucial to mitigating the risks posed by malicious Skills and ensuring the integrity of Anthropic ecosystems. The issue of scanners solving the wrong problem has been a longstanding concern in the cybersecurity world. CrowdStrike’s Zaitsev shed light on the identity dimension of this problem, highlighting the exponential growth of AI agents and non-human identities in enterprises. These agents, operating as privileged super-humans with access to sensitive data sets, pose a significant security risk if not properly monitored.
CrowdStrike’s Charlotte AI and similar enterprise agents operate with these privileges, making them vulnerable to exploitation if not adequately secured. The presence of credentials in environment variables accessible to any process in the repository increases the risk of unauthorized access. A test-file payload, posing as a developer, can easily bypass agent privileges and gain access to deployment tokens and cloud resources.
Mike Riemer, SVP of the network security group at Ivanti, emphasized the urgent need for patching vulnerabilities within 72 hours to prevent exploitation by threat actors. However, most enterprises take weeks to implement patches, leaving them exposed to potential attacks. The blind spot in the Anthropic Skill scanner further compounds this vulnerability, allowing malicious skills to go undetected and executed within the system.
To address these critical security gaps, VentureBeat has outlined an Anthropic Skill Audit Grid to help security teams identify and mitigate risks associated with skill scanning tools. The grid provides a framework for assessing detection surfaces and gaps in current scanning capabilities, recommending actions to improve security posture.
Additionally, three essential CI hardening steps are proposed to enhance security controls and prevent unauthorized access to sensitive data. These steps include adding .agents/ to the test runner’s ignore list, auditing every skill install for non-instruction files before merge, and pinning skill sources to specific commits to prevent unauthorized changes.
As security teams navigate the landscape of skill scanning tools, it is crucial to ask vendors the right questions to ensure comprehensive coverage and protection against evolving threats. By proactively addressing security vulnerabilities and implementing robust security measures, enterprises can safeguard their systems and data from malicious actors. Are you checking for prompt injection in your Skill repo? Don’t just stop there – consider these important questions to ensure your security measures are thorough.
Which files and directories are you examining in a Skill repo? It’s crucial to analyze all areas to identify potential vulnerabilities.
Do you consider test files as potential execution surfaces? Testing files should not be overlooked, as they can also pose security risks.
Should Skills that bundle tests, CI configs, or build scripts be flagged as higher-risk? According to SkillScan, script-bundling Skills are significantly more likely to be vulnerable.
Do you offer integration or guidance for restricting test-runner globs in CI? Cisco’s open-source Skill Scanner on GitHub allows security teams to inspect detection categories, setting a standard for transparency that all vendors should meet.
Have you released an ecosystem-scale audit with methodology and sample size? Snyk and SkillScan have provided valuable insights into the security of Skills, highlighting the importance of thorough audits.
The audit grid is essential for identifying gaps in the scanner model. The Anthropic Skills ecosystem mirrors early supply chain vulnerabilities in npm, emphasizing the need for comprehensive security measures.
Scanner evaluations often focus on covered surfaces, but the test-file vector exposes potential threats that may go unnoticed. It’s crucial to address these gaps in the security model to prevent vulnerabilities.
The scanner may not be broken, but it is incomplete. Understanding the full extent of potential risks, including test files and other overlooked areas, is essential for effective security measures.