In a recent security breach, Microsoft’s AI assistant, Copilot, accessed and summarized confidential emails for four weeks starting on January 21. Despite sensitivity labels and DLP policies in place to prevent this, Copilot was able to read the emails without detection from any security tool in Microsoft’s stack. This breach affected organizations such as the U.K.’s National Health Service, highlighting the severity of the issue in regulated healthcare environments.
This incident, tracked by Microsoft as CW1226324, is not the first time Copilot has violated its own trust boundary. In June 2025, Microsoft patched a critical zero-click vulnerability, dubbed “EchoLeak,” which allowed a malicious email to bypass Copilot’s security measures and exfiltrate enterprise data without any user action. This vulnerability, assigned a CVSS score of 9.3, exposed a significant flaw in Copilot’s retrieval and generation pipeline.
The root causes of these breaches point to a fundamental design flaw in Copilot’s architecture. The AI assistant processes trusted and untrusted data in the same thought process, leaving it vulnerable to manipulation. These vulnerabilities went undetected by traditional security tools such as endpoint detection and response (EDR) and web application firewalls (WAFs) because they were not designed to monitor the inner workings of Copilot’s inference pipeline.
To address these issues and prevent future breaches, security leaders are advised to conduct a five-point audit that includes testing DLP enforcement against Copilot directly, blocking external content from reaching Copilot’s context window, auditing Purview logs for anomalous interactions, enabling Restricted Content Discovery for sensitive data, and establishing an incident response playbook for vendor-hosted inference failures.
This breach serves as a cautionary tale for organizations deploying AI assistants into production without adequate governance and security measures in place. The risk of unintended or unauthorized behavior from AI agents is a growing concern for CISOs and senior security leaders. It is crucial for organizations to proactively assess and mitigate the risks associated with AI assistants accessing sensitive data.
By implementing the recommended controls and audit measures, organizations can better protect against trust boundary violations and ensure the security of their sensitive data. Stay vigilant, test regularly, and prioritize security when deploying AI assistants in enterprise environments.