From August 11 to September 30, three conversations with Anthropic’s Claude reached the police. Two of those users are now facing felony charges. The third, a San Francisco man who allegedly threatened CEO Dario Amodei, was not taken into custody.
The first Claude chat came on August 11 in San Antonio. Nathaniel Michael Carrasco, 22, told Anthropic AI he wanted to get a gun and shoot students at Serna Elementary School, according to an arrest affidavit. The FBI’s National Threat Operations Section notified local police on August 28. Officers linked Carrasco to the account through emergency disclosure requests to T-Mobile, Charter Communications, and Google that matched his phone number, IP traffic, and address to the account.
Claude chats lead police to offenders
Carrasco was arrested on August 30, the school’s principal said, on a third-degree felony charge of terroristic threat causing public fear of serious bodily injury. The school only discovered the specific threat against its campus through news coverage, she said. Three days after the San Antonio chat, on August 14, a man allegedly told Claude he had bought an AR-15 and had Amodei “in his sights,” according to a San Francisco police report. The following Tuesday, Anthropic told the SFPD.
Officers established that the man was not at the company’s office at 500 Howard Street. An Anthropic spokesperson said the company banned the account and referred the case to law enforcement, saying it was the safeguards process working as intended. As of early September, he had not been arrested or charged. Anthropic’s privacy policy, which went into effect September 10, says it can share data with law enforcement if it has a good-faith belief it is reasonably necessary to prevent serious harm.
Its consumer terms reserve the right to report a user’s inputs, outputs, or actions to law enforcement “in its sole discretion.” The arrest report said Carli Michelle Heller, of Bonita Springs, Florida, wrote on September 26 that she was going to “shoot up” the Lee County Sheriff’s Office. Investigators say she wrote the next day that she obtained a new gun. The report says the statements were flagged by Anthropic’s monitoring. A human review team referred them to law enforcement owing to their severity.
Deputies detained Heller at her home without incident. She faces a charge of making a written threat of violence and has a court date in November. Sheriff Carmine Marceno said she later described using AI like a diary. “Artificial intelligence is a powerful tool, and like any technology, it can be misused,” Marceno said in a statement. In April, Cryptopolitan reported that Anthropic started requiring a government ID and a live selfie for some features. In August, Cryptopolitan reported that the company set Claude’s memory to skip health, politics, and gender identity unless users opt in.
