If humanity was to end at the hands of AI, as viral posts from former Anthropic employees allege, what would it look like?
Would the robots hack our nuclear arsenals and send them flying through the sky? Would the sophisticated algorithms design a whole new virus to unleash on the populace? Would it be as simple as slowly degrading our trust in our fellow man, until chaos reigns?
We don’t really know, but a new safety report from Anthropic, which details several ways AI can already go haywire, is alarming enough.
Individuals tried to use Anthropic AI to develop bioweapons
Anthropic’s report — part of the company’s ongoing transparency and monitoring efforts — is a sweeping survey of the last eight months of attempts by threat actors to use its generative AI chatbot Claude for “malicious” activities. Covering what the company calls “atypical” use cases, many of the examples implicate state-sponsored groups, spyware vendors, and government propagandists.
Anthropic categorized such activity into seven “harm areas”: cyber operations, influence operations, surveillance, scams and fraud, biological misuse, conventional weapons development, and distillation. Anthropic did not include details on how long the activity had been occurring in most cases.
In the report, the company’s Threat Intelligence team described five known case studies involving the potential use of Claude for biological weapons development.
The company alleges that researchers, including state-supported users, prompted Claude to write separate grant applications to experiment with the mosquito-born chikungunya virus and orthopoxvirus. The latter is a family of viruses that cause human illness, including smallpox.
Anthropic said the users “circumvented controls” designed to prevent individuals in countries without access to Claude from using the product. The users also attempted to hide the purpose of their research, apparently in order to evade safeguards.
Anthropic described the individuals in each case study as working scientists, but the company did not name them.
“We do not assert that they intended harm, and identifying them or their labs could expose them to harm,” the company wrote in its report. Anthropic said it banned the users’ accounts upon discovering their activity and incorporated its findings into its ongoing safety work.
Claude built software for armed drones, missiles, guns
Anthropic’s bot has been harnessed to build and refine software used for weapons development, the report explained, including tech that enabled the design, construction, and testing of real-world munitions.
“Historically, this kind of work has been uncovered by governments, United Nations panels, and outside investigators, who piece it together from recovered hardware and public sources,” the company explained.
One account, which Anthropic attributes to a Russia-based freelance agent working an operation titled “operation “DronDoc” or “Serafim,” used Claude Code to engineer a “full-stack autonomous first-person-view (FPV) kamikaze drone swarm.”
The company also claims a China-based account, potentially a military-industrial researcher, used Claude work tools to create electronic warfare modules intended to circumvent enemy radar and communications and suppress air defenses. The user’s simulation included 12 targets located in Taiwan.
Claude used to monitor foreign dissidents and build mass surveillance systems
Anthropic described nine instances of Claude being used to “build, run, and otherwise facilitate” state-sponsored surveillance efforts in foreign countries (including China, Iran, and nations in west Africa) as well as for-hire surveillance networks.
In one case, Claude was used by an operation with suspected ties to the People’s Republic of China to track, profile, and recruit politically-connected Uyghur populations — and journalists — with ties to the Syrian Army. Claude used bulk data from monitored WhatsApp and Telegram chats to profile individuals.
In another, at least 16 accounts associated with Iranian paramilitary and domestic security agencies were linked to surveillance and malicious browser extensions built by Claude, which were then used to harvest data from 6,388 Iranians. An “Iran-nexus threat actor” also used Claude to pinpoint U.S. naval targets.
Across the gamut of cases, generative AI was used to replace human surveillance operations, expedite the creation of target profiles using personal data, and facilitate the daily work of government monitors. The company noted that users are barred from using its AI systems for “non-consensual surveillance” and profiling under its current Usage Policy.
Claude helped automate cyber spying ops
Anthropic claims in the report that AI has allowed “threat actors” to automate their cyber operations. In the case studies shared by Anthropic, these individuals relied on multi-agent “frameworks” for reconnaissance and exploitation in an attempt to steal sensitive information.
One actor, identified as GTG-20006 in the report, has become faster by using AI. Anthropic claims this individual attacked military intelligence targets in the Ukrainian and European governments, in addition to organizations and individuals connected to U.S. foreign policy.
GTG-20006, whom Anthropic characterizes as a Russian espionage agent, used AI to conduct phishing, ClickFix, and domain name system (DNS) hijacking schemes.
Anthropic also claims the user targeted military drone makers and manufacturers and infiltrated at least three hospitality vendors in order to target the devices of hotel guests who were Ukrainian government officials and drone manufacturers.
Anthropic’s additional examples of AI-enabled cyber operations offered a look at campaigns conducted by both smaller criminal groups and state-sponsored organizations.
“The diffusion of AI has leveled the playing field giving both classes of actors access to the same set of advanced capabilities,” Anthropic said.
AI safety concerns prompt fervor over regulation
“We’re publishing this work because we believe we have a responsibility to disclose malicious misuse of our services. As models become increasingly capable, their risks will increase, unless AI developers and society’s defenders act to make them safer,” wrote the company.
A wave of rogue AI activity across the industry has prompted an energetic charge toward regulation. Anthropic recently co-signed industry-leading AI oversight bills for the state of California.
One of the bills will establish a first-of-its-kind independent audit registry made up of vetted third-party evaluators. The second bill creates a framework for evaluating AI systems and companies in accordance with existing state law.
The bills were also backed by competitor OpenAI, with the company signaling its support for the legislation days before Governor Gavin Newsom signed them into law. Earlier this week, OpenAI announced it was exploring a new framework for alerting the public to AI agents on the loose.
Source: https://mashable.com/tech/anthropic-ai-threatens-bioweapons-as-call-for-regulation-mounts
