An intelligence report circulated across the US military this spring, in the midst of the war with Iran, immediately set off alarm bells: A Chinese ship in the Middle East was transporting components of a nuclear weapons program.
The US military swung into action with plans to intercept the vessel, according to four sources familiar with the episode. Two of the sources said armed US military personnel were preparing to board the ship. Military aircraft were already in the air, according to one of those sources and another person familiar with the incident.
It was only shortly before the planned operation that US officials took a closer look at the report, which had been prepared by a Special Operations Command analyst, and discovered that it had been generated with the help of artificial intelligence. They also found that a chatbot used by the analyst had incorrectly identified the material the ship was carrying. CNN was unable to determine what cargo had been misidentified.
The report was "entirely false," according to one source. But it also "almost started a war," the source said. Any US operation against a Chinese vessel could have risked spiraling into an armed conflict between the two countries.
Across the US military and intelligence community, officials are working to integrate AI into nearly every aspect of their operations, from analyzing the vast quantities of raw intelligence collected by the United States and selecting targets for strikes to more routine tasks such as managing budgets, logistics and supply chains.
The episode, however, highlights the profound risks of using a powerful new technology that remains relatively poorly understood for targeting decisions in the middle of a war. Analysts have long warned that using AI on the basis of flawed or corrupted data could result in a catastrophic miscalculation, such as the United States opening fire on a Chinese ship because of inaccurate information.
In this case, the analyst queried a chatbot about intelligence concerning the ship's manifest that originated with US Special Operations Command Pacific, which is based in Hawaii. It was unclear whether the chatbot was a commercially available product or a US government system. "The internal tools are mostly just copies of the commercial stuff wearing lipstick," said a former senior US official familiar with the AI systems used by military and intelligence analysts.
The chatbot combined open-source intelligence with classified signals intelligence held in government databases and reached the incorrect conclusion about the material the ship was carrying. The analyst then used AI again to package the findings into a standard intelligence report, the kind of document military officials are accustomed to trusting, and disseminated it.
US Special Operations Command Pacific and the Pentagon did not respond to CNN's request for comment.
The rationale behind the rapid adoption of AI is its ability to help the military make decisions on the battlefield faster, including which targets to strike and where to deploy military assets. US officials say the country cannot afford to fall behind in integrating AI out of concern that it could one day find itself fighting China or another adversary capable of staying one step ahead.
In January, US Defense Secretary Pete Hegseth released the department's "Artificial Intelligence Acceleration Strategy" in an effort to speed up the military's adoption of the technology. "We will unleash experimentation, eliminate bureaucratic barriers, focus our investments and demonstrate the execution approach needed to ensure we lead in military AI," Hegseth said in a speech announcing the strategy.
The strategy also calls for broad AI use throughout the military, directing the Defense Department to make the technology available through several programs with the goal of "democratizing AI experimentation and transformation across the Department by putting America's world-leading AI models directly in the hands of our three million civilian and military personnel, at all classification levels."
Several US officials familiar with the issue, however, said the effort was decentralized, with different parts of the government using different tools under varying directives and safety standards. There is no single set of standards governing how the United States verifies information generated by those systems.
The use of different AI systems by separate parts of the military and intelligence community means their reliability and capabilities vary widely. For weeks, policymakers in Washington have been engaged in an intense debate over AI following a series of stark warnings from Silicon Valley engineers and technology executives about the possibility that artificial intelligence could escape human control, potentially with consequences severe enough to bring about the end of civilization.
The episode involving the Chinese ship, however, highlights another and more immediate risk: Human beings could make disastrous decisions based on inaccurate or misleading information generated by AI or other automated systems.
Sources said the US military is rapidly turning to AI to assist with targeting, an area that carries an obvious risk of deadly mistakes. "AI in targeting is definitely something that is ramping up, and there is no real guidance for how having a human in the loop will prevent civilian casualties or fratricide," said another source familiar with the military's current policies.
According to one source, the kind of "hallucination" produced by the tool used by the analyst in this case has not been an isolated occurrence in the intelligence community since such systems began proliferating across the government.
For some veteran intelligence officials, including those who broadly support the military's use of AI, the technology has placed pressure on analysts to produce and disseminate intelligence more quickly, creating more opportunities for mistakes. Several sources said younger analysts, who have grown up using such tools, are particularly likely to trust them without sufficient scrutiny. "AI allows you to get to a bad idea faster," one source said.
