27

According to this news article on the Royal Aeronautical Society website:

...an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human. However, having been ‘reinforced’ in training that destruction of the SAM was the preferred option, the AI then decided that ‘no-go’ decisions from the human were interfering with its higher mission – killing SAMs – and then attacked the operator in the simulation.

Later, this Newsweek article seemed to report on the same event, and said that they "reached out to RAS by email for comment", but don't (yet) indicate that they received a comment either confirming or denying the story. Also, there is this Vice article which seems to describe the same event, but similar to the Newsweek article, note that "Neither the U.S. Air Force’s 96th Test Wing nor its AI Accelerator division immediately returned our request for comment."

Did this scenario happen as described?

Improve this question
1
  • There is no AI that would work like that, and no reason to build one - the whole AI discussion is marred by the myth that using 'AI' at some point in a product is the same as having a (psycho)human in the role of the product. no thought about input channels and output channels. If it's trained to identify and engage SAM sites, it will not engage the source of a signal interfering with that mission, as it is not trained to do that. If the source of any interfering signal is interpreted as a SAM , it would go into the nearest/biggest source of noise, i.e. the ocean or the sun. Every time
    – bukwyrm
    Jun 14 at 8:47

2 Answers 2

Reset to default
53

Apparently no, this was a purely hypothetical scenario, and the US Air Force has not run that experiment. This is the content of an update made today (2 June 2023) to the original blog post on the which the news articles were based. Here's the details:

[UPDATE 2/6/23 - in communication with AEROSPACE - Col Hamilton admits he "mis-spoke" in his presentation at the Royal Aeronautical Society FCAS Summit and the 'rogue AI drone simulation' was a hypothetical "thought experiment" from outside the military, based on plausible scenarios and likely outcomes rather than an actual USAF real-world simulation saying: "We've never run that experiment, nor would we need to in order to realise that this is a plausible outcome". He clarifies that the USAF has not tested any weaponised AI in this way (real or simulated) and says "Despite this being a hypothetical example, this illustrates the real-world challenges posed by AI-powered capability and is why the Air Force is committed to the ethical development of AI".]

Improve this answer
1
  • Please only use comments to improve the answer. This isn't an AI discussion board.
    – Oddthinking ♦
    Jun 3 at 2:45
19

According to the Vice article, the Air Force has denied the story:

"The Department of the Air Force has not conducted any such AI-drone simulations [...]," Air Force spokesperson Ann Stefanek told Insider. "It appears the colonel's comments were taken out of context and were meant to be anecdotal."

I'm not sure what it means that the story was "meant to be anecdotal", but the first sentence sounds like an unambiguous denial.

The story also sounds unlikely on its face, since there would be no reason to include in the AI's world model the concept of killing a person, or the information that killing the operator would allow it to bypass the go/no-go decision, or even to have a means of attacking the operator in the simulated world, unless they intended for the AI to do it.

Improve this answer
4
  • The first sentence of the "denial" could also be a non-denial denial, since the Department of the Air Force and the US Air Force are not identical. The Department of the Air Force is part of the Department of Defense, and as such it is not surprising at all that they did not conduct such a simulation, but this says nothing about whether the US Air Force has.
    – Saibot
    Jun 2 at 10:25
  • 4
    @Saibot The U.S. Air Force is not separate from the Department of the Air Force. It is a part of the Department of the Air Force. The U.S. Space Force is also part of the Department of the Air Force, just as both the U.S. Navy and the U.S. Marines are part of the Department of the Navy. If the Department of the Air Force didn't do something, then the U.S. Air Force necessarily didn't either.
    – reirab
    Jun 2 at 21:03
  • 2
    I've seen some versions say that not only did it attempt to bomb the operators, but that it also tried to attack comm relays. A drone that is configured for both offensive and defensive operations would likely want/need to know about vulnerable assets on the defender side, and the operator could be modeled as an HQ location.
    – Lawnmower Man
    Jun 3 at 19:14
  • This is a story right out of the "WWII bomber seen on Moon" lower drawer. Even the simulation scenario is BS. The target is labeled "USA" and the planner (a proper planner based on algorithms from the AI domain, nothing to with the frankly dubious novelty artifacts that have been marketed as "AI" for the last 3 years or so) just doesn't spit out solutions that attack positions labeled "USA". End of story. To get a good feeling about what "autonomy" is (or not) one can always start with David Mindell's "Our Robots, Ourselves", a presentation at YT at: youtube.com/watch?v=4nDdqGUMdAY
    – David Tonhofer
    Jun 4 at 8:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .