Level 1 — Absolute Beginner
Many people use AI at work. They write a question, and the AI gives an answer.
Sometimes people put a secret in the question, like a password. Is the secret safe?
A man called HD Moore does a test. He sends two million questions with fake passwords to AI companies.
Then he waits. Thieves use some of the fake passwords. So he knows that three companies have a problem.
- question
- something you ask
- answer
- what you say or write after a question
- secret
- something that other people must not know
- password
- a secret word that opens a computer or an account
- safe
- not in danger
- test
- a way to check if something works
- fake
- not real
- thieves
- people who steal things
Level 2 — Elementary
When you send a prompt to an AI service, your words often travel through several companies before they reach the AI model. A new study shows that this journey is not always safe.
HD Moore, a well-known security researcher at the company Dragos, wanted to find out who was reading the prompts. Between July and November, his team sent about two million prompts to AI providers and resellers. Every prompt contained a fake secret, such as a false key for a cloud account.
Each secret was unique and was sent only once to one company. If somebody later used the secret, the team knew exactly where it had leaked.
The results were worrying. Three hosting providers had been attacked by criminals who were copying prompts, and none of the three knew about it until the researchers told them. Some resellers were also reading and saving their customers' prompts.
- prompt
- the text that you send to an AI
- service
- a system that does work for customers
- researcher
- a person who studies something carefully
- providers
- companies that supply a service
- resellers
- companies that buy a service and sell it to others
- unique
- different from all the others
- leaked
- got out when it should have stayed secret
- criminals
- people who break the law
Level 3 — Intermediate
Companies are pouring confidential material into AI tools, from source code to customer records, usually without knowing how many hands a prompt passes through. A study published on 19 November by HD Moore, chief research officer at the security firm Dragos, suggests that the chain is leakier than most users assume.
Moore's team borrowed an old idea from mining: the canary that warns of danger. From July to November they sent roughly two million prompts to AI providers, routers and resellers, each containing a unique piece of bait, such as a hidden web link or a fake key for Amazon's cloud. Because every secret went to exactly one company, any later use of it pointed straight at the source of the leak.
The bait was taken. The team found that at least three hosting providers had been compromised since mid-July by attackers who were quietly collecting prompts, and none of them knew until Dragos told them. In the largest case, a couple of hundred fake keys were stolen in late October, some were tested within hours, and in total the stolen keys were used for tens of thousands of requests.
Not every leak was a crime. Some resellers on open marketplaces were simply logging what customers sent, and settings that promise "zero data retention" were not reliably respected. Moore advises firms to deal directly with a major provider, to avoid cheap resellers and to plant canaries of their own.
- confidential
- meant to be kept secret
- canary
- a small bird once used in mines to warn of bad air; here, a hidden warning signal
- bait
- something used to attract and catch someone
- compromised
- entered or controlled by an attacker
- logging
- keeping a written record of
- retention
- the act of keeping something
- reliably
- in a way that you can trust every time
- plant
- to put something secretly in a place
Level 4 — Advanced
The artificial intelligence boom has created a sprawling and largely invisible supply chain. A prompt typed into a coding assistant may pass through a router, a reseller and a hosting company before it ever reaches a model, and each intermediary is a potential point of exposure. An experiment disclosed on 19 November by HD Moore, the veteran researcher who now leads research at Dragos, offers the most systematic evidence so far of how often that exposure becomes theft.
The method was elegantly simple. Over roughly five months the team seeded about two million prompts with canary tokens, decoy credentials and concealed links that are worthless except as tripwires, and dispatched each one to a single destination. Any subsequent attempt to use a token therefore incriminated a specific link in the chain.
The tripwires duly went off. At least three hosting providers turned out to have been infiltrated since mid-July by intruders siphoning prompts, a fact that had escaped all three until they were notified. The most serious episode involved a couple of hundred Amazon keys harvested during brief windows in late October; some were probed within hours, and the decoys were eventually used in tens of thousands of calls. Dragos has anonymised most of the companies concerned and says it is now working with about a dozen providers.
Equally troubling was the leakage that required no break-in at all. On open marketplaces, a seller needs only to list a model cheaply and switch on request logging to read everything customers submit, and privacy labels such as zero data retention proved an unreliable guarantee. The practical lesson, Moore argues, is to shorten the chain: host sensitive workloads yourself, contract directly with a first-tier provider, and assume that anything pasted into a prompt may one day be read by a stranger.
- sprawling
- spreading over a wide area in an untidy way
- intermediary
- a person or company in the middle between two others
- systematic
- done according to a careful plan
- decoy
- a fake thing used to trick someone
- tripwires
- hidden wires or signals that set off an alarm when touched
- incriminated
- made someone appear guilty
- siphoning
- taking something away quietly over time
- harvested
- collected in large amounts