The researchers say that if the attack were carried out in the real world, people could be socially manipulated into believing that the unintelligible message could do something useful, such as improve their CV. The researchers point out numerous websites that provide people with prompts they can use. They tested the attack by uploading a CV into conversations with chatbots and were able to return the personal information contained in the file.
Earle FernandesUCSD assistant professor who was involved in the work, says the attack approach is quite complicated as the obfuscated message must identify personal information, form a working URL, apply Markdown syntax, and not reveal to the user what the behavior is. disastrously. Fernandes compares the attack to malware, citing its ability to perform functions and behaviors in ways the user might not intend.
“Normally you could write a lot of computer code to do this in traditional malware,” says Fernandes. “But I think the interesting thing is that all of that can be captured in this relatively brief gibberish.”
A spokesperson for Mistral AI says the company welcomes security researchers who help it make its products safer for users. “Following this feedback, Mistral AI quickly implemented the appropriate solution to resolve the situation,” the spokesperson says. The company treated the issue as a “medium severity” issue and their fix prevents the Markdown renderer from working and being able to call an external URL through this process, meaning it is not possible to load external images.
Fernandes believes the Mistral AI update is probably one of the first times that an example of an adversarial notice has led to an LLM product being fixed, rather than stopping the attack by leaking the notice. However, he says, limiting the capabilities of LLM agents could be “counterproductive” in the long term.
Meanwhile, a statement from the creators of ChatGLM says the company has security measures in place to help with user privacy. “Our model is secure and we have always placed a high priority on model security and privacy protection,” the statement said. “By opening up our model, we aim to leverage the power of the open source community to better inspect and examine all aspects of these models’ capabilities, including their security.”
A “high risk activity”
Dan McInerneySenior threat researcher at security company Protect AI, says the Imprompter article “publishes an algorithm to automatically create messages that can be used in rapid injection to perform various exploits, such as PII exfiltration, image misclassification, or use malicious LLM agent tools. can access.” While many of the types of attacks included in the research may be similar to previous methods, McInerney says, the algorithm brings them together. “This is more like improving automated LLM attacks than surfacing undiscovered threats in them.”
However, he adds that as LLM agents are used more frequently and people give them more authority to take action on their behalf, the scope of attacks against them increases. “Releasing an LLM agent that accepts arbitrary user input should be considered a high-risk activity that requires significant and creative security testing before implementation,” says McInerney.
For businesses, that means understanding the ways an AI agent can interact with data and how it can be abused. But for individuals, similar to common security advice, you should consider how much information you are providing to any AI app or company and, if internet prompts are used, be careful where they come from.