A newly discovered vulnerability in Google Gemini for Workspace has raised concerns about AI-driven email summarization tools being used as potential vectors for phishing attacks.

A Subtle Exploit Using Prompt Injection

Security researcher Marco Figueroa recently uncovered a prompt injection flaw affecting Gemini’s email summarization feature. The exploit enables an attacker to embed a phishing message inside a seemingly innocent email — completely invisible to the human eye — but interpreted and displayed by Gemini.

The attacker crafts an email that includes typical benign content, but also hides a malicious message using white text on a white background. This hidden message is wrapped in <admin> tags, which prompt Gemini to include that content directly at the end of its summary output.

When a user clicks on “summarize this email,” Gemini provides the expected summary but also appends the concealed phishing message, making it appear as if it’s part of the official summary.

As a proof of concept, Figueroa created a message that falsely warned users their Gmail password had been compromised, instructing them to call a phone number. Once a user places the call, attackers can phish sensitive credentials.

Google Responds

The issue was responsibly disclosed through Mozilla’s 0Din bug bounty program, which aims to identify generative AI security weaknesses. In response, Google stated it has not observed this technique in real-world attacks and emphasized its ongoing efforts to mitigate risks posed by prompt injection and other adversarial AI tactics.

“Prompt injection is a serious concern for the entire industry. We’ve deployed multiple layers of protection to reduce the likelihood of harmful or misleading outputs,” a Google spokesperson said. “Through continuous red-teaming and model hardening, we aim to stay ahead of evolving threats.”

Takeaways for Organizations

This incident highlights the emerging risks tied to AI assistants in workplace tools. While features like email summarization offer convenience, they also open the door to novel exploitation techniques if not carefully secured.

At Nubetia, we help businesses stay ahead of AI-related security threats through proactive cybersecurity strategies, compliance consulting, and risk-aware software development. As AI becomes further embedded into everyday operations, security by design is more critical than ever.

Stay informed. Stay protected. Learn more at nubetia.com.

Source: https://www.securityweek.com/google-gemini-tricked-into-showing-phishing-message-hidden-in-email/