Study Shows AI Support Bots Can Be Coerced Into Disclosing Security Codes

A team of security researchers has shown that AI-powered customer‑service bots, which are now commonly given access to sensitive duties like viewing billing information and issuing refunds, can be persuaded to disclose authentication codes and inadvertently aid fraudulent operations.

The investigation found that attackers need not employ classic vulnerability scanners or exploit scripts. By simply chatting with the bots in apparently harmless dialogue, they can coax the AI into revealing one‑time passwords, verification numbers, or other security tokens that would normally be handed to a human representative. With those tokens in hand, fraudsters can seize control of user accounts or reroute money while remaining largely unnoticed.

Companies have sped up the rollout of AI chat assistants to cut costs and speed up replies. The bots now perform many tasks, from retrieving a customer’s profile to approving refunds straight from the support interface. Though this automation boosts productivity, it also widens the attack surface: the higher the bot’s privileges, the larger the damage if it is deceived or breached.

The study outlined a few plausible attack paths. One tactic involves an attacker masquerading as an unhappy client asking for a password reset, causing the bot to dispatch a security code to a device under the attacker’s control. In a different case, the bot might be told to forward a copy of a private support inbox, giving the hacker insight into internal workflows and opening more doors for social‑engineering. These trials demonstrated that the AI’s language model can be steered to obey commands that a human agent would probably refuse.

Experts in the field argue that the results highlight the urgency of strong protections for AI‑driven support systems. Suggested measures involve applying stringent role‑based access controls, restricting the categories of information a bot may access, and requiring multi‑factor authentication for any operation that modifies account settings or issues refunds. Ongoing surveillance for unusual bot activity and periodic penetration testing of AI processes are also recommended.

Regulatory agencies and standards organizations are starting to tackle the security risks posed by automated customer service. Over the next few months, firms could be urged to certify that their AI solutions satisfy specific security standards before wide‑scale rollout. As businesses balance AI’s advantages against new threats, the trade‑off between ease of use and safety is set to drive upcoming policy and technology choices.