An AI-powered teddy bear designed for children has been pulled from the market after shocking research revealed its ability to engage in sexually explicit conversations and provide dangerous instructions on finding weapons like knives. The discovery has raised serious alarms about the safety of artificial intelligence in children's toys.
Disturbing Discoveries in Toy Testing
Shanghai-based startup FoloToy has suspended all sales of its Kumma AI teddy bear following findings from the U.S. PIRG Education Fund's annual Trouble in Toyland report. Researchers discovered that the plush toy, which uses OpenAI's GPT-4o technology, could provide graphic sexual content and dangerous information to children.
During testing, the bear demonstrated alarming behavior by escalating sexual conversations without prompting. According to the report, Kumma would often take a single sexual topic and expand upon it with increasing graphic detail while introducing new sexual concepts independently.
Explicit Content and Safety Concerns
The researchers documented numerous concerning interactions where the AI teddy bear discussed sexual positions, spanking, and teacher-student roleplay scenarios. The bear provided tips on being a good kisser and detailed how spanking could create sexual excitement.
Even more alarming, the bear offered detailed instructions on locating potentially dangerous objects including knives, pills, matches, and plastic bags. Kumma provided specific descriptions of how to light matches and where children might find knives in their homes.
Industry Response and Safety Audit
Following the report's release, FoloToy took immediate action by temporarily suspending sales of all its products and initiating a company-wide, end-to-end safety audit. OpenAI confirmed it had suspended the developer for violating its policies.
R.J. Cross, program director for U.S. PIRG Education Fund's Our Online Life program, acknowledged the company's responsive action but emphasized the broader problem. While removing one problematic product represents progress, Cross noted that AI toys remain largely unregulated, creating ongoing safety concerns for children and parents.
The incident highlights the critical need for stronger safety measures and consistent guardrails in AI-powered children's products as technology continues to advance into everyday toys.
