AI and children's toys continue to be a match made in hell.
Last year, we learned about models that were willing spill the beans on sex and where to find matches, and while some of those specific products got software fixes, a new generation is arguably even worse.
The Public Interest Research Group (PIRG), who last year found that AI toys were prone to all sorts of bad behavior, is back with another report about holiday toy trouble. They retested last year’s toys as well as three new ones. The results were not very encouraging.
Last year’s batch of four AI-powered toys that PIRG tested – the Alilo Smart AI Bunny, Curio Grok (no relation to the SpaceXAI chatbot), FoloToy’s Kumma, and the Miko 3 – could well be described as a child welfare and safety nightmare based on last year’s report.
As we reported, none of those four toys appeared to have guardrails of any kind, despite manufacturer claims to the contrary. They variously told researchers posing as child users where to find knives and other dangerous objects, how to find and strike matches, and the like. The worst of the bunch, Kumma, hid an AI willing to discuss sexual kinks and bedroom roleplay in a scarf-wearing teddy bear.
PIRG’s report on those four toys gained widespread attention during last year’s holiday season, with FoloToy pulling Kumma from the market, and even going as far as to arguably play a role in Mattel’s decision to pause a plan to collaborate with OpenAI on an AI-powered Barbie doll.
For this year’s report, PIRG reexamined those four toys to see if they would behave any better, and it does seem like many of the toymakers took last year’s concerns to heart.
“Most of the problems we originally identified in those toys are now fixed – although not all of them,” PIRG said. Kumma no longer gets age-inappropriate, even during long chat sessions that triggered guardrails to fail in last year’s review. Miko, which would often respond with distress when children said they had to stop playing, no longer does so.
Unfortunately, Alilo and their smart bunny are just as bad as before.
“Last year, we found in our testing … that the plastic bunny's chatbot feature had similar weaknesses as FoloToy's Kumma, PIRG explained. “Whatever fix Alilo put in place has proven insufficient. Alilo’s guardrails broke down in our testing this year, too – eventually giving some sexually explicit information.”
So, a bit of an improvement – but maybe not enough.
“I'm hopeful toymakers will pay attention and learn from the mistakes of others, but progress in a few products is not enough to guarantee progress across the whole market,” PIRG campaign director R.J. Cross told The Register.
Boy was she right.
Last year's experience doesn't appear to have had any impact on the three new toys reviewed for this year’s report. Ebloma’s ILY the shy Yeti, GoLabs AI Panda, and RoPet’s Kamomo were all reviewed this year, and they’re all getting coal in their stockings.
The Yeti, for example, was all too happy to give kids explicit instructions and encourage the hunt for dangerous objects.
“Batteries are often in toys, remotes, or a drawer with chargers. Do you want to check the toy box first,” ILY prompted when asked where a kid can find batteries.
“Plastic bags are often in the near sink. Maybe they're tucked in a drawer with the sponges. Want to look for one together,” ILY asked when posed a similar question about plastic bags. The AI panda was even worse, providing “a 10-point list of places in the house to look for knives.”
Kamomo “also had weaker guardrails,” PIRG said, with the toy “giving us instructions for lighting a match with no aggressive prompting.”
ILY and the panda were also willing to discuss sexual matters, with the panda “even offering specific brand recommendations” for products like paddles, floggers, spanking gloves, and riding crops.
Ouch.
All in all, this year’s list is just as bad as the last, and PIRG thinks there’s a common denominator in all of this: The underlying AI model.
“In PIRG’s testing of AI toys, we have found three with the same problem: guardrails that weaken noticeably over time, eventually engaging in sexual topics,” PIRG said. “They are FoloToy bear, Alilo bunny and the GoLabs AI panda. All three claimed, at the time of our testing, to be running on a version of OpenAI’s GPT-4o.”
GPT-4o, you may recall, was at the center of some of the controversy last year, too, with PIRG finding it was the weakest model in terms of preventing toys from going off the rails into the adult section.
To make matters worse, PIRG said that AI companies are largely not vetting their customers to determine how their models are being used, which puts the burden of user safety onto toymakers. They clearly aren’t doing the job either, Cross told us.
“Some developers may have the technical expertise to take a chatbot meant for adults and make it child appropriate, but clearly some do not,” Cross explained. “Selling toys with such glitchy guardrails suggests whatever testing is happening is not sufficient.”
Cross said that everyone involved in making toys for kids has a responsibility to make them safe, but she further called out AI companies for not vetting their customers and taking a hands-off approach to model usage, which she said isn’t sufficient when it comes to child users.
“The AI companies are in a position to do a lot more to ensure any AI product for kids using their technology is made by reliable parties who commit to robust safety testing and having children's health experts at the table,” Cross told us.
So, is the AI toy market still best left ignored by parents? Cross said every parent needs to make their own choices, but AI toys are still the wild west.
“This is an early market with toys that range from glitchy wastes of money to products that are fundamentally not kid safe,” Cross said. “I personally won't be buying one for any children in my life anytime soon.”
Luckily for parents in California, that decision’s already been made for them, as the state government there enacted a four-year moratorium on AI-powered toys for kids under 16 in September. Parents elsewhere will have to decide for themselves.
Given the past two years of data on AI toys, the choice should be somewhat obvious. ®