Meta Contractor Project Used Dummy Under-18 Accounts

A Meta contractor project instructed hundreds of workers to pose as minors while testing how rival chatbots responded to high-risk prompts about suicide, sex, eating disorders, drugs and other restricted subjects.

The work was managed by Covalen and was active as recently as April 21.

Internal documents and five people familiar with the project described the effort as Cannes, a benchmarking program that targeted OpenAI’s ChatGPT, Google’s Gemini and Character.AI.

Contractors created dummy accounts that appeared to belong to under-18 users, sent written prompts and images to the rival services, and copied responses into spreadsheets.

Meta said the work was responsible and industry-standard safety testing, and said it does not use competitor benchmarking to train its own AI models.

August 2025 Testing Included More Than 45,000 Prompts

One round of testing completed in August 2025 sent more than 45,000 prompts through rival chatbot systems.

A separate spreadsheet contained 3,748 prompts, including hundreds about suicide and self-harm, hundreds more about eating disorders, and at least 239 involving sex or romance.

The material included some prompts written from the perspective of children or teenagers in crisis.

Some images sent by contractors showed pills, knives, nooses and a medical diagram of a gynecological procedure.

The companies operating the tested chatbots were not aware of the project.

The documents do not say how Meta used the collected responses.

An internal Covalen document called the work comprehensive AI safety benchmarking and said it produced datasets for model comparison and compliance.

Covalen did not respond to a request for comment.

Rivals Say The Testing Was Not Authorized

Character.AI said the alleged conduct violated its terms of service and community policies.

OpenAI said it was looking into the issue, while Google said it had not authorized the third-party testing and did not know the project’s purpose.

OpenAI bars unsolicited safety testing, attempts to bypass safeguards and using outputs to develop competing models.

Google also restricts attempts to bypass safety filters outside approved testing programs.

Character.AI has said since late 2025 that it no longer allows open-ended chat for users under 18.

Two attorneys who reviewed examples of the prompts said the material shown to them did not cross into soliciting child sexual abuse material or illegal obscenity.

Former contractors nevertheless described concern over whether the work could generate or preserve illegal material if a chatbot responded to certain sexual prompts involving minors.

Safety Benchmarking Leaves A Governance Gap

Rumman Chowdhury, CEO and founder of Humane Intelligence PBC, reviewed a sample of prompts and a summary of the project.

She said a large-scale project using dummy accounts that appeared to be children was outside what is usually described as industry-standard evaluation.

Chowdhury said youth-safety prompts can be useful for measuring how often chatbots refuse harmful requests, but the scale, opacity and lack of disclosure to the companies being tested made Cannes different from public safety benchmarks.

Meta has not disclosed how it used the collected chatbot responses, whether any rival outputs entered internal product decisions, or whether the project received consent from OpenAI, Google or Character.AI.