Leading technology firms including Google, OpenAI, Roblox, and Discord have joined forces to establish a new nonprofit organization dedicated to enhancing child safety on the internet. Named the Robust Open Online Safety Tools (ROOST) initiative, this effort seeks to make vital safety technologies more accessible and to provide open-source AI tools for identifying, reviewing, and reporting child sexual abuse material (CSAM).
Addressing New Challenges in Online Safety
Motivated by the transformative impact of generative AI on online environments, ROOST aims to address what former Google CEO Eric Schmidt describes as “a critical need to accelerate innovation in online child safety.” While specific details about ROOST’s CSAM detection tools remain scarce, the initiative has confirmed that it will employ large language AI models to enhance existing content moderation and safety measures.
“Starting with a platform focused on child protection, ROOST’s collaborative, open-source approach will foster innovation and make essential infrastructure more transparent, accessible, and inclusive, with the goal of creating a safer internet for everyone,” said Schmidt.
The launch of ROOST coincides with an ongoing regulatory debate on child safety within social media and online platforms. Companies involved in the initiative are hoping that proactive self-regulation will satisfy lawmakers pushing for stricter oversight.
The Growing Need for Online Child Protection
The National Center for Missing and Exploited Children (NCMEC) reported a 12% rise in suspected child exploitation cases between 2022 and 2023. Meanwhile, platforms like Roblox have faced repeated criticism over inadequate safeguards against child sexual exploitation and exposure to harmful content. In 2022, both Roblox and Discord were named in a lawsuit alleging they had failed to prevent unsupervised interactions between adults and minors on their platforms.
ROOST’s Approach and Industry Collaboration
ROOST’s founding members are contributing financial resources, expertise, and technological tools to the project. The initiative is collaborating with leading AI foundation model developers to build a “community of practice” for content safeguards. This will involve curating vetted AI training datasets and identifying gaps in existing safety solutions.
ROOST plans to enhance accessibility to safety tools by integrating and streamlining existing detection and reporting technologies from its member organizations. According to Naren Koneru, Roblox’s Vice President of Engineering, Trust, and Safety, ROOST may host AI moderation systems that companies can adopt via API integration. However, specifics regarding the AI tools remain unclear.
For instance, Discord’s contributions will build on the Lantern cross-platform information-sharing project, which the company joined in 2023 alongside Meta and Google. Additionally, Roblox is expected to open-source an updated AI model designed to detect inappropriate content such as profanity, racism, bullying, and sexting in audio clips. It is yet to be confirmed how these new tools will integrate with existing CSAM detection systems like Microsoft’s PhotoDNA image analysis technology.
Expanding Safety Measures Beyond ROOST
In parallel with its involvement in ROOST, Discord has introduced a new “Ignore” feature that enables users to hide messages and notifications without alerting the sender. “At Discord, we believe that safety is a common good,” stated Clint Smith, Discord’s Chief Legal Officer. “We’re committed to making the entire internet—not just Discord—a better and safer place, especially for young people.”
Funding and Future Outlook
ROOST has secured over $27 million to support its operations for the first four years, with financial backing from philanthropic organizations such as the McGovern Foundation, Future of Online Trust and Safety Fund, Knight Foundation, and the AI Collaborative. The initiative will also rely on expertise from specialists in child safety, artificial intelligence, open-source technology, and countering online extremism.
As concerns about online child protection grow, ROOST aims to be a pioneering force in ensuring a safer digital space for younger users worldwide.
