Since the integration of technology-based learning in the 2018-2019 school year, nearly 85 thousand students have been subjected to surveillance by content moderation technology. Monitoring activity including homework, projects, essays, emails, artwork, chats, and even social media activity, students’ digital trails are monitored “24/7/365,” by Gaggle, a student surveillance service employed by LCPS and 1,400 other schools nationwide.  

Gaggle markets itself as a one-stop solution to student safety in the digital world that supports “digital citizenship” and “creates positive school environments.” Promising “real-time content analysis,” the Bloomington, Ill. based company claims on their website that in the 2018-2019 school year, it helped “save 722 students from carrying out an act of suicide.”

Gaggle operates on a combination of machine-learning technology and human content moderation to alert school districts of potential threats of violence, substance use, suicide and self-harm, harassment, sexual content/pornography, and profanity. Gaggle scans communications against a “Blocked Words List,” which screens for predetermined words and phrases. Additionally, the service reviews content with an in-house, AI “Anti-Pornography Scanner.” 

Available to three major educational services, Gaggle acts as a software plug-in for the Google G Suite for Education, Office 365, and Canvas. In LCPS, the software monitors the Google G Suite and Office 365.

Gaggle files User Violations on a “three strikes” system and warrants different responses by school administrators depending on the severity of the violation. According to invoices, incident reports, and company documents aggregated from the county by the Freedom of Information Act and documents obtained previously by BuzzFeed News in a nationwide study of Gaggle, the response by Gaggle’s safety management team depends on whether the infraction is marked as a “User Violation,” “Questionable Content,” or “Possible Student Situation” by the company’s Safety Management Rubric.

After AI technology marks any form of violation, human moderators assess the urgency of the infraction to determine the appropriate response. According to Gaggle’s Safety Management Procedures, Safety Managers review over “a million blocked student communications” each month. In the event that the algorithm flags a false positive that includes inappropriate images/language for the purpose of schoolwork, human moderators unflag tagged work on a case-by-case basis.

Content marked by the “Violations” category could constitute profanity/vulgar language or suggestive/provocative content. Gaggle’s standard response for “Violations,”  the lowest tier of offense by the rubric, constitutes an email to the student, with administration CC’d to the email following multiple offenses. However, according to documents provided to The Blaze, LCPS does not notify students in the event of a standard violation. Hugh McArthur, the Supervisor for Information Security for LCPS, said that Gaggle “doesn’t often use that function or recommend it anymore” and “it wasn’t [LCPS’s] goal when implementing the product” as its purpose was “student safety.”