Click With Caution

Since the integration of technology-based learning in the 2018-2019 school year, nearly 85 thousand students have been subjected to surveillance by content moderation technology. Monitoring activity including homework, projects, essays, emails, artwork, chats, and even social media activity, students’ digital trails are monitored “24/7/365,” by Gaggle, a student surveillance service employed by LCPS and 1,400 other schools nationwide.  

Gaggle markets itself as a one-stop solution to student safety in the digital world that supports “digital citizenship” and “creates positive school environments.” Promising “real-time content analysis,” the Bloomington, Ill. based company claims on their website that in the 2018-2019 school year, it helped “save 722 students from carrying out an act of suicide.”

Gaggle operates on a combination of machine-learning technology and human content moderation to alert school districts of potential threats of violence, substance use, suicide and self-harm, harassment, sexual content/pornography, and profanity. Gaggle scans communications against a “Blocked Words List,” which screens for predetermined words and phrases. Additionally, the service reviews content with an in-house, AI “Anti-Pornography Scanner.” 

Available to three major educational services, Gaggle acts as a software plug-in for the Google G Suite for Education, Office 365, and Canvas. In LCPS, the software monitors the Google G Suite and Office 365.

Gaggle files User Violations on a “three strikes” system and warrants different responses by school administrators depending on the severity of the violation. According to invoices, incident reports, and company documents aggregated from the county by the Freedom of Information Act and documents obtained previously by BuzzFeed News in a nationwide study of Gaggle, the response by Gaggle’s safety management team depends on whether the infraction is marked as a “User Violation,” “Questionable Content,” or “Possible Student Situation” by the company’s Safety Management Rubric.

After AI technology marks any form of violation, human moderators assess the urgency of the infraction to determine the appropriate response. According to Gaggle’s Safety Management Procedures, Safety Managers review over “a million blocked student communications” each month. In the event that the algorithm flags a false positive that includes inappropriate images/language for the purpose of schoolwork, human moderators unflag tagged work on a case-by-case basis.

Content marked by the “Violations” category could constitute profanity/vulgar language or suggestive/provocative content. Gaggle’s standard response for “Violations,”  the lowest tier of offense by the rubric, constitutes an email to the student, with administration CC’d to the email following multiple offenses. However, according to documents provided to The Blaze, LCPS does not notify students in the event of a standard violation. Hugh McArthur, the Supervisor for Information Security for LCPS, said that Gaggle “doesn’t often use that function or recommend it anymore” and “it wasn’t [LCPS’s] goal when implementing the product” as its purpose was “student safety.” 

According to their Internet Safety Program outlined in 2008, LCPS follows guidelines regarding student safety outlined in the Children’s Internet Protection Act enacted by Congress in  2000 in order to combat minors’ access to “obscene or harmful content.” Per the Federal Communications Commission, in order for districts to receive discounted rates on Internet access via “E-rates,” districts must monitor minors’ online activity and teach common-sense internet safety, alongside blocking inappropriate content. 

Gaggle presents itself as a tantalizing option for districts that seek to minimize students’ exposure to a vast range of potentially harmful materials. However, the high-tech surveillance technology services provided by the for-profit company come at a steep cost. According to invoices for the 2018-2019 school year, LCPS paid upwards of 303 thousand dollars for coverage on the G Suite and Office 365 platforms. For the 2019-2020 school year, the bill amounted to nearly 375 thousand dollars, an increase of nearly 71 thousand dollars due to an increase in school-issued devices and a higher enrollment. 

According to previous years’ invoices, LCPS has maintained contracts with Gaggle beginning in 2013 with only Office 365, and later expanding to include the Google G Suite after its implementation in 2018.

In the digital age, the market for one-stop security solutions has become increasingly lucrative, as companies such as Gaggle, Securly, and Bark, among others, contend for schools’ business promising CIPA compliance as well as above-and-beyond monitoring services. However, students are being forced to acknowledge and, in some cases, actively maneuver around the loss of privacy that comes with constant digital surveillance.

“It’s definitely always in the back of my head now when I look something up,” Taman said, “It’s not worth getting called in.”

Though the LCPS Student Technology Acceptable/Responsible Use Policy does disclose that there is “no right or expectation of privacy” on school-issued devices, Taman believes there should be greater transparency about monitoring systems that includes providing students with general guidelines as to what falls under flagged content. 

According to principal John Duellman, though the county does implement technology lessons that cover topics including “social media presence” and “how to effectively use devices,” and requires that students sign a “memo of understanding” upon receiving their devices, Duellman agrees that more training could be effective.  

“There’s probably some additional training that could go out there [to make students aware of Gaggle], but in that same respect, if you asked any student, ‘Have you heard of a Gaggle alert?’ I think most would say yes,” Duellman said. 

“But, in terms of Gaggle, they’re doing the same internal debate I’m thinking about. I want to tell you things, but I don’t want to tell you too much,” Duellman said, “Because then, the people who are bad people are going to continue to find ways around what we do.” 

“I think the primary thing is that it’s not a matter of surveillance, it’s a matter of trying to keep folks safe,” McArthur said. “We’re not looking for the bad kids, we’re looking for the sad kids,” McArthur said, quoting a Gaggle official.

Duellman believes that the act of documenting mental health on school networks is often a “cry for help,” though he doesn’t think that Gaggle is “solely a tool just to combat mental health.” 

“This is just an electronic means,’ Duellman said, “But you and I know that there are a hundred different ways to see that your friend is in trouble.”

Regarding student confidentiality, Duellman believes that sacrificing privacy for safety is a tradeoff worth making:  “I don’t know that Gaggle is an overstep. I don’t know that surveilling what goes on the networks here and on school-owned devices is an overstep,” Duellman said. “The success of Gaggle is ‘Did it save a life?’”