Click With Caution
How LCPS surveils every document, email, and photo on its network 24/7 with Gaggle, a student monitoring service.
March 11, 2020
When students log on to their school-issued devices and programs, the veil of surveillance murmurs softly, monitoring, flagging, and tracking students’ every document, email, and stray social media notification that passes through the network. To students, the monitoring system exists as a nameless idea, supported only by warnings to avoid connecting personal devices to Chromebooks, until one day, it makes itself tangible as a visit to counseling. Even then, students are left wondering about what triggered the elusive system, the system’s identity, and if they should alter their technology usage to avoid a repeat incident.
Senior Nordina Taman had always taken steps to avoid signaling the surveillance system: never plugging her phone into her Chromebook and avoiding potentially problematic internet searches. That’s why she couldn’t have anticipated a flag on her college essay on her reality of “coming out of the worst of my severe depression.” Taman said that her counselor sat down with her and went through the essay with her without any other problems since she had discussed how she had overcome her depression, dispelling any threats. Through it all, Taman’s parents were never involved.
As the dust began to settle, Taman felt left in the dark, unaware of what in particular triggered the system, and was still left clueless about the extent of surveillance she was being subjected to daily, or what it was even called. “I didn’t even know it was called Gaggle,” Taman said, learning of its name during an interview for this story, “I just knew they had a ‘something’.”
Since the integration of technology-based learning in the 2018-2019 school year, nearly 85 thousand students have been subjected to surveillance by content moderation technology. Monitoring activity including homework, projects, essays, emails, artwork, chats, and even social media activity, students’ digital trails are monitored “24/7/365,” by Gaggle, a student surveillance service employed by LCPS and 1,400 other schools nationwide.
Gaggle markets itself as a one-stop solution to student safety in the digital world that supports “digital citizenship” and “creates positive school environments.” Promising “real-time content analysis,” the Bloomington, Ill. based company claims on their website that in the 2018-2019 school year, it helped “save 722 students from carrying out an act of suicide.”
Gaggle operates on a combination of machine-learning technology and human content moderation to alert school districts of potential threats of violence, substance use, suicide and self-harm, harassment, sexual content/pornography, and profanity. Gaggle scans communications against a “Blocked Words List,” which screens for predetermined words and phrases. Additionally, the service reviews content with an in-house, AI “Anti-Pornography Scanner.”
Available to three major educational services, Gaggle acts as a software plug-in for the Google G Suite for Education, Office 365, and Canvas. In LCPS, the software monitors the Google G Suite and Office 365.
Gaggle files User Violations on a “three strikes” system and warrants different responses by school administrators depending on the severity of the violation. According to invoices, incident reports, and company documents aggregated from the county by the Freedom of Information Act and documents obtained previously by BuzzFeed News in a nationwide study of Gaggle, the response by Gaggle’s safety management team depends on whether the infraction is marked as a “User Violation,” “Questionable Content,” or “Possible Student Situation” by the company’s Safety Management Rubric.
After AI technology marks any form of violation, human moderators assess the urgency of the infraction to determine the appropriate response. According to Gaggle’s Safety Management Procedures, Safety Managers review over “a million blocked student communications” each month. In the event that the algorithm flags a false positive that includes inappropriate images/language for the purpose of schoolwork, human moderators unflag tagged work on a case-by-case basis.
Content marked by the “Violations” category could constitute profanity/vulgar language or suggestive/provocative content. Gaggle’s standard response for “Violations,” the lowest tier of offense by the rubric, constitutes an email to the student, with administration CC’d to the email following multiple offenses. However, according to documents provided to The Blaze, LCPS does not notify students in the event of a standard violation. Hugh McArthur, the Supervisor for Information Security for LCPS, said that Gaggle “doesn’t often use that function or recommend it anymore” and “it wasn’t [LCPS’s] goal when implementing the product” as its purpose was “student safety.”
“Questionable Content” flags warrant an email from Gaggle representatives to “district specified contacts.” The QCON filter flags content that is serious but not an imminent threat. Gaggle responds to QCON incidents with an email to school representatives.
The most critical category, “Possible Student Situation,” is reserved for content that displays an “immediate threat” to student safety, and constitutes a phone call directly to school emergency contacts and, in some cases, law enforcement.
LCPS students are not notified by Gaggle when their content is flagged as a QCON or PSS violation by Gaggle guidelines. However, Gaggle maintains a record of all infractions and identifies repeat offenders on a “Top Concerns” chart on their User Dashboard.
Per incident reports provided by LCPS, during the 2018-2019 school year, Gaggle flagged 3096 PSS and QCON incidents, and over half of them were flagged on the basis of self-harm/suicide; 120 of the “Violence_Self” flags were marked as “Possible Student Situations.”
LCPS was unable to provide the “Blocked Words List,” saying in an email that the list was “proprietary” and not disclosed by the company. However, from incident reports disclosed by various Illinois school districts, BuzzFeed News compiled a list of words that had been flagged as QCON or PSS. In potential self-harm situations, Gaggle flagged “suicide,” “hate myself,” and various iterations of “end my life.” Gaggle also flagged LGBT terms like, “lesbian,” “queer,” and “gay” in the context of bullying and harassment. In general, profanities, specific drug and alcohol references, and language conducive to sexual content and sexual violence were marked as QCON violations.
According to their Internet Safety Program outlined in 2008, LCPS follows guidelines regarding student safety outlined in the Children’s Internet Protection Act enacted by Congress in 2000 in order to combat minors’ access to “obscene or harmful content.” Per the Federal Communications Commission, in order for districts to receive discounted rates on Internet access via “E-rates,” districts must monitor minors’ online activity and teach common-sense internet safety, alongside blocking inappropriate content.
Gaggle presents itself as a tantalizing option for districts that seek to minimize students’ exposure to a vast range of potentially harmful materials. However, the high-tech surveillance technology services provided by the for-profit company come at a steep cost. According to invoices for the 2018-2019 school year, LCPS paid upwards of 303 thousand dollars for coverage on the G Suite and Office 365 platforms. For the 2019-2020 school year, the bill amounted to nearly 375 thousand dollars, an increase of nearly 71 thousand dollars due to an increase in school-issued devices and a higher enrollment.
According to previous years’ invoices, LCPS has maintained contracts with Gaggle beginning in 2013 with only Office 365, and later expanding to include the Google G Suite after its implementation in 2018.
In the digital age, the market for one-stop security solutions has become increasingly lucrative, as companies such as Gaggle, Securly, and Bark, among others, contend for schools’ business promising CIPA compliance as well as above-and-beyond monitoring services. However, students are being forced to acknowledge and, in some cases, actively maneuver around the loss of privacy that comes with constant digital surveillance.
“It’s definitely always in the back of my head now when I look something up,” Taman said, “It’s not worth getting called in.”
Though the LCPS Student Technology Acceptable/Responsible Use Policy does disclose that there is “no right or expectation of privacy” on school-issued devices, Taman believes there should be greater transparency about monitoring systems that includes providing students with general guidelines as to what falls under flagged content.
According to principal John Duellman, though the county does implement technology lessons that cover topics including “social media presence” and “how to effectively use devices,” and requires that students sign a “memo of understanding” upon receiving their devices, Duellman agrees that more training could be effective.
“There’s probably some additional training that could go out there [to make students aware of Gaggle], but in that same respect, if you asked any student, ‘Have you heard of a Gaggle alert?’ I think most would say yes,” Duellman said.
“But, in terms of Gaggle, they’re doing the same internal debate I’m thinking about. I want to tell you things, but I don’t want to tell you too much,” Duellman said, “Because then, the people who are bad people are going to continue to find ways around what we do.”
“I think the primary thing is that it’s not a matter of surveillance, it’s a matter of trying to keep folks safe,” McArthur said. “We’re not looking for the bad kids, we’re looking for the sad kids,” McArthur said, quoting a Gaggle official.
Duellman believes that the act of documenting mental health on school networks is often a “cry for help,” though he doesn’t think that Gaggle is “solely a tool just to combat mental health.”
“This is just an electronic means,’ Duellman said, “But you and I know that there are a hundred different ways to see that your friend is in trouble.”
Regarding student confidentiality, Duellman believes that sacrificing privacy for safety is a tradeoff worth making: “I don’t know that Gaggle is an overstep. I don’t know that surveilling what goes on the networks here and on school-owned devices is an overstep,” Duellman said. “The success of Gaggle is ‘Did it save a life?’”