Some NC schools pay thousands to monitor for social media threats
A handful of school systems are hoping algorithms created by private companies can help surface issues of violence, bullying and self-harm. But do they work?
Posted — UpdatedStudents were fighting on social media, the alert indicated, an important kernel of information sifted and plucked from thousands of online posts specifically for the administrator of Orange County Schools in North Carolina.
Her phone lights up when certain keywords or other indicators are posted publicly.
“Death, shootings, murder, kill – things like that,” she said.
As director of student support services, Cobb doesn’t scour the social media accounts of Orange County’s 7,000 students, much less posts from the broader community, herself. She relies on a social media monitoring service the school system pays to scan it all. For nearly $12,000 a year, she can get email and text alerts about potentially problematic posts that may affect Orange County students and schools.
But not all of the alerts are helpful. When her team investigated the social media post about students fighting, they discovered it was students in Orange County, Florida.
Only a handful of school systems in North Carolina have tried social media monitoring services, platforms and software designed to alert users to a range of threats posted to sites like Facebook, Twitter and Instagram. But dozens more are actively exploring the idea or being pursued by companies marketing the product, according to a census of every one of the state’s 115 school systems conducted by WRAL News.
Contracts with these private firms can be comparatively cheap, compared to other security solutions. The two school systems that employ the technology in North Carolina spend less than $15,000 a year depending on their needs. And as schools grapple with concerns over school shootings – as well as other more common threats like fights and self-harm – administrators say the technology holds promise: Identifying when teachers, counselors or law enforcement officers need to intervene before something bad happens.
“The service is akin to a smoke detector. We scan public digital content. We don't surveil, we don't monitor, we don't follow,” said Gary Margolis, the founder and CEO of Social Sentinel, one of the platforms used by schools in North Carolina and nationwide. “We're not targeting individuals or groups. We don't target anyone. We're just looking at content.”
Privacy advocates, however, are doubtful.
They’re concerned about the rate of false positives that can waste the time of schools already struggling to provide both education and other needed services to students. And they point to other risks of the unproven technology, like the potential to exacerbate the consistent overpolicing of minority students.
“There's not a lot of research that shows these things do what you think they're going to do,” said Elizabeth Laird, a senior fellow at the Washington, D.C.-based Center for Democracy and Technology. “We're concerned these tools and data sharing are being implemented and procured without a clear use in mind.”
Social Sentinel isn’t alone in the media monitoring space, but WRAL’s census found it was the only paid service currently used by North Carolina school systems to specifically look for threats posted to social media.
Two districts – Orange and Dare county schools – use the Vermont-based company.
Two other North Carolina school systems used Social Sentinel in the past but no longer do. Vance County Schools discontinued the service due to a limited budget, according to district spokeswoman Terri Hedrick.
Onslow County Schools spokesman Brent Anderson said the district wasn’t satisfied with the service and decided not to renew it after finding that “nearly all” of the messages it flagged were Twitter posts.
“There was an obvious inability to monitor accounts where posts were set to ‘private,’ and the program did not adequately monitor social media sites which are popular with students and with a growing number of adults,” Anderson told WRAL News by email.
The software’s focus on publicly available information, Margolis said, is there by design.
“We've really taken the idea that by the time something tragic happens, however you want to define tragic, there are often bread crumbs along the way,” Margolis said. “If we can help identify some of those breadcrumbs long before it escalates to whatever that event turns out to be, then that's our mission.”
To do that, Social Sentinel scans public posts on social media accounts worldwide (most social media content isn’t explicitly tied to a specific location), comparing the language to a library of indicators of harm. That can include threats of mass violence, Margolis said, but much more commonly consists of posts about issues like bullying and self-harm.
“If we can identify something that triggers the library, and if we can associate it with a client, then we deliver that information to our clients in a way that they choose to receive it,” Margolis said.
That could be an email, or as in Cobb’s case at Orange County Schools, by email and text.
“The few things that we have had to act on, thank goodness, let me just say, thank goodness that they have not necessarily been Orange County Schools’ students,” Cobb said. “But there have been a few that we've had to intervene on, and probably the majority of those were out of state.”
In a typical week, she receives about five to 15 alerts on her phone. But during the ACC basketball tournament, she was overwhelmed with “shooting” alerts.
“‘Oh, look at that shooting!' and, 'Look how he's shooting!'” Cobb said, recalling some of the posts that were picked up by the monitoring service and sent to her. “I was able to let [Social Sentinel] know, ‘Hey, I'm getting a lot of alerts around basketball.’ They tightened their monitoring and then that changed. So that was really good.”
Margolis said tuning their automated systems to detect those differences is a big part of his company’s challenge. And they confront it with a team of data scientists and linguists who partner with universities to better understand the meaning behind language.
“Now, I'm always quick to say: We're pretty good at it. We've reduced north of 90 percent of the false positives. But we don't get everything,” Margolis said. “And computers haven't replaced yet the human mind, and the experience that goes with understanding the nuance.”
Many of the alerts sent to Orange County Schools involve people talking on social media about politics, news or opinion topics, Cobb said. She has received “a few” alerts that pertain to students in her school system, but nothing life-threatening.
She didn’t share details to protect students’ privacy, but said her team has talked with a few students who seemed to be struggling based on messages they posted.
“We've had some that I've said, ‘Wow, this is good. I'm glad we were able to intervene,’” Cobb said. “But nothing I would say that was of a nature where a student was going to harm themselves or others.”
“Our belief has been that if we can be proactive and be made aware, it can be one more layer we can add for our schools,” Parker said.
That narrower focus also cuts down on the number of alerts to sift through, he said.
Gaggle is a paid service. But a similar product called Bark provides services to school systems for free, in addition to offering a companion paid service for parents designed to track their specific child’s social media accounts. Bark is now being used in 31 schools and school districts in North Carolina, according to Titania Jordan, the company’s chief marketing officer.
But these services are distinct from automated monitoring software that surveys the broader universe of social media.
“We are pretty unconvinced these programs are able to identify the next school shooter, which is essentially what they're promising to do,” said Faiza Patel, co-director of the Liberty & National Security Program at the Brennan Center and a co-author of the study.
One of the issues, she said, is their inability to accurately derive meaning from speech – especially the rapidly evolving speech of children and teenagers online.
"If you are getting alerts from these people, you have to have someone looking at them, responding to them and deciding which ones are nonsense and which ones aren't,” Patel said.
That raises the cost of such a system, as districts shuffle the duties of their personnel. But Patel said there’s also the potential for other unintended consequences for minority students, whose use of nonstandard English isn’t always well represented in training data used to tune the threat monitoring algorithms.
“There is a sense that algorithms can figure anything out,” Patel said. “That's just not true when it comes to figuring out what people are saying.”
Even without automated systems, the disparity in discipline for students of color is well established in this state and across the country.
Patel worries social media monitoring services could make that trend worse.
“I don't want my kids to go to a school where they are looking at them as potential suspects,” Patel said. “I don't think anyone else does either.”
Margolis, who has children in high school, said he doesn’t want that either. That’s why in addition to only monitoring public accounts, he said his company is committed to building an unbiased system.
“How we speak in inner-city Chicago is different than how we speak in Southern California. It's different than how we speak in Burlington, Vermont,” Margolis said. “We want to be able to identify those indicators of harm and do it fairly.”
Orange County parent Latarndra Strong, who has two children in the district, shares some of Patel’s concerns. She first heard about social media monitoring during a Board of Education meeting, when board members discussed the Social Sentinel contract. She’s since reviewed the contract herself.
School safety has been a top priority for Strong, who leads the Orange County Hate-Free Schools Coalition.
“I could see where there might be a good thing to do this, but it feels a little bit like Big Brother to me,” she said. “Anytime somebody is collecting data, I think it always presents a concern because, like, what are you doing with that data? How often are you purging it? You know, there's all these questions.”
Orange County Schools leaders say they do not have a list of students’ social media accounts, nor have they asked Social Sentinel to review students’ accounts specifically.
“We don't send a list of students to the company. This is very general,” Cobb said.
“We haven't built a tool that follows people or groups. You can't do that with our system," Margolis said. "We've been very conscious about building something which is safety related and content focused so we can protect people's rights to privacy and assembly.”
As for the data itself, Social Sentinel spokesperson Alison Miley said although the company keeps a dataset of potential threats culled from social media to train and refine the system, that content contains no usernames. She said no other data is kept or archived.
“That dataset is only used for this purpose and not accessible outside our data science and engineering team and can't be referenced in the future if someone wanted to look up a user, for instance,” Miley said in an email.
It’s not terribly surprising that schools are turning to data and technology to try to improve school safety, says Laird, with the Center for Democracy and Technology. A former privacy officer in D.C.’s state superintendent's office, she now leads the D.C. nonprofit’s student privacy project and sees the use of these tools as an emerging trend.
She shares Patel’s concerns about the potential for unintended harm – either due to chilled speech or discriminatory outcomes. And there are questions, she says, about just how effective these systems actually are.
“The challenge is that there's a tremendous urgency to act,” Laird said. “At the same time, lack of evidence around what works.”
Although few schools use social media monitoring services in North Carolina so far, several administrators WRAL News spoke with noted they were mulling their options as conversations continue around the issue of school safety.
Their caution makes sense to Michele Gay, executive director of Safe and Sound Schools. A former elementary school teacher, Gay co-founded the advocacy group after her 7-year-old daughter, Josephine, was killed in the 2012 shooting at Sandy Hook Elementary in Newtown, Conn.
“The reason you don't see it taking off like wildfire is because our schools are being careful,” Gay said. “They're not wanting to cause harm, they're wanting to use it effectively.”
Although she said the software offers educators a proactive way to keep “another finger on the pulse” of the school, it does have potential drawbacks.
“The primary concern most of us have is that it will affect the climate and culture of the community,” Gay said. “That's how people learn and grow best, when people are in a place they feel comfortable and aren't looking over their shoulders feeling spied on.”
She said schools systems can implement these platforms effectively by routing alerts to multidisciplinary teams made up of public safety professionals, counselors and other educators. That way, schools can appropriately address a range of potentially harmful behavior – from threats to thoughts of suicide.
“The schools that are doing well with it, they've taken the time to learn about intelligence gathering and how to do it in a way that's not a violation of privacy,” Gay said.
That’s also the approach Margolis recommends, noting that not every alert should be reported to law enforcement.
“Our system is designed to bring to bear help,” he said. “It's not a system to get people in trouble.”
Gary Gibson, who worked as superintendent of Clay County Schools until about two weeks ago, said he’s seen the benefits of these types of teams first hand. His district, nestled in the North Carolina mountains along the western tip of the state, doesn’t use social media monitoring. But he said he worked to sell the community on its benefits, having come from a district in Georgia where they paired software with staff to monitor alerts and respond accordingly.
"You've got to have a director or some administrator that gives some oversight to those working with the software and brings sanity and wisdom to the decision making,” Gibson said in a phone interview in early May. “Is this something where we're stepping over their rights? Or is this something we're doing to protect them and protect all the students at school?”
He said he wasn't terribly bothered with the prospect of false positives, given the potential gains in safety.
“Sometimes you investigate and it's nothing, but at least you know it's nothing,” Gibson said. “That's what we need is a little peace of mind and to be able to say we're doing this because we care about our kids.”
No matter how schools choose to monitor social media for threats and other problems, Raleigh high school senior Greear Webb urges them to include students in the discussions.
“I think a lot of students, especially after the Parkland shooting where social media was such a big factor, where students were able to see bodies and people running and teachers shouting and screaming down in Florida, I think that really affected us and really hit us hard,” he said. “We brought in about 700 members of the community into Sanderson High School and kind of talked about what students can do to make sure that we're leading the conversation on keeping ourselves safe in school.”
Since then, Webb has gotten more involved in education policy and is working as a legislative intern with the State Board of Education. He plans to attend the University of North Carolina at Chapel Hill this fall and study public policy.
During his time at Sanderson High, Webb has experienced threats to his school and endured lockdowns. He has heard the words “Code Red” over the school’s loudspeaker and huddled in a corner with classmates, texting his family members and friends as his teacher locked the door, turned off the lights and closed the blinds.
“I've definitely felt fearful and I've definitely been scared for the lives of my peers and for my own life,” he said. “When these ‘Code Reds’ are happening, that's what I'm thinking, ‘How can I save my life, the life of family members that I may have in the school and the life of my peers?’ while also going through those emotions of being afraid and being fearful and trying to console classmates. It's really a juggling act, and it's really sad in this day and age.”
Constant news of school lockdowns and shootings is affecting students’ mental health, Webb said, so he welcomes any conversations about how to better use social media to help find and prevent threats.
“It's really taking a strain on students and administrators and teachers,” Webb said. “And it's making us question, ‘Are we really safe at our schools these days?'”
Whether the approach to school safety involves social media monitoring or not, school administrators said one thing is clear: It will only be one part of the solution.
“By and large, the most valuable thing we can do is build healthy relationships with our kids,” Parker, of Dare County, said. “It takes a lot of things that come together to continue to have healthy and safe schools.”
Related Topics
• Credits
Copyright 2024 by Capitol Broadcasting Company. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.