In 2012, Instagram started moderating certain terms used by pro-anorexia groups, making it one of the more tightly moderated social media platforms. A group of Georgia Tech researchers decided to study if banning such words helped the communities using them. Instead, they found that it may make matters worse.
Like many social media sites, Instagram found itself host to a number of questionable communities. Groups of people with eating disorders formed communities to promote anorexia, bulimia, and other eating disorders as a lifestyle. They traded tips on how to diet or purge, shared pictures of their bodies that made them feel ashamed and pushed them to diet more, and “thinspiration” photos of extremely thin celebrities.
So Instagram started aggressively moderating key terms like “thighgap”, “imugly”, and “thinspiration”. Searching the terms produced no results, and when posts were tagged with those results, users had to first view a screen advising them that they would be viewing questionable content, and occasionally providing links to helpful sites or hotlines.
Georgia Tech researchers used a program to view 2.5 million posts made on Instagram between 2011 and 2014, searching for those banned words. They presented their results was at the ACM Conference on Computer-Supported Cooperative Work and Social Computing on March 1 in San Francisco.
They found that banning exact terms did little to stop the communities who used the terms. Instead it spawned variations in spelling, such as “thyghgapp” and “thinspoooooo”. Instagram banned 17 terms but soon there were at least 250 such augmented terms specifically used by pro-eating disorder communities. The alternate spellings also popped up on tumblr and other social media platforms.
Did banning the terms foster an “us versus them” attitude in the communities, or make them double down on their commitment?
Graduate student Stevie Chancellor, one of the study’s co-authors, told Gizmodo that she is reluctant to guess at the psychology of the people in the communities. “That’s a correlation or causation problem”, she said. But after the bans, the communities tended to talk about, and tag, more extreme behaviour. “Our observations show that people who used the new terms also used terms like ‘isolation’, ‘loneliness’, or ‘self-harm', she said. What caused the change isn’t exactly clear. Chancellor believes it could be “not having the outside influence of someone saying ‘come talk to me if you need me,’ instead of ‘hey, good job, keep it up’”.
“We’re not drawing causal inferences,” co-author Munmun De Choudhury, an assistant professor at Georgia Tech, confirmed, adding that the increase in terms mentioning self-harm and isolation, “does temporally follow the banning”.
Instead of banning terms, De Choudhury and Chancellor think social media platforms should modify the recommendation system to show users “content that is very different from what they are looking for”. A search for “thinspo” should return not just results for thinspiration photos, but for communities of people in recovery from eating disorders, body acceptance posts, social services posts, and other posts that allow searchers to get a glimpse of other points of view and groups that might help them get healthier.
Might these insights be applicable to hate groups or criminal groups? They can certainly be just as inventive when it comes to dodging banned search terms. “It is human nature to come up with a different way out,” said De Choudhury. However, the psychology and social context of hate groups is very different from those of people suffering from mental health problems or eating disorders, so in such cases, “it might make sense to ban things”.
Chancellor agreed that “people are going to evade” restrictions on search terms under almost any circumstances. However, she thinks it might be possible for a similar modified approach to work in other groups: modifying search terms so that they return results that give people searching for them a different perspective or a wider base of knowledge. “This might be the first step in that general direction.” [Presentation, CSCW 2016]