A lot of neural networks are black boxes. We know they can successfully categorize things—images with cats, X-rays with cancer, and so on—but for many of them, we can’t understand what they use to reach that conclusion. But that doesn’t mean that people can’t infer the rules they use to fit things into different categories. And that creates a problem for companies like Facebook, which hopes to use AI to get rid of accounts that abuse its terms of service.
Most spammers and scammers create accounts in bulk, and they can easily look for differences between the ones that get banned and the ones that slip under the radar. Those differences can allow them to evade automated algorithms by structuring new accounts to avoid the features that trigger bans. The end result is an arms race between algorithms and spammers and scammers who try to guess their rules.
Facebook thinks it has found a way to avoid getting involved in this arms race while still using automated tools to police its users, and this week, it decided to tell the press about it. The result was an interesting window into how to keep AI-based moderation useful in the face of adversarial behavior, an approach that could be applicable well beyond Facebook.
The problem
Facebook sees billions of active users in a month, and only a small fraction of those fall into the category that the company terms abusive: fake and compromised accounts, spammers, and those using the social network to run scams. So while the company can (and does) use human moderators, the problem is simply large enough that they can’t be expected to catch everything. Which means an automated system of some sort is necessary if the service doesn’t want to be swamped by content it doesn’t want to see.
