When algorithms go bad. Facebook was left red-faced this week after ProPublica reported that you could target audience categories such as ‘NaziParty’, ‘Jew Hater’, ‘How to burn Jews’ and more.

So how did this happen? It’s not that complicated really. Facebook has four different free text fields used to bulk out your profile. These are Education, Employment, Field of Study, and Job Title. The problem with these free text fields is that people can put whatever they want there, including hate speech. If enough people put Nazi Party as their employer, then guess what? That’s right, you can create an audience segment using it.

Essentially, Facebook ran a pure algorithm across these fields to see if it is statistically significant enough to use as an audience segment. What they should have done, is run and AI over the fields to try to understand context.

Needless to say, Facebook was pretty embarrassed by the incident and have since removed audience segmentation for those four fields.

According to Facebook:

“To help ensure that targeting is not used for discriminatory purposes, we are removing these self-reported targeting fields until we have the right processes in place to help prevent this issue,”