It Rejected the Word “Women’s”

In 2014 Amazon’s team began building an internal AI tool that was meant to speed up hiring. Feed it a pile of resumes and it ranks the best candidates.

They trained it on ten years of Amazon's own hiring history. Which, like most of tech at the time, skewed heavily male.

By 2015, their own engineers noticed something odd. The model had started penalising resumes that contained the word "women's." As in "women's chess club captain." It had also started downgrading graduates of two all-women's colleges, without anyone ever telling it those colleges existed or that gender was a category worth noticing.

Nobody programmed it to do that. It worked it out on its own, from the pattern in ten years of decisions that were already skewed before the model’s time.

Amazon's team tried editing it to stop that specific behaviour. But they reportedly couldn't be confident it wasn't just finding some other, quieter way to arrive at the same bias. The project was shelved in 2017 and it never screened a real candidate or decided on a real hire on its own. This only became public a year later, through a Reuters investigation.

What gets missed when this story is told is that the AI didn't invent anything. It simply mirrored reality. It reflected back what had been unfair for a decade, and did exactly what it was built to do with that information.

SOURCES

Jeffrey Dastin, "Amazon scraps secret AI recruiting tool that showed bias against women," Reuters (11 October 2018), the original investigation that broke this story.

Privacy International, "After three years, Amazon stopped using an AI-based hiring tool that discriminated against women" (2018), for the 2015 editing attempt and 2017 abandonment timeline.

OECD AI Incidents and Hazards Monitor, Incident 37, for independent verification of the timeline and outcome