In 2014, Dutee Chand, winner of medals at two Asian Games, learned from newspapers that her testosterone levels were “too high” for her to be a woman. Asked by the Commonwealth Games committee to undergo treatment, she refused. Instead, she challenged the matter before the Court of Arbitration for Sport.

There, her lawyers asked if anybody could demonstrate the link between high testosterone levels and higher athletic performance. Was there actually a link or medical evidence to demonstrate it? The evidence never showed up, and the testosterone levels standard was made void.

Dutee won.

Here’s why Dutee’s case is pertinent to our conversation today: when a standard or a classification is built on an idea of “normal”, then the decision-making system will fail those outside that idea of “normal”. It will fail the extraordinary. And, if you think about it, nature is not neat. It is the outliers that push the species forward – something automated decision-making systems, including algorithms, don’t really understand.

Dutee could challenge this exclusionary decision in court because she knew why she was being rejected. Understanding the decision making, and the transparency in it, was crucial to her right to redress.

In the final episode of Let’s Talk About Big Data, we talk to Laura Reig, a PhD student at Denmark Technical University, on how AI makes mistakes in gender classification, and Chirag Agarwal, a research fellow at Harvard University, about what explainability in AI means. We also talk to Joy Lu, an associate professor at Carnegie Mellon University, on what makes for a good explanation of what an algorithm is doing. Is it accuracy? Is it ease of understanding?

Listen.