The development and use of artificial intelligence and machine learning (AI) is growing quickly in Australia and across the globe. As the prevalence of artificial intelligence increases, so too do concerns about the possibility for design flaws to have far-reaching negative consequences.

The concern that AI technology may reproduce bias via the people building it or the data used to train it has gained attention in several recent cases. A predictive healthcare algorithm, created by Optum, attracted criticism after a study published in the journal Science found that the algorithm discriminated against black patients. The percentage of black patients who should have been enrolled in specialised care programs increased from 17.7% to 46.5% when subject to human review by researchers. The algorithm used to set credit limits for Apple Card, Apple’s credit card launched in August this year, sparked criticism of gender discrimination following a series of claims, including one from the Apple co-founder Steve Wozniak, that significantly lower credit limits were being generated for women than for men in the same circumstances. New York’s Department of Financial Services has initiated an investigation into the credit card practices of Goldman Sachs, who provides the Apple Card. The UK Financial Conduct Authority (FCA) has expressed similar concerns that data based on race or ethnicity may be used within firms’ pricing models to produce different results in offered prices.

Many existing services, such as financial firms, discriminate within the parameters of the law, making differentiations in the provision of products such as mortgages and credit cards. Protecting against unfair discrimination in such decision-making is not a new issue. However, a new fear inspired by emerging technologies is that our biases will be embedded in design, then amplified exponentially as human agency is forfeited in favour of the functioning of automated systems.

Current approaches to address the potential for AI to perpetuate discrimination have focussed on the development of guidelines for ethical AI design and use. In the Australian Government’s recently released set of AI Ethics Principles the principle of fairness directs that AI systems should be “inclusive and accessible” and should not result in unfair discrimination. The eight principles developed by the Australian Government, in addition to fairness, are: human, social and environmental wellbeing; human-centred values; privacy protection and security; reliability and safety; transparency and explainability; contestability; and accountability.

Australia also endorsed the Organisation for Economic Co-operation and Development (OECD) Principles on AI in May 2019, which include respect for diversity as a core principle for the design of trustworthy AI. The European Commission’s Ethics guidelines for trustworthy AI similarly include a key requirement of diversity, non-discrimination and fairness. Other countries and corporations, including the IEEE, Google and Microsoft, have also joined the effort to design principles for ethical AI.

It is impossible to know the precise shape AI will take in the future. What is undeniable is that the evolution of this technology raises new legal questions including accountability in the use of automated processes and how to mitigate the risks of AI.

Authors: Michael Caplan, Kirish Kularajah, Asha Keaney