The UK Government recently released its Guidelines for AI procurement. The UK is a pilot for the World Economic Forum’s global procurement guide, AI Procurement in a Box. 

The Guidelines are a useful ‘buyer beware’ template for public and private organisations which are considering acquiring or commissioning their first AI app.

Only use AI if you need it

The Guidelines repeatedly emphasise that AI solutions are not a goal in themselves: the Guidelines suggest asking “how could AI technologies potentially benefit us?” rather than “how can we make our problem fit an AI system solution?”.

Focus on the challenge, not the solution

The Guidelines repeatedly encourage a focus on the clearly identifying the problem to be addressed, rather than specifying a solution. They recognise that the understanding of the problem – and AI’s suitability to address – may change over the course of the project, and so an iterative development model is recommended. This includes revisiting the analysis of benefits vs risks of solving the problem through AI.

Think (collectively) before leaping to market

The Guidelines caution that you need to settle on two key goal posts before going to market:

1. set the ethical boundaries of the AI decision making, including as it learns.  This involves understanding the human and socio-economic impacts of your AI system. Before starting an AI project, you must have an internal AI ethics approach in place - with examples of how it will be applied to design, develop, and deploy AI-powered solutions so you get past bland statements (this could be called the Rowena Orr ‘are you meeting community expectations’ test).
2. as AI ‘feeds’ off your data set, you must thoroughly audit the limitations of the data you have available.  Address flaws and potential bias within your data (e.g. postcode as indirect signifier of race or social status).   If this is not possible before going to market, make the AI developer’s first priority to conduct a comprehensive check of the data the AI system will use to base its decisions upon.

Establish a multi-disciplinary team  to comprising the following skill sets: Domain expertise (e.g. healthcare, transportation); Commercial expertise; Systems and data engineering; Model development (e.g. deep learning); Data ethics; and Visualisation/information design.  Government/community relations, Legal and CSR should be consulted.

Understanding is control

The Guidelines caution:

As AI is an emerging technology, it can be more difficult to establish the best route to market for your requirements, to engage effectively with innovative suppliers or to develop the right AI specific criteria and terms and conditions that allow effective and ethical deployment of AI technologies.

You need to be actively engaged with the AI developer to test that your ethical and data ‘goal posts’ are not being compromised.  You must:

• periodically check the AI developer has addressed any issues of bias within the data? Why are their strategies appropriate and proportionate? How will they address issues you may have missed?
• draft evaluation questions that inform you about the algorithms and models used by the AI developer. Establish any limitations of their model to avoid “black-box” algorithms.

Oversight must actually oversee

Methods of governance which are used elsewhere are not good enough. As AI learns, traditional ‘set and forget’ compliance strategies will not work.

It has to be a key requirement of the design brief that t the AI developer build into the AI the tools which you will need to monitor the AI’s performance:

Enable end-to-end auditability by implementing process logs that gather the data across the modelling, training, testing, verifying, and implementation phases of the project lifecycle. Such a log should allow for the variable accessibility and presentation of information with different users in mind to achieve interpretable and justifiable AI.

Your governance process should monitor erroneous decision-making creeping into the AI which adversely impact customers. Make sure the AI developer has transferred the skills you need to track the AI’s learning trajectory – and recognise that AI ‘aftercare’ may be beyond the current skill set of your inhouse IT team and you may need to hire in expertise.

Read more: AI Procurement in a Box: Project overview