A face matching algorithm points the police at entirely the wrong brown guy. Despite it being obviously wrong, the innocent guy is nicked anyway (because brown?) and has experienced ongoing consequences of the erroneous arrest.
This is not a data problem. Or rather, it’s not just a data problem.
The data problem
Facial recognition systems that are developed, trained and built in environments where pale-skinned people are the majority tend to have higher rates of erroneous matches for faces of people with darker skin. Cross-racial identification is known to be more difficult than intra-racial identification, and so an insufficiently racially-diverse development team is unavoidably going to impart their unconscious biases to the system. When social power dynamics and historical societal injustice are factored in, this ceases to be a mere statistical artefact and becomes a cause of real harm. It is the decisions made by humans about input data, training data, confidence weighting data and reliability data which embed their biases into the system.
…is not the only problem
Social bias coded into algorithmic systems is then exacerbated by humans’ willingness to defer to the system’s outputs even when they are contradicted by the evidence of their own eyes, because nodding along with the machine is the path of least resistance. A worker who accepts algorithmic output despite it being self-evidently wrong is going to be given more grace by management than a worker who refuses an algorithmic output which is ambiguous or unreliable - it’s just easier and safer for the worker to refrain from contradicting the machine.
Policy documents are no defence against automation bias in a workplace that discourages challenge or critique. For these tools to be genuinely assistive rather than directive, it must be safe to refuse their ‘assistance’ without punishment for doing so over-cautiously. ‘Training’ is also ineffective as a countermeasure for automation bias on its own, because the issue isn’t just recognisance of erroneous output, it also arises from the power dynamics of credibility and authority to make independent decisions within the organisation.
If a person defers to the algorithm, the algorithm can be blamed for adverse outcomes and accountability is diffused to the point of evaporation. However, if a person does not defer to the algorithm they alone will carry the burden of responsibility for any adverse outcomes.
None of that is necessarily intentional or deliberately nefarious - it’s just how humans roll. But to pretend it’s legitimate to install algorithmic judgment tools in a workplace without radical measures to detoxify its power dynamics, is irresponsible and naive.
Therefore, IMHO, any DPIA carried out for high-impact automated judgement systems must start with the default assumption that the environment is unsafe for deployment unless active measures are implemented (and sustained!) to identify and counter automation bias. We need to tech-proof humans, not human-proof tech.