Diversity and discrimination practice questions.

An audit published in 2018 tested three commercial gender-classification services on a set of parliamentarians' portraits and reported error rates below 1 per cent for lighter-skinned men and up to about 35 per cent for darker-skinned women. Each supplier had published a high overall accuracy figure. What does the audit's method show about such figures?

• A single accuracy figure can hide a failure that affects one group
• Accuracy cannot be measured reliably outside a laboratory setting
• The services had been developed using portraits of politicians only
• Classification accuracy always falls once a test set is enlarged

Answer: A single accuracy figure can hide a failure that affects one group

Aggregate accuracy is an average over whoever happens to be in the test set, so a subgroup with few members can fail badly without moving it; the audit's contribution was to report results disaggregated by skin tone and sex. The fault therefore sits in the benchmark and in the reporting practice, not only in the model.

An online retailer trained an experimental CV-screening model on ten years of its own hiring decisions. Reporting in 2018 described how the model downgraded CVs containing the word 'women's', and the tool was abandoned. A colleague argues that deleting the applicant's sex from the data would have prevented the problem. Why is that insufficient?

• The model could not process free text, only structured fields
• A model cannot be retrained once its training data are fixed
• Wording, clubs and colleges on a CV still correlate with sex
• Ten years of decisions is too short a period for a model to learn

Answer: Wording, clubs and colleges on a CV still correlate with sex

The protected field carries the label, not the information: correlated attributes remain in the record and the model optimises on whatever predicted the historical decision, which was itself made by people. This is why removing a characteristic from the inputs is a weaker remedy than testing outcomes for the group it identifies.

A risk-scoring tool used in pre-trial decisions in the United States was shown by a 2016 analysis to label a higher share of Black defendants than white defendants as high risk among those who were not rearrested. The supplier replied that any given score carried the same probability of rearrest for both groups. Which statement best describes the dispute?

• One of the two analyses must contain an arithmetic mistake somewhere
• The disagreement disappears once race is removed from the inputs
• The tool was accurate, so the analysis was not really about fairness
• Each side measured a different property of the same tool

Answer: Each side measured a different property of the same tool

Equal predictive value within a score band and equal false-positive rates across groups cannot both hold when the underlying rearrest rates differ, so a tool can satisfy one definition of fairness while failing another. The examinable move is to replace 'is it biased?' with 'which fairness property was chosen, and who chose it?'

In January 2020 police in Detroit arrested a man after a face-recognition search of a driving-licence database returned his photograph as a candidate match to shop security footage; investigators then showed that photograph to a witness, who agreed. The charge was later dropped. Which step most directly produced the wrongful arrest?

• The security footage was recorded at too low a frame rate for analysis
• The database of driving-licence photographs was too small to search well
• A candidate match was treated as an identification rather than a lead
• The software reported the match with a confidence score below fifty

Answer: A candidate match was treated as an identification rather than a lead

The system returns a ranked list of possibilities, and the failure was to build the rest of the case on that list instead of seeking independent evidence, with the photo array then confirming the machine's suggestion. Vendor and police guidance describe the output as an investigative lead precisely because a probabilistic match is not proof of identity.

A study published in 2020 tested five commercial speech recognition systems in the United States and reported average word error rates roughly twice as high for Black speakers as for white speakers. A company screens recorded video interviews by scoring the automatic transcript. What is the most significant implication for applicants?

• Interviews from some applicants will take noticeably longer to process
• Some applicants are scored on a poorer transcript of the same answer
• The company needs more storage for the recordings of some applicants
• Applicants can defeat the system by speaking more slowly than usual

Answer: Some applicants are scored on a poorer transcript of the same answer

The scoring stage never sees the candidate's answer, only a reconstruction of it, so a group whose speech is transcribed worse is assessed on degraded evidence and the disadvantage is invisible in the score. The affected stakeholders are applicants, but the accountability sits with the employer that adopted the tool, not with the supplier alone.

A retail chain with shops in the European Union wants software that infers employees' emotional states from webcam images in order to rate their service. Under the EU Artificial Intelligence Act, whose prohibitions have applied since February 2025, how is this use classified?

• Prohibited, because it infers workers' emotions at their workplace
• High risk, so it is permitted once a conformity assessment is passed
• Limited risk, so it is permitted if employees are told it is in use
• Outside the Act, because the images are processed and deleted on site

Answer: Prohibited, because it infers workers' emotions at their workplace

The Act bans a short list of practices outright, and emotion inference in workplaces and educational settings is one of them, on the grounds that the power imbalance makes consent meaningless and the inference itself is contested. Tools that screen job applications sit one tier below, in the high-risk category, which regulates rather than forbids.