Artificial Intelligence and Ethics · Weeks 19-27

Algorithmic Bias and Fairness

Investigating how human prejudices can be encoded into automated decision-making tools.

About This Topic

Algorithmic bias occurs when the data used to train a machine learning model, or the design choices made by engineers, reflect existing social prejudices. In US 11th-grade computer science, this topic connects abstract programming concepts to real-world consequences. Landmark examples like the COMPAS recidivism scoring tool used in US courts and Amazon's scrapped hiring algorithm give students concrete cases where code had measurable, unequal effects on real people. CSTA standards 3B-IC-25 and 3B-IC-26 push students to analyze these impacts and propose systemic responses, not just individual fixes.

Students often underestimate how bias enters a system. Training data reflects historical inequalities, and if those inequalities go unchallenged, the model amplifies them. Features like zip code or name can act as proxies for race even without explicit coding of race as a variable. Understanding this mechanism is foundational to responsible AI development.

Active learning is particularly effective here because bias is contested and contextual. Students who debate real cases, audit sample datasets, or role-play as audit committee members develop the critical reasoning needed to evaluate AI systems throughout their careers. Structured argumentation and case analysis give students practice making evidence-based claims about systemic issues rather than vague generalizations.

Learning Objectives

• Analyze how specific features in training data, such as zip codes, can act as proxies for protected attributes like race or socioeconomic status.
• Evaluate the societal impact of biased AI systems by comparing outcomes for different demographic groups in scenarios like loan applications or predictive policing.
• Design a mitigation strategy to address bias in a hypothetical machine learning model, detailing steps for data preprocessing or model adjustment.
• Explain the ethical implications of deploying AI systems that perpetuate or amplify existing societal inequalities.

Before You Start

Introduction to Machine Learning Concepts

Why: Students need a basic understanding of how models are trained on data to grasp how bias can be encoded.

Data Representation and Types

Why: Understanding different data types and how they are structured is essential for analyzing potential proxy variables.

Key Vocabulary

Algorithmic Bias	Systematic and repeatable errors in a computer system that create unfair outcomes, such as privileging one arbitrary group of users over others.
Training Data	The dataset used to train a machine learning model; biases present in this data can be learned and amplified by the model.
Proxy Variable	A variable that is correlated with a sensitive attribute (like race or gender) and can inadvertently introduce bias into a model even if the sensitive attribute itself is not used.
Fairness Metrics	Quantitative measures used to assess whether an AI model's outcomes are equitable across different demographic groups.
Disparate Impact	A situation where a policy or practice has a disproportionately negative effect on members of a protected group, even if the policy is neutral on its face.

Active Learning Ideas

Case Study Analysis: COMPAS and Hiring Algorithms

Assign small groups one of two documented bias cases (COMPAS criminal risk scoring or Amazon's hiring algorithm). Groups read a summary, identify where bias entered the system, and present findings to the class using a structured claim-evidence-reasoning format.

⏱ 40 min·Small Groups

Generate mission

Structured Academic Controversy: Should Biased AI Be Banned?

Pairs argue that a specific biased AI system should be banned outright, then switch and argue for regulation instead of prohibition. After both rounds, partners synthesize a position that addresses both the harms and the practical tradeoffs of each response.

⏱ 35 min·Pairs

Generate mission

Dataset Audit: Find the Bias

Provide groups with a simplified synthetic dataset (e.g., fictional loan approval records). Groups use frequency counts and comparison tables to identify which features correlate with protected characteristics, then present what mitigation steps they would take.

⏱ 30 min·Small Groups

Generate mission

Gallery Walk: Mitigation Strategies

Post six posters around the room, each describing a different bias mitigation technique (e.g., re-sampling, fairness constraints, post-hoc correction). Students rotate, evaluate each strategy's strengths and limitations on sticky notes, and the class debriefs on which strategies address root causes vs. symptoms.

⏱ 25 min·Whole Class

Generate mission

Real-World Connections

Facial recognition software has shown higher error rates for women and people of color, impacting its use by law enforcement agencies like the NYPD and potentially leading to wrongful identification.

Hiring algorithms, like one previously used by Amazon, have been found to discriminate against female candidates because they were trained on historical data reflecting male dominance in the tech industry.

Credit scoring models used by financial institutions such as Chase or Wells Fargo can exhibit bias if historical lending data reflects discriminatory practices, affecting access to loans for certain communities.

Watch Out for These Misconceptions

Common Misconception

If an algorithm doesn't explicitly use race or gender as inputs, it can't be biased.

What to Teach Instead

Proxy variables like zip code, school attended, or name can encode protected characteristics indirectly. Active dataset audits where students trace correlations firsthand make this mechanism concrete rather than abstract.

Common Misconception

Algorithmic bias is purely a technical problem that better data will solve.

What to Teach Instead

Bias is often a structural problem rooted in historical inequities; more data collected from a biased system just encodes the inequity at larger scale. Case-based discussions help students see why policy and design choices matter alongside data quality.

Common Misconception

Bias only matters in obviously high-stakes domains like criminal justice.

What to Teach Instead

Content recommendation, targeted advertising, and search rankings also reflect and reinforce biases with broad societal effects. Examining a range of domains during class broadens students' radar for where bias operates.

Assessment Ideas

Discussion Prompt

Present students with a case study, such as a biased AI in college admissions. Ask: 'Identify at least two ways bias could have entered this system. Discuss the potential consequences for applicants from underrepresented groups. What is one specific step an engineer could take to address this bias?'

Quick Check

Provide students with a short description of a hypothetical AI system (e.g., an AI for recommending job candidates). Ask them to write down: 'One potential source of bias in the training data. One proxy variable that might lead to unfair outcomes. One fairness metric that could be used to evaluate the system.'

Exit Ticket

Ask students to write: 'One real-world example of algorithmic bias we discussed. One reason why it is challenging to eliminate bias from AI systems. One question you still have about AI fairness.'

Frequently Asked Questions

How does algorithmic bias happen if programmers don't intend to discriminate?

Bias enters systems through training data that reflects historical inequalities, feature selection that uses proxies for protected characteristics, and optimization metrics that ignore differential outcomes across groups. Intent is irrelevant when the mechanism produces unequal results across demographic groups.

What real-world AI systems have been found to be biased?

Documented examples include the COMPAS recidivism tool (higher false positive rates for Black defendants), Amazon's hiring algorithm (penalized resumes mentioning women's colleges), and facial recognition systems that perform significantly worse on darker-skinned faces, particularly women.

What does fairness mean in machine learning?

Fairness has multiple competing technical definitions, including equal accuracy across groups, equal false positive rates, and calibrated probability scores. These definitions can be mathematically incompatible, which is why choosing a fairness criterion is a values decision, not just a technical one.

How does active learning help students understand algorithmic bias?

Active learning puts students in the role of auditors and critics rather than passive recipients of examples. When students trace bias through a real dataset or argue competing positions on a bias case, they build the analytical habits needed to evaluate AI systems they will encounter in the workplace and as citizens.