Sampling and Data Bias

Active learning helps students confront sampling bias directly, where abstract definitions often fail to stick. When students manipulate sampling methods themselves, they see firsthand how convenience samples distort results, making the abstract concept of bias tangible and memorable.

• Year 12
• Mathematics
• 4 activities
• 25 min–45 min

Learning Objectives

1. Critique the limitations of simple random sampling in achieving a truly representative sample.
2. Analyze how specific sampling methods, such as convenience sampling, introduce systematic bias.
3. Compare and contrast stratified sampling with simple random sampling, justifying the choice for heterogeneous populations.
4. Design a stratified sampling plan for a given scenario, ensuring proportional representation of key subgroups.
5. Evaluate the validity of statistical conclusions drawn from biased or non-representative samples.

Simulation Game

⏱ 30 min·Pairs

Simulation Game: Dice Roll Sampling

Assign students a population of 100 dice rolls (pre-generated data sheet). In pairs, they take simple random samples of size 20, then stratified by even/odd numbers. Compare means and discuss bias. Graph results for class share.

Prepare & details

Explain why a truly random sample is often difficult to achieve in practice.

Facilitation Tip: During Dice Roll Sampling, have students record their results in a shared table to visibly compare biased versus random outcomes.

Setup: Flexible space for group stations

Materials: Role cards with goals/resources, Game currency or tokens, Round tracker

• Apply
• Analyze
• Evaluate
• Create
• Social Awareness
• Decision-Making

Case Study Analysis

⏱ 45 min·Small Groups

Survey Station Rotation: Bias Hunt

Set up stations with mock surveys: convenience (near door), quota (fixed groups), and cluster (class sections). Small groups sample classmates on a topic like study habits, rotate, then analyze response distributions for bias indicators.

Prepare & details

Analyze how the choice of sampling method introduces systematic bias into a study.

Facilitation Tip: In Survey Station Rotation, rotate student groups every 7 minutes so they encounter multiple examples of bias in a short time.

Setup: Groups at tables with case materials

Materials: Case study packet (3-5 pages), Analysis framework worksheet, Presentation template

• Analyze
• Evaluate
• Create
• Decision-Making
• Self-Management

Case Study Analysis

⏱ 35 min·Whole Class

Case Study Debate: Real Polls

Provide excerpts from flawed polls (e.g., 1936 Literary Digest). Whole class divides into method critique teams, debates sampling errors, and proposes fixes like stratification. Vote on best justifications.

Prepare & details

Justify when it is appropriate to use stratified sampling over simple random sampling.

Facilitation Tip: Use Population Jar Draw to physically demonstrate how simple random sampling works, letting students draw and immediately see the distribution in the sample.

Setup: Groups at tables with case materials

Materials: Case study packet (3-5 pages), Analysis framework worksheet, Presentation template

• Analyze
• Evaluate
• Create
• Decision-Making
• Self-Management

Case Study Analysis

⏱ 25 min·Individual

Population Jar Draw: Hands-On Randomness

Fill jars with colored beads representing population subgroups. Individuals draw simple random vs. stratified samples, calculate proportions, and reflect on deviations from true values in journals.

Prepare & details

Explain why a truly random sample is often difficult to achieve in practice.

Facilitation Tip: For Case Study Debate, assign roles (pollster, statistician, critic) to ensure all students engage with the material actively.

Setup: Groups at tables with case materials

Materials: Case study packet (3-5 pages), Analysis framework worksheet, Presentation template

• Analyze
• Evaluate
• Create
• Decision-Making
• Self-Management

Teaching This Topic

Teachers should focus on two key moves: first, let students experience the failure of biased methods before introducing correct techniques, and second, require constant justification of choices. Research shows that when students predict and then test a biased sample, the contrast creates stronger retention than lectures alone. Avoid spending too much time on definitions without immediate application; students learn sampling by doing, not by memorizing terms.

What to Expect

Successful learning looks like students explaining why a biased method skews data, selecting appropriate sampling techniques for varied populations, and justifying their choices with evidence from simulations or case studies.

Watch Out for These Misconceptions

Common Misconception

During Dice Roll Sampling, watch for students who assume rolling dice many times removes bias because the sample is large.

What to Teach Instead

Pause the activity and ask groups to calculate the percentage of even numbers in their rolls versus the expected 50%. This reveals that size does not fix method flaws, prompting a shift to discussing systematic error.

Common Misconception

During Survey Station Rotation, watch for students who label any non-random method as 'bad' without considering context or population homogeneity.

What to Teach Instead

Prompt students to defend why a convenience sample might work for a homogeneous group, using examples from their station data to ground the discussion in evidence.

Common Misconception

During Case Study Debate, watch for students who argue that stratified sampling is always superior, regardless of population structure.

What to Teach Instead

Provide a case where stratified sampling introduces unnecessary complexity, then have students calculate the cost of stratification versus simple random sampling for that scenario, forcing them to weigh trade-offs.

Assessment Ideas

Discussion Prompt

After Survey Station Rotation, present students with the cafeteria queue survey scenario and ask them to compare it to one of the biased samples they encountered during the activity. Have them identify the overrepresented group and justify their answer using data from their station work.

Quick Check

During Dice Roll Sampling, pause after the first round and ask each group to predict how their sample results would change if they used a biased method instead, such as only rolling even numbers.

Exit Ticket

After Population Jar Draw, ask students to write a short paragraph explaining how stratified sampling would improve a biased sample they encountered during the activity, naming the population and strata they would use.

Extensions & Scaffolding

• Challenge early finishers to design a stratified sampling plan for a heterogeneous population, then test it using the Population Jar Draw method.
• For students who struggle, provide pre-labeled jar samples with clear over or underrepresentation to analyze before creating their own.
• Deeper exploration: Ask students to research a real poll with known bias, then redesign the sampling method and compare their proposed results to the original flawed data.

Key Vocabulary

Simple Random Sampling	A sampling method where every member of the population has an equal chance of being selected, often using random number generators.
Stratified Sampling	A sampling technique where the population is divided into subgroups (strata) based on shared characteristics, and then random samples are taken from each stratum.
Systematic Bias	A consistent error or prejudice in a study's results that arises from the sampling method or study design, leading to an unfair or inaccurate representation of the population.
Sampling Frame	A list or map of all the individuals or items within a population from which a sample is to be drawn.
Convenience Sampling	A non-probability sampling method where individuals are selected based on their easy availability and proximity, often leading to bias.