Testing Bias

Examples of Testing Bias in the following topics:

• Standardized Tests

  • A standardized test is a test that is administered and scored in a consistent manner.
  • A standardized test is a test that is administered and scored in a consistent manner.
  • They are designed so that the questions, conditions for administering, scoring procedures, and interpretations are purportedly without bias.
  • Finally, critics have expressed concern that standardized tests may create testing bias.
  • Testing bias occurs when a test systematically favors one group over another, even though both groups are equal on the trait the test measures.

• Distorting the Truth with Descriptive Statistics

  • Bias is another common distortion in the field of descriptive statistics.
  • The following are examples of statistical bias.
  • Spectrum bias arises from evaluating diagnostic tests on biased patient samples, leading to an overestimate of the sensitivity and specificity of the test.
  • Funding bias may lead to selection of outcomes, test samples, or test procedures that favor a study's financial sponsor.
  • Analytical bias arises due to the way that the results are evaluated.

• Controversies in Intelligence and Standardized Testing

  • Intelligence tests and standardized tests face criticism for their uses and applications in society.
  • Intelligence tests (such as IQ tests) have always been controversial; critics claim that they measure factors other than intelligence.
  • They also cast doubt on the validity of IQ tests and whether IQ tests actually measure what they claim to measure—intelligence.
  • Questions of bias raise similar questions to the questions around whether intelligence tests should be used to predict social outcomes.
  • IQ tests are often criticized for being culturally biased.

• Data Snooping: Testing Hypotheses Once You've Seen the Data

  • Testing hypothesis once you've seen the data may result in inaccurate conclusions.
  • Sometimes, people deliberately test hypotheses once they've seen the data.
  • Data-snooping bias is a form of statistical bias that arises from this misuse of statistics.
  • Although data-snooping bias can occur in any field that uses data mining, it is of particular concern in finance and medical research, which both heavily use data mining.
  • This table depicts the difference types of errors in significance testing.

• Creating a test statistic for one-way tables

  • While some sampling variation is expected, we would expect the sample proportions to be fairly similar to the population proportions if there is no bias on juries.
  • We need to test whether the differences are strong enough to provide convincing evidence that the jurors are not a random sample.
  • H 0 : The jurors are a random sample, i.e. there is no racial bias in who serves on a jury, and the observed counts reflect natural sampling fluctuation.
  • H A : The jurors are not randomly sampled, i.e. there is racial bias in juror selection.

• Gender Bias in the Classroom

  • Gender-based achievement gaps suggest the existence of gender bias in the classroom.
  • Teachers may reinforce gender bias simply by drawing distinctions between boys and girls.
  • Although girls tend to stay in school longer, have better attendance records, and earn better report card grades, boys outscore girls on most high-stakes tests, including both the math and verbal sections of the SAT.
  • Men also outscore women on standardized tests for graduate school, law school, and medical school.
  • If test score gaps are evidence of gender bias, where does that gender bias come from?

• t-Test for Two Samples: Independent and Overlapping

  • The two sample t-test is used to compare the means of two independent samples.
  • In the latter case the estimated t-statistic must either be tested with modified degrees of freedom, or it can be tested against different critical values.
  • The two-sample t-test is probably the most widely used (and misused) statistical test.
  • If, for any reason, one is forced to use haphazard rather than probability sampling, then every effort must be made to minimize selection bias.
  • Paired t-tests are a form of blocking, and have greater power than unpaired tests when the paired units are similar with respect to "noise factors" that are independent of membership in the two groups being compared.

• Finding a p-value for a chi-square distribution

  • In Section 6.3.2, we identified a new test statistic ($X^2$ ) within the context of assessing whether there was evidence of racial bias in how jurors were sampled.
  • The null hypothesis represented the claim that jurors were randomly sampled and there was no racial bias.
  • The alternative hypothesis was that there was racial bias in how the jurors were sampled.
  • In other words, the data do not provide convincing evidence of racial bias in the juror selection.
  • Failing to check conditions may affect the test's error rates.

• Exercises

  • What is the reliability if the true score variance is 80 and the test score variance is 100?
  • What is the effect of test length on the reliability of a test?
  • What is the theoretical maximum correlation of a test with a criterion if the test has a reliability of .81?
  • What type of sampling bias is likely to occur?
  • Give an example of survivorship bias not presented in this text.

• Sex Bias in Graduate Admissions

  • This has not negated the fact that gender bias exists in higher education.
  • Representatives of the companies that publish these tests have hypothesized that greater number of female applicants taking these tests pull down women's average scores.
  • Controlling for the number of people taking the test does not account for the scoring gap.
  • In fact, most departments had a small but statistically significant bias in favor of women.
  • Therefore, the admission bias seemed to stem from courses previously taken.