significance criterion

(noun)

a statement of how unlikely a positive result must be, if the null hypothesis of no effect is true, for the null hypothesis to be rejected

Related Terms

Examples of significance criterion in the following topics:

  • Statistical Power

    • The power of the test is the probability that the test will find a statistically significant difference between men and women, as a function of the size of the true difference between those two populations.
    • The Statistical Significance Criterion Used in the Test: A significance criterion is a statement of how unlikely a positive result must be, if the null hypothesis of no effect is true, for the null hypothesis to be rejected.
    • One easy way to increase the power of a test is to carry out a less conservative test by using a larger significance criterion, for example 0.10 instead of 0.05.
    • Let's say we look for a significance criterion of 0.05.
    • Discuss statistical power as it relates to significance testing and breakdown the factors that influence it.

  • Introduction to Multiple Regression

    • Test the difference between a complete and reduced model for significance
    • We begin by presenting the formula for testing the significance of the contribution of a set of variables.
    • We will then show how special cases of this formula can be used to test the significance of R2 as well as to test the significance of the unique contribution of individual variables.
    • If the F is significant, then it can be concluded that the variables excluded in the reduced set contribute to the prediction of the criterion variable independently of the other variables.
    • The significance test of the variance explained uniquely by a variable is identical to a significance test of the regression coefficient for that variable.

  • How Fisher Used the Chi-Squared Test

    • Fisher's exact test is a statistical significance test used in the analysis of contingency tables.
    • Fisher's exact test is one of a class of exact tests, so called because the significance of the deviation from a null hypothesis can be calculated exactly, rather than relying on an approximation that becomes exact in the limit as the sample size grows to infinity.
    • It is used to examine the significance of the association (contingency) between the two kinds of classification.
    • In Fisher's original example, one criterion of classification could be whether milk or tea was put in the cup first, and the other could be whether Dr.
    • However, the significance value it provides is only an approximation, because the sampling distribution of the test statistic that is calculated is only approximately equal to the theoretical chi-squared distribution.

  • Stepwise Regression

    • Usually, this takes the form of a sequence of $F$-tests; however, other techniques are possible, such as $t$-tests, adjusted $R$-square, Akaike information criterion, Bayesian information criterion, Mallows's $C_p$, or false discovery rate.
    • Forward selection involves starting with no variables in the model, testing the addition of each variable using a chosen model comparison criterion, adding the variable (if any) that improves the model the most, and repeating this process until none improves the model.
    • This problem can be mitigated if the criterion for adding (or deleting) a variable is stiff enough.
    • The key line in the sand is at what can be thought of as the Bonferroni point: namely how significant the best spurious variable should be based on chance alone.
    • A way to test for errors in models created by stepwise regression is to not rely on the model's $F$-statistic, significance, or multiple-r, but instead assess the model against a set of data that was not used to create the model.

  • An objective measure for finding the best line

    • Perhaps our criterion could minimize the sum of the residual magnitudes:
    • The line that minimizes this least squares criterion is represented as the solid line in Figure 7.12.
    • The following are three possible reasons to choose Criterion (second equation) over Criterion (first equation):
    • Computing the line based on Criterion (second equation) is much easier by hand and in most statistical software.
    • The first two reasons are largely for tradition and convenience; the last reason explains why Criterion (7.10) is typically most helpful.

  • Inferential Statistics for b and r

    • This section shows how to conduct significance tests and compute confidence intervals for the regression slope and Pearson's correlation.
    • The column X has the values of the predictor variable and the column Y has the values of the criterion variable.
    • The formula for a significance test of Pearson's correlation is shown below:

  • Exercises

    • What criterion is used for deciding which regression line fits best?
    • Is this correlation statistically significant at the .05 level?
    • If the standard error of b is .4, is the slope statistically significant at the .05 level?
    • True/false: If the slope of a simple linear regression line is statistically significant, then the correlation will also always be significant.
    • Is this correlation statistically significant at the .01 level?

  • Elements of a Hypothesis Test

    • The critical region was the single case of 4 successes of 4 possible based on a conventional probability criterion ($< 5\%$; $\frac{1}{70} \approx 1.4\%$).
    • The lady correctly identified every cup, which would be considered a statistically significant result.
    • In statistics, a result is called statistically significant if it has been predicted as unlikely to have occurred by chance alone, according to a pre-determined threshold probability—the significance level.
    • Select a significance level ($\alpha$), a probability threshold below which the null hypothesis will be rejected.
    • The decision rule is to reject the null hypothesis if and only if the $p$-value is less than the significance level (the selected probability) threshold.

  • Elements of a Designed Study

    • If $n$ independent comparisons are performed, the experiment-wide significance level $\bar { \alpha }$, also termed FWER for familywise error rate, is given by:
    • Another procedure is the Holm–Bonferroni method, which uniformly delivers more power than the simple Bonferroni correction by testing only the most extreme $p$-value ($i=1$) against the strictest criterion, and the others ($i>1$) against progressively less strict criteria.
    • Typically these methods require a significant ANOVA/Tukey's range test before proceeding to multiple comparisons.
    • Multiple comparison procedures are commonly used in an analysis of variance after obtaining a significant omnibus test result, like the ANOVA $F$-test.
    • The significant ANOVA result suggests rejecting the global null hypothesis $H_0$ that the means are the same across the groups being compared.

  • Glossary

    • In general, the criterion variable is the dependent variable.
    • In multiple regression, the criterion is predicted from two or more predictor variables.
    • Thus, if 10 significance tests were each conducted at 0.05 significance level, then the per-comparison error rate would be 0.05.
    • In the following example, the criterion (Y) is predicted by X, X2 and, X3.
    • Common significance levels are 0.05 and 0.01.