Statistical power and variance of effect size
Within the current clinical context, will the outcome have limited or extensive variance in the population?
Perhaps even harder to hypothesizing an effect size in terms of magnitude, researchers must make hypotheses about the nature of the distributions (variance) associated with the independent groups (between-subjects), multiple observations across time (within-subjects), or multivariate effects.
If an effect size is highly varied (wide 95% confidence intervals for proportions or large standard deviations associated with means), then more observations of the outcome will be needed to detect significant effects in the diverse and heterogeneous population.
If the effect size has little variation (narrow 95% confidence intervals for proportions or small standard deviations associated with means), then less observations of the outcome will be needed to detect statistical significance.
It is often a good practice to overestimate the variance of an effect size. This overestimation forces researchers to collect more observations of the outcome, just in case the hypothesized magnitude of the effect size is smaller than expected. These extra observations of the outcome will also deter Type II errors associated with smaller sample sizes.
The need to hypothesize the variance of effect sizes further reinforces the use of evidence-based measures of effect when conducting a priori sample size calculations. The values reported in the literature often have more credibility and validity.
If an effect size is highly varied (wide 95% confidence intervals for proportions or large standard deviations associated with means), then more observations of the outcome will be needed to detect significant effects in the diverse and heterogeneous population.
If the effect size has little variation (narrow 95% confidence intervals for proportions or small standard deviations associated with means), then less observations of the outcome will be needed to detect statistical significance.
It is often a good practice to overestimate the variance of an effect size. This overestimation forces researchers to collect more observations of the outcome, just in case the hypothesized magnitude of the effect size is smaller than expected. These extra observations of the outcome will also deter Type II errors associated with smaller sample sizes.
The need to hypothesize the variance of effect sizes further reinforces the use of evidence-based measures of effect when conducting a priori sample size calculations. The values reported in the literature often have more credibility and validity.
How much variance is hypothesized to exist in the outcome of interest?
Limited variance means that observations of the outcome variable are homogenous and do not vary.
Extended variance means that observations of the outcome variable are heterogeneous and widely dispersed.
Hire A Statistician - Statistical Consulting for Students
DO YOU NEED TO HIRE A STATISTICIAN?
Eric Heidel, Ph.D. will provide the following statistical consulting services for undergraduate and graduate students at $75/hour. Secure checkout is available with Stripe, Venmo, Zelle, or PayPal.
- Statistical Analysis
- Research Design
- Sample Size Calculations
- Diagnostic Testing and Epidemiological Calculations
- Survey Design and Psychometrics