ANOVA vs T-Test — Choosing the Right Test for Group Comparisons
By CalcMulti Editorial Team··9 min read
Both the t-test and ANOVA (Analysis of Variance) compare means across groups to determine whether differences are statistically significant. The key rule: use a t-test when comparing exactly two groups; use ANOVA when comparing three or more groups.
A common mistake is running multiple t-tests to compare several groups — e.g., Group A vs B, A vs C, and B vs C separately. This inflates the Type I error rate (false positive rate) dramatically. ANOVA tests all groups simultaneously with the correct α, then post-hoc tests identify which specific groups differ.
Side-by-Side Comparison
| Property | T-Test | ANOVA |
|---|---|---|
| Number of groups | 2 groups only | 2 or more groups (designed for 3+) |
| Test statistic | t-statistic | F-statistic (ratio of between-group to within-group variance) |
| Distribution | t-distribution (df = n−2 or n₁+n₂−2) | F-distribution (df₁ = k−1, df₂ = N−k) |
| Null hypothesis | H₀: μ₁ = μ₂ (two means are equal) | H₀: μ₁ = μ₂ = μ₃ = ... = μₖ (all means equal) |
| What a significant result means | The two group means differ | At least one group mean differs from at least one other |
| Follow-up needed? | No — the t-test identifies which groups differ | Yes — post-hoc tests needed to identify which pairs differ |
| Type I error control | Controlled at α for one comparison | Controlled at α for the omnibus test |
| Relationship | F = t² for two groups (one-way ANOVA = t-test when k=2) | Generalisation of t-test to k groups |
| Assumes equal variances? | Student's t (yes); Welch's t (no) | Standard ANOVA (yes); Welch ANOVA (no) |
| Effect size measure | Cohen's d | η² (eta-squared) or ω² |
Why Not Just Run Multiple T-Tests?
Suppose you have 3 groups (A, B, C) and want to know if any differ in mean performance. You might think: run t-tests for A vs B, A vs C, and B vs C separately, each at α = 0.05. What is wrong with this?
The problem: each test has a 5% chance of a false positive. With 3 independent tests, the probability of at least one false positive is: 1 − (1 − 0.05)³ = 1 − 0.857 = 14.3% — nearly three times the intended 5%.
With 4 groups (6 pairwise tests): 1 − 0.95⁶ = 26.5% false positive risk. With 5 groups (10 tests): 40%. With 10 groups (45 tests): 90%.
ANOVA tests all groups in a single omnibus F-test that controls the Type I error rate at α across the entire family of comparisons. Only after a significant F-test do you proceed to post-hoc pairwise tests — and those tests include their own multiple comparison correction.
Understanding the F-Statistic
The F-statistic is the ratio of between-group variance to within-group variance: F = MS_Between / MS_Within.
MS_Between (Mean Square Between): how much the group means vary from the overall mean — the signal you are trying to detect. If the groups have very different means, MS_Between is large.
MS_Within (Mean Square Within, or residual): how much individual observations vary within their groups — the background noise. This is what you expect even if all groups have the same true mean.
A large F means the groups differ more than you would expect by chance. Under H₀ (all means equal), F follows an F-distribution. The p-value is the probability of observing an F this large under H₀.
Example: Three diets are tested on weight loss (kg) in 30 people (10 per diet). F = 6.2, p = 0.006. Reject H₀ — at least one diet produces significantly different average weight loss.
Post-Hoc Tests After ANOVA
A significant ANOVA only tells you that at least one group differs — not which ones. Post-hoc tests (also called multiple comparison procedures) perform all pairwise comparisons while controlling the family-wise error rate.
Common post-hoc tests and when to use them:
| Post-Hoc Test | Best For | Conservative? |
|---|---|---|
| Tukey HSD | All pairwise comparisons when group sizes are equal | Moderate |
| Tukey-Kramer (Tukey's b) | Pairwise comparisons with unequal group sizes | Moderate |
| Bonferroni | Any number of comparisons; straightforward interpretation | Very conservative (reduces power) |
| Scheffé | All possible contrasts (not just pairwise) | Most conservative; most flexible |
| Games-Howell | Unequal variances between groups | Moderate; does not assume equal variances |
| Dunnett's | Comparing all treatment groups to a single control | Less conservative (tailored for control comparisons) |
Assumptions for T-Test and ANOVA
Both tests share three core assumptions:
1. Independence: observations within and between groups are independent.
2. Normality: the outcome variable is approximately normally distributed within each group. Robust to violations when n is large (CLT); problematic for small, clearly non-normal samples.
3. Homoscedasticity (equal variances): all groups have approximately the same population variance. T-test: use Welch's t-test if variances are unequal. ANOVA: use Welch ANOVA (Brown-Forsythe or Welch correction) if Levene's test shows unequal variances.
When assumptions are violated: for non-normal data with small samples, use non-parametric alternatives — Mann-Whitney U test instead of t-test; Kruskal-Wallis test instead of one-way ANOVA.
Types of ANOVA
One-way ANOVA: one factor (categorical variable) with 3+ levels. "Does diet type (low-carb, Mediterranean, vegan) affect weight loss?"
Two-way ANOVA: two factors simultaneously. "Does weight loss depend on diet type AND exercise level, and do these factors interact?"
Repeated-measures ANOVA: same subjects measured at multiple time points. Equivalent of a paired t-test extended to 3+ time points. Requires a different correlation structure (sphericity assumption).
MANOVA (Multivariate ANOVA): multiple outcome variables simultaneously. "Do groups differ on a combination of weight loss, blood pressure, and cholesterol?"
ANCOVA (Analysis of Covariance): ANOVA with one or more continuous covariates added to control for confounding variables.
Summary
Use a t-test when comparing exactly two group means. Use ANOVA when comparing three or more groups. Never run multiple t-tests to compare several groups — this inflates the false positive rate. After a significant ANOVA, use post-hoc tests to identify which specific groups differ.
- For 2 groups: use Welch's t-test (does not assume equal variances) as the default
- For 3+ groups: use one-way ANOVA, then Tukey HSD (equal n) or Games-Howell (unequal variances) for pairwise follow-up
- Multiple t-tests inflate Type I error to unacceptable levels when k ≥ 3
- When F = t² mathematically — ANOVA and t-test give identical results for k = 2
- Non-normal small samples: use Mann-Whitney U (2 groups) or Kruskal-Wallis (3+ groups)
Related Calculators
One-way ANOVA with F-statistic and p-value
T-Test CalculatorTwo-group comparison with full output
T-Test vs Z-TestChoosing between t and z tests
Effect Size CalculatorCohen's d and η² effect sizes
Statistical Significance ExplainedWhat p-values mean
Statistics HubAll statistics calculators & guides
Frequently Asked Questions
Educational use only. Content is based on publicly documented mathematical formulas and reviewed for accuracy by the CalcMulti Editorial Team. Last updated: February 2026.