Cohen's Kappa Calculator

StatisticsLast updated: August 22, 2026

Cohen's kappa (κ) measures how far two raters who classify the same cases independently agree, after subtracting the agreement expected by chance. Coder reliability in content analysis, diagnostic agreement between two clinicians, and labeler consistency in machine learning are all reported with this coefficient.

Raw percent agreement is misleading: even two raters deciding at random reach a certain level of agreement on their own. Kappa removes that chance component. From your agreement matrix this tool computes κ, the observed agreement pₒ and the chance agreement pₑ, the asymptotic standard error, the z statistic, the p-value, the confidence interval, and κ_max, the highest value reachable while the marginals stay fixed; it then reads the result on the Landis-Koch scale.

Enter a square matrix: matrix[i][j] is the number of cases rater 1 assigned to category i and rater 2 to category j. The diagonal holds the exact agreements. Rows and columns must list the categories in the SAME order.
Separate them with commas. If left empty, "Category 1, Category 2, …" is used.
The probability that the interval covers the true parameter.
Sets the false-positive risk. In the social sciences the usual choice is 5%.

Cohen's Kappa Formulas

Observed agreement:  pₒ = Σᵢ pᵢᵢ  (sum of the diagonal proportions)
Chance agreement:    pₑ = Σᵢ pᵢ. · p.ᵢ  (products of the marginal proportions)
Cohen's kappa:       κ = (pₒ − pₑ) / (1 − pₑ)
Confidence interval: κ ± z_{1−α/2} · SE(κ)
Test statistic:      z = κ / SE₀,  SE₀ = the standard error under H₀: κ = 0
Highest attainable:  pₒ,max = Σᵢ min(pᵢ., p.ᵢ),  κ_max = (pₒ,max − pₑ) / (1 − pₑ)
Interpretation:      < 0 poor · 0–0.20 slight · 0.21–0.40 fair
                     0.41–0.60 moderate · 0.61–0.80 substantial · 0.81–1.00 almost perfect

The standard error uses the asymptotic variance formula of Fleiss, Cohen, and Everitt (1969) and matches the value in the SPSS Crosstabs > Kappa output. The Landis-Koch thresholds are conventional and arbitrary; they shift with the clinical stakes of the field.

How to Calculate

  1. Lay the two raters' classifications out in a square agreement matrix: rows are rater 1's categories, columns are rater 2's.
  2. Type each row of the matrix on its own line, and make sure the categories appear in the same order across rows and columns.
  3. Enter the category names separated by commas; the matrix and the charts are labeled with them.
  4. Choose the confidence and significance levels, then read κ together with its confidence interval.
  5. Compare the observed agreement with kappa: a large gap shows how misleading percent agreement can be.
  6. Check κ_max; if it is clearly below 1, a low κ may come from marginal imbalance rather than genuine disagreement.

Worked Examples

Diagnostic agreement between two clinicians (2×2)

The two clinicians reached the same decision in 85 of 100 cases; observed agreement is 85.00% and chance agreement 50.00%. With κ = 0.7000 (95% CI: 0.5607 – 0.8393), z = 7.0353, and p < 0.001, the agreement is significantly better than chance. On the Landis-Koch scale this is substantial agreement, and κ_max = 0.9000.

Cohen kappa (κ): 0.7000 · 95% confidence interval: 0.5607 – 0.8393 · Observed agreement (pₒ): 85.00%

Three-category content coding (3×3)

The two coders chose the same category for 79 of 100 texts: pₒ = 79.00% and pₑ = 33.80%. This gives κ = 0.6828 (95% CI: 0.5617 – 0.8039), substantial agreement, with p < 0.001. Because κ_max = 0.9698, the two coders use the categories with similar frequencies.

Cohen kappa (κ): 0.6828 · 95% confidence interval: 0.5617 – 0.8039 · Observed agreement (pₒ): 79.00%

The kappa paradox: high percent agreement, low kappa

Observed agreement is again 85.00%, but because one category dominates, chance agreement rises to 78.00% and κ drops to 0.3182 (95% CI: 0.0566 – 0.5798). On the Landis-Koch scale that is only fair agreement, and with κ_max = 0.7727 the marginal imbalance is also pulling the result down.

Cohen kappa (κ): 0.3182 · 95% confidence interval: 0.0566 – 0.5798 · Observed agreement (pₒ): 85.00%

Frequently Asked Questions

Why use kappa instead of percent agreement?
Percent agreement includes the agreement that arises by chance. Two coders deciding by coin flip still agree about 50% of the time on a two-category task, and if one category dominates the figure can climb above 80%. Kappa subtracts the chance agreement from the observed agreement and divides the remainder by the maximum agreement beyond chance, removing that illusion.
What is the kappa paradox?
When one category is very common, chance agreement (pₑ) becomes very high and the denominator shrinks, so kappa can stay below 0.20 even with an observed agreement of 90%. It is common in diagnostic studies of rare events. In such cases you should report the observed agreement, the marginal distributions, and κ_max alongside kappa.
What is κ_max good for?
κ_max is the highest kappa attainable given the row and column marginals. If the raters use the categories with different frequencies (one keeps saying "positive" while the other keeps saying "neutral"), perfect agreement is mathematically impossible and κ_max falls below 1. Only the κ / κ_max ratio tells you whether a low kappa comes from genuine disagreement or from marginal imbalance.
How high should kappa be?
The 1977 Landis and Koch scale is widely used: 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81 and above almost perfect. These thresholds are nevertheless arbitrary. In publication practice, 0.80 is the usual floor in content analysis and 0.60 in clinical diagnostic studies. The heavier the consequences of the decision, the higher the kappa you should expect.
What if there are more than two raters?
Cohen's kappa is for two raters only. With three or more, use Fleiss kappa, which also works when different groups of raters assess the same cases. For ordinal or continuous measurements use the intraclass correlation coefficient (ICC), and in content analysis Krippendorff's alpha is often preferred.
Which kappa should I use for ordinal categories?
If the categories are ordered (mild – moderate – severe, say), simple kappa penalizes a mild-versus-severe disagreement exactly as much as a mild-versus-moderate one, which throws information away. Use weighted kappa instead: linear weights give partial credit to neighboring categories, while quadratic weights punish large departures more heavily and, with two categories, give the same result as the ICC.
Will SPSS and R give the same result?
Yes. In SPSS, ticking Crosstabs > Statistics > Kappa produces the same κ, asymptotic standard error, z ("Approx. T"), and p-value. In R, cohen.kappa() from the psych package or kappa2() from the irr package returns the same values, and psych also reports weighted kappa. Small differences can arise only from the confidence interval method.