Code quality metrics that predict production bugs

Code quality metrics are measurements of code and its change history, such as complexity, duplication, test coverage, churn, defect density and change failure rate, that teams use to judge how likely code is to break and how costly it is to change. In peer-reviewed studies, relative code churn predicts defect density well, process metrics generally beat code metrics at predicting defects, complexity helps but varies by project, thinly reviewed code goes with more post-release defects, and test coverage says little about test quality on its own.

These numbers feed a wider technical debt analysis, where they help decide which parts of a codebase to fix first. For the security side of code quality, our guide to AI code security covers the flaws research keeps finding in AI-written code.

What code quality means

Code quality has two parts that show up in production: whether the code does what it should, and how safely people can change it. Readability, small functions, tests and consistent structure matter because they make the second part cheaper and the first part easier to check.

No single number captures either part. Code quality metrics are proxies, and the outcome most teams care about, bugs that reach production, only shows up after code ships. That's why most of the studies below measure post-release defects or failures (problems found after a release).

Code quality metrics teams use

Lines of code, commit counts and pull request velocity show up on dashboards too, but they describe activity, not quality. A 1,000-line file can be easy to maintain and a 100-line file can be a tangle, and fast merges can mean thin review as easily as good code.

Which code quality metrics research links to defects

Code churn and other change history

The best-known study is Nachiappan Nagappan and Thomas Ball's "Use of Relative Code Churn Measures to Predict System Defect Density" (ICSE 2005). They measured churn in Windows Server 2003 between its release and Service Pack 1, across 2,465 binaries and about 45 million lines of code. Absolute churn, such as the raw number of changed lines, was a poor predictor of defect density. Relative churn, such as changed lines divided by total lines, was highly predictive, and their churn measures separated fault-prone binaries from the rest with 89.0% accuracy (paper).

A later study compared process metrics, which describe how code was developed, with code metrics, which describe the code itself. Foyzur Rahman and Premkumar Devanbu built defect prediction models across 85 releases of 12 large open-source projects (ICSE 2013). Code metrics, despite their wide use in defect prediction research, were generally less useful than process metrics. Code metrics barely change from release to release, so models built on them keep flagging the same files, and those files turned out to be comparatively less defect-dense (paper).

Who changes the code matters too. In Windows Vista and Windows 7, Christian Bird and colleagues found that ownership measures, such as the number of low-expertise developers changing a component, were related to both pre-release faults and post-release failures (ESEC/FSE 2011) (paper).

Complexity

Nagappan, Ball and Andreas Zeller mined the post-release defect history of five Microsoft systems (ICSE 2006). Failure-prone parts of the code were statistically correlated with code complexity measures, but no single set of complexity metrics was a universally best defect predictor. Predictors built on one project could still be significant for similar projects (paper).

Complexity is a useful signal, not a universal one. It earns its place at the function level, where it tells you how many paths need tests and which functions will be hard to review. The cyclomatic complexity guide covers the formula and the limits McCabe and NIST suggest.

Test coverage

Laura Inozemtseva and Reid Holmes generated 31,000 test suites for five Java systems of up to 724,000 lines and used mutation testing to measure how many seeded faults each suite caught (ICSE 2014). Once they controlled for the number of tests in a suite, the correlation between coverage and effectiveness was low to moderate, and stronger forms of coverage (decision, modified condition) gave no better insight. Their conclusion was that coverage, "while useful for identifying under-tested parts of a program, should not be used as a quality target" (paper).

The reason is easy to see in code: a test that runs a function without asserting anything adds coverage and catches nothing. Coverage tells you what is untested, not what is tested well. In review, cubic's Enforce Pragmatic Test Coverage agent flags snapshot-only or truthy-only assertions, skipped tests, mocks that hide the behavior, and logic changes without success and failure path tests.

Duplication

Elmar Juergens and colleagues asked whether inconsistent changes to cloned code indicate faults, in a large case study of commercial and open-source systems ("Do Code Clones Matter?", ICSE 2009). Inconsistent changes to clones were very frequent, and they found a significant number of faults caused by those changes (paper). The paper calls one case particularly dangerous, the inconsistent bug fix, where a fault is fixed in one copy and not the others.

Code review coverage and participation

Shane McIntosh and colleagues studied the Qt, VTK and ITK projects (MSR 2014). Both review coverage (the share of changes that were reviewed) and review participation (how involved reviewers were) had a significant link with software quality. Components with low coverage or low participation were estimated to have up to two and five additional post-release defects, respectively (paper).

Defect density and change failure rate measure outcomes

Defect density (defects per thousand lines of code) and change failure rate measure quality after the fact. Defect density is what the churn study above set out to predict. DORA defines change fail rate as "the ratio of deployments that require immediate intervention following a deployment," likely ending in a rollback or a hotfix (DORA). Track both, because they tell you whether your leading indicators work, but neither warns you before a deploy.

What the studies don't show

These are correlations at the level of files, binaries, components or test suites, mostly from a handful of large systems. None of the studies gives a cutoff that predicts defects across projects, and the 2006 study found the best complexity predictors differ by project. Treat a rule like "coverage above 80%" or "complexity under 20" as a team decision, not a research finding.

Which code quality metrics are easy to game

Any metric that becomes a target invites gaming. DORA's guide lists setting metrics as a goal among its pitfalls: ignoring Goodhart's law and issuing broad mandates "increases the likelihood that teams will try to game the metrics."

Churn is hard to game because nobody targets it; it falls out of normal work. Keep it that way by using it to decide where to look, not as a goal.

A practical set of code quality metrics to track

Pick a handful, measure them the same way every time, and watch trends per component rather than single readings. This set covers change history, code shape, tests, review and outcomes.

1. Churn hotspots

Files changed most often in the last 90 days:

git log --since="90 days ago" --name-only --pretty=format: \
  | grep -v '^$' | sort | uniq -c | sort -rn | head -20

git log --since="90 days ago" --name-only --pretty=format: \
  | grep -v '^$' | sort | uniq -c | sort -rn | head -20

git log --since="90 days ago" --name-only --pretty=format: \
  | grep -v '^$' | sort | uniq -c | sort -rn | head -20

Lines added per file over the same window. Git records a changed line as one deletion plus one addition, so added lines approximate the added-plus-changed lines the Windows study used. Divide each total by the file's current line count to get relative churn:

git log --since="90 days ago" --numstat --pretty=format: -- src/ \
  | awk 'NF==3 {added[$3] += $1} END {for (f in added) print added[f], f}' \
  | sort -rn | head -20

git log --since="90 days ago" --numstat --pretty=format: -- src/ \
  | awk 'NF==3 {added[$3] += $1} END {for (f in added) print added[f], f}' \
  | sort -rn | head -20

git log --since="90 days ago" --numstat --pretty=format: -- src/ \
  | awk 'NF==3 {added[$3] += $1} END {for (f in added) print added[f], f}' \
  | sort -rn | head -20

2. Complexity of changed code

Gate new code in CI and keep a per-file count of functions over your limit:

flake8 --max-complexity 10 src/   # Python: reports C901
gocyclo -over 10 .                # Go: exits 1 if any function is over 10

flake8 --max-complexity 10 src/   # Python: reports C901
gocyclo -over 10 .                # Go: exits 1 if any function is over 10

flake8 --max-complexity 10 src/   # Python: reports C901
gocyclo -over 10 .                # Go: exits 1 if any function is over 10

A file that is both complex and high on the churn list is a good first place to look. Combining the two is a working heuristic, not a finding from the studies above.

3. Coverage of changed code

# Python (coverage.py)
coverage run --branch -m pytest
coverage report --fail-under=80

# Go
go test -coverprofile=cover.out ./...
go tool cover -func

# Python (coverage.py)
coverage run --branch -m pytest
coverage report --fail-under=80

# Go
go test -coverprofile=cover.out ./...
go tool cover -func

# Python (coverage.py)
coverage run --branch -m pytest
coverage report --fail-under=80

# Go
go test -coverprofile=cover.out ./...
go tool cover -func

--fail-under makes coverage.py exit with status 2 when total coverage is below the number; the 80 is an example, not a research-backed value. Where your tooling supports it, report coverage for the lines a pull request changes, which says more than a repository-wide average, and read the assertions in review.

4. Review coverage and participation

On GitHub, branch protection can require approving reviews before anyone merges into a protected branch (GitHub docs), which enforces review coverage. To count recently merged pull requests that had no review at all:

gh pr list --state merged --limit 200 --json number,reviews \
  --jq '[.[] | select(.reviews | length == 0)] | length'

gh pr list --state merged --limit 200 --json number,reviews \
  --jq '[.[] | select(.reviews | length == 0)] | length'

gh pr list --state merged --limit 200 --json number,reviews \
  --jq '[.[] | select(.reviews | length == 0)] | length'

Once coverage is enforced, watch participation: how many people review each change, and whether reviews produce comments and follow-up commits.

5. Change failure rate

Count deployments over a period and the ones that needed a rollback or hotfix right after, then divide. Write down what counts as a failure before you start, and keep the definition fixed.

6. Escaped defects by file

If your team marks bug fixes consistently, for example with a fix: commit prefix, you can count fixes per file:

git log --since="180 days ago" -i --grep='^fix' --name-only --pretty=format: \
  | grep -v '^$' | sort | uniq -c | sort -rn | head -20

git log --since="180 days ago" -i --grep='^fix' --name-only --pretty=format: \
  | grep -v '^$' | sort | uniq -c | sort -rn | head -20

git log --since="180 days ago" -i --grep='^fix' --name-only --pretty=format: \
  | grep -v '^$' | sort | uniq -c | sort -rn | head -20

Files near the top of both this list and the churn list deserve the closest review and the first refactoring time.

7. Review findings

If you use AI code review, its findings are a metric too: count the findings developers accept or fix, per changed line, and track the rate by component. cubic's Bug Bench applies this idea to AI coding models. It counts a bug when a developer upvotes a cubic review finding on AI-written code or an AI coding agent addresses it, then compares bugs per AI-attributed line with the rate across all AI-attributed lines. Across the seven ranked models, relative bug rates run from 0.65 to 1.50, where 1.00 is average. For code maintainability issues alone, the five ranked models run from 0.88 to 1.31 against the all-model rate for that bug type, and each one is statistically tied with at least one other.

Bug Bench is observational, not causal: teams choose their models, so each rate reflects the code a model was used for as well as the model. The practical point for quality metrics is that AI-written code ships bugs too, so measure it the same way you measure code people write. To see whether a change to your review process moves these numbers, see tracking bug rates before and after changes in your code review process.

How to read the numbers

• Use metrics to decide where to look, not to grade people. DORA's guide also cautions against using its metrics to compete across teams; compare a team with its own past.

• Pair leading indicators (churn, complexity, coverage of changed code, review participation) with outcomes (change failure rate, escaped defects). If the leading numbers improve and the outcomes don't, you're measuring the wrong things.

• Recheck hotspots every quarter, because change history moves with the work. Code metrics barely change between releases, and Rahman and Devanbu found that models built on them kept flagging the same files.

FAQ

What are some examples of code quality metrics?

Common examples are cyclomatic and cognitive complexity, duplication, test coverage, code churn, defect density, change failure rate, and review coverage. The first four come from the code and its history and can warn you early; defect density and change failure rate measure what already went wrong.

Which code quality metric best predicts bugs?

The studies above don't name a single winner. Relative code churn was highly predictive of defect density in Windows Server 2003, and across 12 open-source projects, process metrics were generally more useful than code metrics for defect prediction. Complexity adds signal, but the best complexity predictors differed between projects.

What is a good code coverage percentage?

Research doesn't support a single number. Inozemtseva and Holmes found only a low to moderate correlation between coverage and how many faults a suite catches once suite size is controlled, and advised against using coverage as a target. Use it to find untested code, and judge tests by their assertions.

What is a good tool for checking code quality?

No single tool covers it. Linters such as ESLint and Pylint measure complexity and flag risky patterns, coverage tools measure tests, git history gives churn, and code review catches logic errors that no metric sees. Platforms such as SonarQube report several of these together, including cyclomatic and cognitive complexity, duplication and coverage. Run them in CI on every pull request so the numbers stay current.