Skip to main content
Back to all posts

Automated vs Manual Accessibility Testing: The Data

Benjamin Morton

Two numbers dominate the automated-versus-manual debate, and they sound like they contradict each other. Automated tools catch 57% of accessibility issues by volume (Deque, 2021). Automated tools reliably detect only 13% of WCAG success criteria (Accessible.org, 2025). Both are correct, and understanding why is the whole game.

The gap between them is not a disagreement; it's a difference in what you're counting. One measures the volume of bugs found, the other measures the breadth of rules covered. Once you see that, the "should I automate or test by hand" question dissolves into a much better one: which layer catches which kind of problem, and how do you split the work? This post walks through the data on both sides and lands on a practical split. It builds on our developer's guide to web accessibility testing.

TL;DR: Automated tools catch about 57% of accessibility issues by volume but only 13% of WCAG 2.2 AA criteria with high accuracy, with another 42% undetectable by any scanner (Deque, 2021; Accessible.org, 2025). The high-volume bugs are the auto-detectable ones, so automation clears the bulk cheaply in CI. Keyboard operability, focus order, and alt-text quality need a human. The data says combine them, not choose.

How Much Do Automated Accessibility Tools Actually Catch?

By volume, more than most people expect. A Deque study of over 2,000 audits, covering more than 13,000 pages and roughly 300,000 issues, found that automated testing completely covered 57% of accessibility issues (Deque, 2021). That was notably higher than the "widely-accepted" prior benchmark of 20-30% coverage the study set out to test.

That 57% is not a fluke of one dataset. It lines up with what WebAIM finds in the wild: in 2026, the six most common failure types accounted for 96% of all detected errors across the top million home pages (WebAIM Million, 2026). Those six (low-contrast text, missing alt text, missing form labels, empty links, empty buttons, and a missing document language) are exactly the failures a scanner detects reliably. The high-frequency bugs and the auto-detectable bugs are largely the same set, which is why automation clears so much volume.

Citation capsule: A Deque study of more than 2,000 audits, 13,000+ pages, and roughly 300,000 issues found automated testing completely covered 57% of accessibility issues by volume (Deque, 2021), well above the prior 20-30% benchmark. This holds because the six most common failure types, which make up 96% of detected errors (WebAIM Million, 2026), are precisely the ones scanners catch reliably.

Why "57%" and "13%" Are Both Correct

They measure different things. The 57% is a share of issues by volume; the 13% is a share of WCAG success criteria. Count the bugs and automation looks strong. Count the rules and it looks weak. Neither is wrong, because a single high-frequency rule like colour contrast produces enormous bug volume while still being just one criterion.

Here's the criteria view. Of the 55 WCAG 2.2 AA success criteria, only a small fraction can be checked with confidence by a machine (Accessible.org, 2025):

Automated detectability

Share of WCAG 2.2 AA criteria

Count

Detectable with high accuracy

13%

7 of 55

Partially detectable (misses parts)

45%

25 of 55

Not detectable at all

42%

23 of 55

So a scanner reliably handles 7 criteria, gives you a partial read on 25 more, and is blind to 23. That's the source of the confusion: a tool can catch most of the bug volume (57%) while still leaving the majority of the rulebook (87% of criteria at less than high accuracy) needing human eyes. Both numbers describe the same tool honestly.

What Only Manual Testing Catches

The 42% of criteria no scanner detects are the ones that need human judgment (Accessible.org, 2025). These aren't edge cases; they're core to whether a site actually works. The W3C is blunt about it: "no tool alone can determine if a site meets accessibility standards. Knowledgeable human evaluation is required to determine if a site is accessible" (W3C-WAI, 2026).

What falls in that gap? Keyboard operability and focus order, whether alt text is actually meaningful rather than merely present, reading order, error-message quality, captions accuracy, and information conveyed by colour or sensory characteristics. A tool can confirm an image has an alt attribute; only a person can tell you alt="image" is useless. That's the difference between the letter of a rule and the experience it protects.

What we see: Across the scans a11yFlow runs, the violations that fill a report are the six WebAIM categories, and those are exactly the ones the engine flags and explains automatically. What never shows up in an automated report is the failure a keyboard user hits first: focus landing nowhere on a custom dropdown or modal. That gap is human-only, every time.

Citation capsule: The W3C-WAI states plainly that "no tool alone can determine if a site meets accessibility standards. Knowledgeable human evaluation is required" (W3C-WAI, 2026). This is because 42% of WCAG 2.2 AA criteria are not machine-detectable at all (Accessible.org, 2025), covering keyboard operability, focus order, alt-text quality, and reading order, all of which require human evaluation.

There's a subtler cost to over-relying on tools: false positives. Because 45% of criteria are only partially detectable, scanners sometimes flag compliant text as failing or miss a real barrier entirely (Accessible.org, 2025). A green scan is not proof of accessibility, which is exactly the false promise that accessibility overlays sell.

What Automated and Manual Testing Cost

Cost is where the "why not just do everything manually" argument breaks down, and where automation earns its place. Building accessibility in reactively adds around 15% to a project's cost; shifting it left into development drops that to 5% or less, roughly $25,000 instead of $75,000 on a $500,000 project (Deque, 2024). The earlier you catch an issue, the cheaper it is.

Deque's phase-by-phase numbers make the curve concrete: fixing an issue in development rather than QA saves around $350, catching it in QA rather than production saves about $450, and a defect that reaches production can cost $800 or more (Deque, 2024). Automated tests are the cheapest possible catch point, running on every commit at near-zero marginal cost.

Manual testing is not free, which is the honest counterweight. A professional manual audit is a fully manual evaluation that takes several hours, commonly priced at $100 to $250 per standard page (Accessible.org, 2026). That expense is exactly why you don't want humans re-checking colour contrast on every build. Spend the automation budget on the high-volume, detectable failures, and spend the scarce manual hours on the judgment calls only people can make.

So Which Should You Use? The Data Says Both

Neither alone is sufficient, and the numbers explain why with no ambiguity. Automation catches 57% of the volume for almost nothing per run but misses 42% of the criteria entirely. Manual testing catches the judgment-dependent failures but costs $100-250 per page, too much to run on every commit. Use each where its economics win.

Dimension

Automated

Manual

Catches

~57% of issues by volume

The 42% of criteria no tool detects

Best for

High-volume, measurable failures (contrast, alt, labels)

Keyboard, focus, alt-text quality, cognitive load

Cost

Near-zero per run in CI

$100 to $250 per standard page

Cadence

Every commit

Key templates, periodically

A workable split, straight from the data:

  • Automate the high-volume, detectable failures in CI. The six WebAIM failure types are auto-detectable and make up 96% of errors (WebAIM Million, 2026). Gate every pull request on them with axe-core. Our GitHub Actions guide shows the setup, and the common WCAG violations guide covers the fixes.
  • Reserve manual testing for the 42%. Keyboard journeys, focus management, alt-text quality, and screen-reader flow, tested by a person on key page templates, not every route on every build.
  • Treat the 57% as a floor, not a finish line. Automation frees your human reviewers to spend their expensive hours on the 43% that actually needs them, rather than re-checking contrast.

Frequently Asked Questions

Is automated or manual accessibility testing better?

Neither; they cover different failures. Automated tools catch about 57% of issues by volume (Deque, 2021) but only 13% of WCAG 2.2 AA criteria with high accuracy (Accessible.org, 2025). The W3C states no tool alone can determine accessibility. Combine automated CI checks with manual testing.

Why do automated tools only catch 57% of issues?

Because 42% of WCAG 2.2 AA success criteria are not machine-detectable, and another 45% only partially (Accessible.org, 2025). Scanners handle measurable rules like contrast and missing alt attributes but can't judge keyboard flow, focus order, or whether alt text is meaningful. The 57% figure counts issue volume, not rule coverage.

Can I rely on an accessibility scanner alone for compliance?

No. A passing scan proves only that the auto-detectable rules pass, roughly 13% of WCAG 2.2 AA criteria at high accuracy (Accessible.org, 2025). The W3C-WAI states plainly that knowledgeable human evaluation is required to determine if a site is accessible (W3C-WAI, 2026). Scanners are necessary but not sufficient.

How much does manual accessibility testing cost?

A professional manual audit is a fully manual evaluation that takes several hours, commonly priced at $100 to $250 per standard page (Accessible.org, 2026). That cost is why automation should handle the high-volume, repeatable checks, leaving the scarce manual hours for judgment calls a tool can't make.

Does automated testing save money?

Yes, by catching issues early. Reactive accessibility work adds about 15% to project cost, while shifting testing left drops it to 5% or less (Deque, 2024). Fixing an issue in development instead of production saves hundreds of dollars per defect, and automated CI checks are the cheapest catch point.

The Bottom Line

The automated-versus-manual debate is a false choice built on a misread of two numbers. Automation catches 57% of issues by volume but only 13% of WCAG 2.2 AA criteria with high accuracy (Deque, 2021; Accessible.org, 2025). Those aren't in conflict; they describe a tool that clears bug volume cheaply and leaves judgment to humans.

So build the split the data points to. Gate the high-volume, detectable failures in CI with axe-core, and reserve manual testing for keyboard, focus, and content-quality checks no scanner can make, exactly as the W3C-WAI recommends (W3C-WAI, 2026). Start with the automated layer in our GitHub Actions guide, see the React testing guide for a component-level setup, and read the developer's guide to web accessibility testing for the full workflow across both layers.

Sources

PreviousNext.js Accessibility Best Practices (2026)
NextADA Website Lawsuits: The Statistics Developers Need in 2026