Automated vs. Manual Accessibility Testing: When You Need Both

TL;DR: Automated accessibility tools reliably catch only about 30 to 40 percent of WCAG issues — a directional figure most testing teams agree on — because many success criteria require human judgment to evaluate. Scanners excel at finding missing alt attributes, contrast failures, unlabeled forms, and broken ARIA at scale. Keyboard operability, screen reader experience, cognitive load, and contextual correctness require a person. The right approach for a government website is both: continuous automated monitoring to catch regressions across every page, plus periodic manual testing to validate the experience software can’t see.

When a government agency starts working toward ADA Title II and WCAG 2.2 AA compliance, the first instinct is usually to run a free automated scanner, see a score, and treat that number as the finish line. It isn’t. An automated tool that reports “no errors” on your homepage is telling you something useful but dangerously incomplete. The hard truth of accessibility testing is that no software can fully determine whether a website is accessible — and no amount of manual testing can keep pace with a site that publishes new content every day. You need both, and you need to understand what each is actually good for.

This article breaks down exactly what automated tools catch, what only humans can catch, and how to combine the two into a program that’s both thorough and sustainable.

Why No Single Method Is Enough

WCAG 2.2 Level AA has roughly 55 success criteria. Some are purely mechanical: an image either has an alt attribute or it doesn’t. Some are purely subjective: alt text either conveys the image’s meaning or it doesn’t. Most testing methods are strong on one end of that spectrum and weak on the other.

The widely cited industry estimate is that automated tools detect 30 to 40 percent of WCAG issues. This number comes up repeatedly in accessibility research and vendor analyses, and while the exact figure varies by tool, site, and how you count, the consensus is clear and consistent: automated testing finds a minority of issues. Treat it as directional. The practical takeaway doesn’t change — most accessibility problems are invisible to scanners.

That’s not a knock on automated tools. It’s a statement about the nature of accessibility. A large share of WCAG is about whether content makes sense to a human using assistive technology, and “makes sense” is not something code can evaluate.

What Automated Testing Catches Well

Automated scanners — axe, WAVE, Lighthouse, Pa11y, Govzu, and others — parse the rendered DOM and accessibility tree and apply rules. They’re genuinely excellent within their domain, and you should absolutely use them. Here’s where they shine:

• Missing alternative text. Images with no alt attribute. (Scanners can’t tell you if existing alt text is good — see below — but they reliably flag what’s missing. See how to fix missing alt text.)
• Color contrast failures. Tools compute the contrast ratio between text and background and flag anything below 4.5:1 (SC 1.4.3) or 3:1 for large text. This is pure math, and scanners do it perfectly for standard text on solid backgrounds.
• Missing form labels. Inputs without an associated <label>, aria-label, or aria-labelledby.
• Document structure issues. Missing page lang attribute, missing page title, skipped heading levels, duplicate IDs.
• Empty links and buttons. Interactive elements with no accessible name.
• Broken ARIA. Invalid ARIA roles, references to nonexistent IDs, required ARIA attributes that are missing.
• Parsing and HTML validity issues that affect assistive technology.

These are exactly the failures that pile up on government sites: a content editor uploads a photo without alt text, a designer picks a too-light gray for a footer, a form gets built without labels. Crucially, scanners do this at scale and continuously — you can check thousands of pages every day, which is the one thing manual testing fundamentally cannot do.

What Automated Testing Misses

Here’s where the other 60-plus percent lives. These criteria are not edge cases — they include some of the most consequential barriers on any site.

Keyboard operability

A scanner can confirm an element is focusable. It cannot confirm that the keyboard experience works:

• Can you actually reach every interactive element with Tab, in a logical order?
• Is there a visible focus indicator so a keyboard user knows where they are? (SC 2.4.7) A scanner can’t reliably judge whether a custom focus style is perceivable.
• Can you operate custom widgets — menus, sliders, date pickers, accordions — with the expected keys?
• Can you escape a modal dialog, or does focus get trapped? (SC 2.1.2)
• Does a dropdown menu that opens on hover also work on keyboard focus?

These require a person to put down the mouse and drive the site by keyboard alone. Our guide to keyboard accessibility walks through the test sequence.

Screen reader experience

This is the biggest blind spot. An automated tool verifies that alt text exists; only a human using NVDA, JAWS, or VoiceOver can judge whether the page actually makes sense when read aloud:

• Does the alt text convey the image’s purpose in context, or is it alt="image123.jpg" technically present but useless?
• Are headings and landmarks structured so a screen reader user can navigate by them?
• Is link text meaningful out of context? A page of “click here” links passes most scanners and fails real users.
• Are dynamic updates announced? When a form shows an error, does the screen reader say so?
• Is the reading order logical, or does CSS visually reorder content in a way that breaks the spoken sequence?

Cognitive accessibility and content quality

• Are instructions and error messages clear and specific? (“Enter a date in MM/DD/YYYY format” vs. “Invalid input.”)
• Is plain language used? A permit page written at a graduate reading level is a barrier even if it passes every code check.
• Do important relationships rely only on color or position to be understood?

Context-dependent correctness

Many criteria are conditionally machine-detectable but require a human to resolve:

• Is a <table> an actual data table needing headers, or a layout table?
• Is an image genuinely decorative (correct to mark alt="") or meaningful?
• Is that big bold line of text a real heading or just styled text?
• Does a video need captions, audio description, or both — and are the captions accurate?

A scanner can flag a video with no caption track. It cannot tell you whether the auto-generated captions are correct, or whether the visual-only content needs audio description. Our guidance on video captions covers the human side.

A Side-by-Side Summary

The pattern: automated tools own the mechanical, high-volume checks; humans own meaning, experience, and context.

How to Combine Them: A Practical Program

The goal isn’t to pick one. It’s to use each where it’s strongest. Here’s a workable model for a government web team.

1. Run automated scanning continuously, across the whole site

Government sites change daily. A point-in-time audit goes stale the moment a new agenda PDF or department photo goes up. Continuous automated monitoring is the only way to catch regressions — the missing alt text that gets published Tuesday — before they accumulate into a complaint. This is the high-frequency, broad-coverage layer.

2. Do periodic deep manual testing on representative pages

You can’t hand-test 40,000 pages. You can hand-test a representative sample — homepage, a key form, a data table, a video page, the search results, the login flow, a typical content page — that exercises every template and interaction pattern. Manual findings on a template generalize to every page built from it. Schedule this on a cadence (quarterly is common) and after major releases.

3. Test with real assistive technology

At minimum, your team should keyboard-test and run a screen reader (NVDA is free on Windows; VoiceOver ships with macOS and iOS). Where stakes are high — a new site, a complaint response — bring in third-party testers, ideally including people who use assistive technology daily.

4. Map everything to success criteria and severity

Whether a finding comes from a scanner or a human, record the specific WCAG success criterion (e.g., SC 2.1.1) and a severity. This keeps automated and manual results in one coherent picture and feeds directly into an accessibility remediation plan. For the full process of pulling this together into a baseline, see our accessibility audit how-to.

5. Close the loop with monitoring

After you remediate, automated monitoring confirms fixes hold and catches new regressions. This is what turns a one-time clean audit into durable compliance instead of a snapshot that expires.

Where Continuous Automated Monitoring Fits

It’s worth being precise about the role of continuous monitoring, because it’s easy to overclaim. Monitoring does not replace manual testing — it can’t judge whether your alt text is meaningful or whether your modal traps focus. What it does is make the automated layer continuous instead of periodic, so that the 30 to 40 percent of issues software can catch are caught the day they appear, on every page, not six months later in the next audit.

That’s a big deal for government sites specifically, because the volume and decentralization of publishing is the core problem. Dozens of staff across departments publish content; most have never heard of WCAG. Continuous monitoring is the safety net under that reality. Pair it with continuous compliance monitoring practices and periodic manual validation, and you have a program that’s both thorough and sustainable.

Automated and manual accessibility testing aren’t competing approaches — they’re two halves of the same job. Scanners give you breadth, speed, and consistency across every page; human testing gives you the depth and judgment to evaluate the experience software can’t see. Lean on either one alone and you’ll miss most of what matters. Govzu handles the continuous automated layer for government websites — scanning every page against WCAG 2.2 AA, mapping findings to success criteria and severity, and alerting your team the moment a regression appears — so your staff can focus their manual testing time where human judgment is the only thing that works.