Primary topic Mobile UX
A static accessibility scan can tell you a lot about a webpage without ever using it. It can inspect markup, flag missing alternative text, catch certain ARIA mistakes, and identify some contrast problems. But many accessibility failures do not exist until someone actually moves through the interface: focus jumps somewhere unexpected, an error appears without a useful announcement, a menu opens and traps the keyboard, or a status update changes without being exposed to assistive technology.
A new research preprint, InterA11y: Interactive Web Accessibility Error Detection with LLM Agents, explores whether an AI-based checker can do better by interacting with webpages instead of treating them mainly as documents to inspect. The authors combine WCAG-based testing rubrics with browser tools that can render a page, move focus, use the keyboard, inspect visual states, and observe feedback generated by interaction.
The headline result is promising: on the researchers’ expert-labeled evaluation set, InterA11y achieved an average recall of 79.0% while maintaining 93.1% precision. But the more important lesson for a mobile web team is not that a new AI tool has “solved” accessibility testing. It is that accessibility evaluation increasingly has to test behavior, not just source code and a screenshot.
That distinction matters especially on mobile sites, where responsive navigation, overlays, sticky elements, validation flows, dynamically loaded content, and JavaScript-driven controls can radically change what a user experiences after the initial page loads. At the same time, this study should not be read as a mobile-specific validation, a complete WCAG audit, or evidence that an automated agent can certify a website as compliant.
The accessibility problem that appears only after you interact
Consider a form with perfectly reasonable HTML when the page first appears. A conventional scanner may recognize the input, its label, and the submit button. But the interesting questions begin after submission. If the form contains an error, does the page explain which field failed? Does focus move somewhere sensible? Is an error message exposed programmatically? Can someone navigating by keyboard get back to the problem?
Focus behavior is another example. The Web Content Accessibility Guidelines 2.2 include requirements concerning keyboard operation, visible focus, focus order, focus that is not obscured by author-created content, error identification, and status messages. Some of these properties can be inferred partly from markup. Others depend on what the interface actually does after a sequence of actions.
Even an apparently simple focus state can be contextual. A control may technically receive keyboard focus but be covered by a sticky banner. A menu may reveal additional elements that alter the focus sequence. A script can remove focus as soon as an element receives it. W3C documents this last behavior as an accessibility failure because it can make an otherwise interactive control effectively usable only with a pointing device.
Traditional automated testing has not ignored dynamic behavior entirely; specialized tools already exist for some keyboard, navigation, and responsive-layout problems. InterA11y’s contribution is different. It gives an LLM agent a set of accessibility-focused tools and detailed rubrics, then lets the agent decide which evidence it needs to gather for a particular criterion.
How InterA11y works
The system separates the problem into two parts: collecting evidence and interpreting it.
Its rubrics break accessibility requirements into smaller testable questions. Instead of asking a language model a vague question such as “Is this page accessible?”, the system guides it toward evidence relevant to a particular WCAG success criterion.
The browser tools then let the agent inspect or manipulate a rendered page. According to the paper, those tools can gather information about markup and metadata, rendered appearance, keyboard behavior, focus, error messages, and screen-reader-related states. The agent can call tools over multiple turns rather than making one judgment from a static snapshot.
This is significant because the researchers found that missing evidence was a major reason previous LLM-based checkers made mistakes. In their analysis, errors often came not from a model being incapable of reasoning about accessibility, but from asking it to make a decision without enough information or without a sufficiently precise testing rubric.
The InterA11y repository contains the datasets, saved experiment results, rubrics, tools, baseline adapters, and experiment runners used for the paper. The published implementation requires Python, Node.js, and Chrome, and the researchers tested the system with several contemporary foundation models rather than tying the method to a single LLM.
The study tested ten WCAG success criteria, not all of WCAG
This limitation is essential. InterA11y does not attempt comprehensive automated WCAG conformance testing.
The researchers concentrated on 10 success criteria that they considered particularly relevant to interactive web contexts. Examples cited in the paper include Non-text Content, Focus Order, Error Identification, and Status Messages. The set spans the WCAG principles of perceivable, operable, understandable, and robust, but it is still a subset of the standard.
Development and ablation testing used 682 curated cases. Of those, 223 came from W3C Accessibility Conformance Testing material and 459 were AI-generated cases that the authors say were independently validated. That dataset was deliberately useful for examining difficult cases and system components; the paper also acknowledges that it overrepresents edge cases relative to the distribution of problems encountered on ordinary websites.
That distinction becomes important when interpreting the large performance improvements reported in the development experiments.
With the baseline GenA11y approach, mean recall was 31.0%. Adding InterA11y’s more detailed rubrics raised recall to 64.2%. Adding interactive agent tools increased the full system’s recall to 76.4% while reducing its false discovery rate to 9.4%.
The strongest result is therefore not merely “LLMs are better than rules.” In fact, the study’s results are more nuanced. Better instructions helped substantially, and giving the system relevant evidence helped again. The combination was also less sensitive to which of the tested language models happened to be underneath it.
The real-page evaluation is more useful, but it is still controlled
The authors recognized that small curated fixtures are not the same as real websites, so they created a second dataset from 54 webpages belonging to 48 sites across 14 areas including retail, education, finance, social networking, and public-record services.
They initially sampled 300 elements from saved copies of those pages. Elements that were obviously outside the study’s scope were removed, including videos, invisible or non-focusable elements, and elements requiring dynamic server resources. After annotation and further exclusions, the final dataset contained 197 elements: 107 labeled as violations and 90 as non-violations.
Accessibility experts with more than five years of relevant experience participated in the labeling. Each element received an annotation from one expert and one author, with disagreements reviewed further.
On that dataset, InterA11y produced recall ranging from 76.6% to 84.1% across the tested models. Its false discovery rate ranged from 5.7% to 7.8%. The paper reports an average recall of 79.0% and precision of 93.1%.
For comparison, the strongest LLM-based checker evaluated by the authors reached 44.4% recall. General-purpose browser agents also performed substantially worse when used without InterA11y’s specialized accessibility tools.
For keyboard and focus errors specifically, the picture remains far from complete. InterA11y detected 52.7% of the violations in that category, with an 8.8% false discovery rate. That is an improvement over the specialized rule contributions reported in the comparison, but it still means a significant share of these errors went undetected.
That number is a useful antidote to the idea that an interactive agent automatically equals comprehensive interactive testing.
Why a mobile site makes the static-versus-interactive distinction especially important
The paper evaluates web accessibility broadly. It does not establish separate accuracy for smartphones, mobile browsers, touch interaction, iOS Safari, Android Chrome, mobile screen readers, responsive breakpoints, or mobile-specific input patterns. Those results should not be inferred from the study.
Still, the underlying testing problem is directly relevant to mobile web development.
A responsive site is often not merely a smaller rendering of the desktop page. Navigation may collapse into a menu. Filters may become drawers. Dialogs may cover almost the entire viewport. Sticky headers and purchase controls may occupy much more of the visible screen. Content may arrive after scrolling or after another action. Form layouts can change, and controls that were side by side may become a sequence.
Those transitions create states that a scan of the initial document can miss.
Take focus visibility. WCAG 2.2’s Focus Not Obscured requirement addresses cases where author-created content hides the component currently receiving keyboard focus. On a narrow viewport, an overlay, cookie notice, sticky footer, or virtual-keyboard-related layout change can consume a large fraction of the screen. Whether the focused control remains visible is a property of the rendered state, not simply the HTML element that exists in the DOM.
Or consider additional content triggered by focus. WCAG’s Content on Hover or Focus guidance requires certain additional content to be dismissible, hoverable where applicable, and persistent under specified conditions. Inspecting CSS may reveal that a tooltip exists. Testing the behavior reveals whether the user can actually keep it open or dismiss it as required.
Mobile teams should nevertheless resist translating these examples into a claim that InterA11y has been validated for touch accessibility. Keyboard and focus testing remains relevant to websites used with physical keyboards, switch devices and other input methods, but touch exploration and mobile screen-reader behavior introduce their own testing requirements. The preprint does not report a dedicated VoiceOver-on-iPhone or TalkBack-on-Android benchmark.
An AI browser agent is not automatically an accessibility expert
One of the paper’s more interesting findings is that giving an AI agent more general browser abilities did not make it a better accessibility tester.
The researchers evaluated two general-purpose web agents, Browser Use and AgentLab. Both could interact with webpages using generic browser tools. Yet without InterA11y’s specialized tools, their accessibility results were comparatively weak and their false discovery rates were high.
When the researchers added InterA11y’s tools, AgentLab’s average recall increased from 22.2% to 44.9%, while its false discovery rate fell from 22.7% to 11.7%. Browser Use improved from 36.0% to 42.3% recall, while its false discovery rate dropped from 24.4% to 16.6%.
That suggests an important design principle for automated testing systems: the ability to click, type and scroll is not enough. The agent needs tools that expose the evidence required for the judgment, and it needs a procedure that tells it what evidence matters.
This is similar to the difference between giving a human tester a browser and giving that tester a browser plus a well-designed test protocol. Interaction capability is necessary for many dynamic checks, but it is not a substitute for a testing method.
Why this is not compliance certification
Nothing in the paper establishes that running InterA11y and receiving no findings means that a website conforms to WCAG.
First, the system targets only 10 success criteria. Second, even within those categories its recall is not 100%. Third, the real-page evaluation used selected elements rather than full end-to-end audits of entire production sites. Fourth, the evaluated pages were saved offline replicas.
The authors explicitly identify that last point as a limitation. Their real webpages originated from live sites, but the evaluation used simplified offline copies. The study excluded elements that depended on dynamic server resources, and the authors say future work should test authenticated and server-side functionality.
That exclusion matters because some of the hardest mobile web experiences occur precisely inside those flows: account creation, checkout, authentication, search suggestions, address entry, one-time-password screens, account settings, and applications that update after remote data arrives.
A saved page can reproduce many important DOM, rendering, focus, and scripting behaviors, but it does not reproduce every production state a user can reach.
The study also does not replace testing with disabled users or with people experienced in particular assistive technologies. An automated system may be able to determine that a label is missing, that focus behaves incorrectly, or that a status message is not exposed as expected. It is much less suited to answering broader questions such as whether a task is understandable, whether the interaction is unnecessarily difficult, or whether an interface works well with the strategies real users adopt.
What the research changes about an accessibility testing workflow
For a site team, the practical takeaway is not to abandon existing scanners. Static and rule-based tools remain valuable precisely because they are fast, deterministic, inexpensive, and well suited to problems that can be identified reliably from code or rendered properties.
The more useful model is layered testing.
- Keep deterministic checks for deterministic problems. Missing attributes, invalid relationships, certain naming problems, and other machine-testable failures should continue to be caught as early as possible.
- Add interaction where the criterion depends on interaction. Keyboard order, focus changes, dialogs, errors, status updates, menus, and changing states need tests that actually produce those states.
- Preserve evidence. A useful automated finding should say what the system did and what it observed, rather than merely assigning a pass or fail label.
- Test real user journeys. Production flows involving authentication, server responses and asynchronous data deserve separate attention because the InterA11y evaluation does not establish coverage for them.
- Keep human evaluation in the loop. Automated tools can reduce repetitive work and identify candidates for investigation; they cannot establish the complete lived accessibility of a site.
The researchers themselves suggest that systems like InterA11y could eventually fit into continuous-integration workflows. The results make that plausible, but the current cost and speed also show why such testing may need to be selective. In their experiments, InterA11y used 31% to 81% more tokens per case than the LLM-based checkers it was compared with, although it used fewer tokens than the general-purpose browser agents. It averaged 109 seconds per case in the reported setup.
That is very different from a near-instant static linting pass. For a large product, interactive agent testing may therefore make the most sense around components and flows where dynamic behavior actually determines accessibility, rather than replacing every inexpensive automated check with an agent.
The broader lesson is about page state, not AI
It would be easy to frame InterA11y as another story about language models outperforming older software. The paper supports a narrower and more useful conclusion.
Its strongest improvements came from giving the system better rubrics and better evidence. Interactive tools were particularly helpful for keyboard and focus testing, while the rubrics improved coverage in several other categories. The architecture matters because it stops asking a model to infer a dynamic interaction from an incomplete static representation.
That principle applies even if a team never runs InterA11y itself.
If a mobile site changes after a tap, keystroke, form submission, navigation event or asynchronous update, accessibility testing should reach that state before deciding whether the experience works. A screenshot of the starting screen cannot tell you where focus goes next. Source code alone cannot show that an overlay has hidden the active control. A DOM snapshot taken before submission cannot establish whether the resulting error is communicated correctly.
InterA11y is evidence that automated testing can move further into this interactive territory. It is not evidence that the territory has been completely automated.