Case Study: Why Exploratory AI Agents Fail at UI Testing (Playwright MCP vs. AIR)

Automating a test suite using an exploratory AI web agent looks like magic in a sandboxed demo. But what happens when you throw a raw LLM at a chaotic, real-world e-commerce platform like Flipkart?

We ran a benchmark test to find out. The goal was simple: Dismiss the initial login overlay, navigate through Beauty and Skincare categories, apply a specific brand filter, and select a product.

Here is the exact code-level breakdown of why brute-force AI test generation completely collapsed on a production DOM, and how providing deterministic LLM context saved the run.

In our first run, we used the official Playwright MCP server to let an LLM agent blindly navigate, snapshot, and guess its way through Flipkart's interface.

The Technical Failure Breakdown:

• The Overlay Blindspot: Because the agent relies on raw accessibility trees and exploratory snapshots, it failed to recognize that the initial login modal was a hard physical barrier. Instead of explicitly dismissing it, the LLM hallucinated an arbitrary page.waitForTimeout(1000) sleep step.
• The Forced-Click Band-Aid: Because the modal was still blocking the DOM, the agent couldn't naturally interact with the navigation links. To bypass Playwright's visibility checks, the AI generated a sloppy forced action: await this.beautyLink.click({ force: true });.
• The CI/CD Death Spiral: Bypassing visibility checks meant Playwright fired a click coordinate at a hidden element. When the generated script was executed, it hit a hard wall on the dynamic product grid, triggering a locator.click: Timeout 5000ms exceeded error and crashing the automated test pipeline.

For the second run, we flipped the paradigm. Instead of asking the AI to guess the workflow from a text prompt, we used AIR (Automation Intent Runtime) to capture the exact human interaction intent directly from the live browser session.

Why the Deterministic Layer Succeeded:

• Explicit Overlay Clearing: AIR captured the precise event coordinate when the human clicked the "✕" button, generating an explicit, reusable selector step: page.getByRole('button', { name: '✕', exact: true }). This cleared the viewport layout before any automated navigation began.
• Zero Guesswork, High Fidelity: Rather than letting the LLM invent abstract regular expressions, AIR pulled the exact asset paths (href and title parameters) directly painted in the live DOM runtime. The generated Page Object Model accurately mapped the target elements on the first try.
• A Healthy Assertion Failure: When the AIR-generated script ran, the Playwright locators worked flawlessly. The framework handled the modal, navigated the SPA, and successfully triggered the product tab switch. The test only paused at the very end due to a strict URL validation assertion step—meaning the automation plumbing succeeded 100%, leaving only the test validation state for the AI to maintain.

This benchmark highlights a fatal flaw in current AI QA tools: LLM agents should not be used for exploratory web parsing on volatile production DOMs. It burns thousands of API tokens, causes massive latency, and generates flaky locators.

The solution is a split architecture:

1. The Human Foundation: Use a runtime interceptor like AIR to record the exact DOM structure, flow transitions, and intent from a human session.
2. The AI Maintenance: Use tools like Playwright MCP strictly to read that rock-solid context foundation and handle test self-healing or assertion updates when the UI changes.

By feeding the LLM an immutable ledger of human intent instead of forcing it to guess, you eliminate hallucinations, protect your token context window, and build automated suites that survive real-world CI/CD pipelines.