Clarity Adds Detail to AI Shopping Tests: What to Check

Primary topic Mobile UX

On November 4, 2026, Microsoft Clarity explained how its AI-agent simulations assess shopping and travel tasks: completion, time taken, interaction count, and opportunities to improve the journey. For a store owner, that is a useful addition to checkout testing. Start with a purchase that already matters to your business, then check whether the resulting order matches the request.

If your Clarity project offers simulations, add one focused agent test alongside your mobile release checks. A shopper’s agent and a person with a phone can reach different failures. For a full purchase test, a pass should mean one correct order and the expected payment state.

Which Clarity feature should you use?

Use Agent Activity to investigate agents visiting your website; use a simulation assessment, where available, to attempt a particular task. Brand Agents serves a different purpose: helping people discover products and providing an assistant on a business’s site. Installing that assistant is a separate decision from investigating checkout failures.

The dashboard was already generally available for projects receiving agent traffic on October 29. The November announcement fills in the simulation outputs and presents them alongside discovery and analytics features.

Start in your existing Clarity project and look for Agent Activity. If no simulation controls are offered, ask Clarity support about access using the smiley-face feedback control at the top right. Run the acceptance case below manually while access is unresolved. Installing Brand Agents is a separate storefront change, not a documented way to unlock simulations.

Conversational Analytics, the name used in the shorter announcement, refers to the Ask Clarity experience. For checkout debugging, begin with a specific failed session. A broad question about falling sales can wait until you know which purchase route the customer used.

First find out where the purchase happens

Test the merchant website when a recommendation sends the customer there to buy. That is the route described for Brand Agents product discovery across Copilot, Bing, and MSN. A successful product recommendation leaves the store responsible for everything that happens after arrival.

Shopify also has a direct Copilot checkout without an online-store visit. Client-side Google Analytics and custom pixels do not fire there. Verify channel orders rather than expecting website sessions.

Leave a working channel enabled while diagnosing it. Direct checkout excludes subscription purchases, bundles, customizable products, B2B-only products, local delivery, and pickup. Required app fields may be absent. If an order depends on those features, prefer your website checkout. Shopify Functions and discount codes are supported; test those rules on the actual route.

To change the route in Shopify, open Sales channels > Agentic, deactivate Allow Shopify to manage for me, select Microsoft Copilot, change Direct checkout, and save. Turning it off preserves product discovery and sends customers to your store.

Direct checkout is active by default for eligible stores, displays only to US customers, and requires selling to the US. For UK and European audiences, test the website route. Shopify charges standard payment processing fees for direct checkout.

Define a successful order before running the agent

Write the expected purchase as a short test case. Include the product variant, quantity, delivery option, final total, and the point at which a person must approve payment. Keep that approval boundary explicit: an agent that correctly stops for confirmation may have done its assigned job.

Here is an illustrative sandbox basket. A navy, medium cotton shirt costs $40; the requested quantity is two. Shipping is $5 and tax is deliberately configured as $0 in this test fixture. The expected total is 2 × $40 + $5 + $0 = $85. Use these configured fixture values for the comparison.

Accept this full-purchase test only when exactly one stored order has the navy, medium variant, quantity two, the chosen delivery option, and an $85 total. Match its order ID to the test payment and confirmation. If the screen shows $85 but the order contains one shirt, fail the test and investigate the quantity update. If the task stops at an approval handoff, check the prepared basket and mark that limited task separately.

Use a sandbox that cannot trigger live fulfillment

Run an exploratory purchasing agent against a controlled test environment first. Before granting it permission to submit, verify the payment gateway is in test mode and disconnect live warehouse, shipping, and customer messaging automations. Use test customer details. If a simulation cannot run within those boundaries, stop it before payment and verify the transactional portion with your own test runner.

For WooPayments, the documented test workflow starts at Payments > Settings: enable test mode, confirm, and save. In that mode, the test card 4242 4242 4242 4242 accepts a future expiration date and a three-digit CVC. After placing the order, inspect WooCommerce > Orders and Payments > Transactions. Test orders carry a test-mode notice, and their emails have a [Test] prefix.

Use the gateway’s own test documentation if the store uses another provider. Test mode still creates records, so label each run in reporting. Confirm whether simulated visits are included in your analytics before comparing conversion rates. Keep the basket and test customer constant when comparing an agent run with a manual run; otherwise, different inventory or delivery rules can explain the difference.

Read the failure before changing the storefront

Start with the first state where the attempted purchase differs from the test case. An agent choosing the wrong size calls for a different investigation from a payment that succeeds but never creates an order. Save the attempted action and resulting page state, then reproduce that transition yourself. If the page times out after payment submission, inspect the existing order and gateway transaction before submitting again. Retry only after resolving whether the first attempt created an order.

Use the simulation’s time and interaction count to compare repeated attempts at the same task. Avoid setting a universal pass time: a purchase with an approval handoff has a different shape from an uninterrupted add-to-cart task. Completion with the right data comes first; a shorter run is useful only after that is stable.

The underlying Clarity study shows why these journeys deserve attention. It analyzed more than 5.2 million agent-classified sessions across 230,511 projects from August 13 through September 9, 2026. The published Checkout row reports 306,621 sessions and 25.21%, against 14.65% for human sessions. Those overlapping event counts do not form a shopping funnel, and an agent-classified session can include a human handoff.

The percentage base also needs care: dividing 306,621 by 25.21% implies roughly 1.22 million sessions, which is smaller than the headline cohort. Do not apply that percentage to all agent traffic or treat it as a paid-order rate. Use your own verified orders as the outcome measure.

If the failure is an address field or an error message, fix that interaction for everyone. Identify required fields, accept reasonable input formats, and validate on the server as well as in the browser. For a postal-code field, check whether the control wrongly expects numbers only. Then submit an invalid value and verify that the customer can find and correct the error without losing the basket.

Keep a separate mobile checkout check

For a store used on phones, keep a manual purchase on iPhone Safari and Android Chrome in the release checklist. Use your audience’s common devices where you have them. A narrow desktop window helps find layout problems, but it will not tell you how a phone’s keyboard covers the delivery form.

Start at a suggested width of 375 CSS pixels, then check a real phone with its keyboard open. Can you reach the final button, change a quantity, recover from an invalid address, and return from payment authentication? Repeat on a slow connection and through a declined test payment, checking that the basket survives. Try the journey with VoiceOver or TalkBack, paying particular attention to field names and error feedback. Record the browser and device you used so a later tester can repeat it.

Measure human performance separately from an agent’s task duration. The good Core Web Vitals thresholds are LCP of 2.5 seconds or less, INP of 200 milliseconds or less, and CLS of 0.1 or less, assessed at the 75th percentile with mobile and desktop separated. A fast agent run does not provide those field measurements. Check whether an added shopping widget makes the product page slower or covers a control before expanding its rollout.

For a stable cart and payment flow, keep deterministic regression tests as the repeatable baseline. An agent is useful for exploring unfamiliar paths and exposing assumptions in the interface.

Check access and data sharing before enabling more

The detected operator ecosystems are Google, OpenAI, and Anthropic. Other agents may visit your store, so use that list when deciding what a quiet dashboard tells you. Before submitting an order, confirm the simulator’s agent, run limits, price, device coverage, and analytics treatment with the controls and terms offered to your project. Those details are not established by the launch description. Treat a pass as evidence for that task and environment; keep testing the routes your customers actually use.

Brand Agents is a beta for Shopify and WooCommerce websites, with a catalog copy published to Microsoft Merchant Center when you accept its terms. Native mobile apps are outside that scope. Conversations and interaction events are shared with Microsoft, and the store must add the required privacy disclosure. Catalog discovery already appeared in documentation dated July 17, 2026. Verify the copied catalog’s price and stock before testing a recommendation; Microsoft describes synchronization as real time, but a storefront test still needs the current values.

If discovery is your goal, begin with the discovery-only option instead of adding a chat widget to checkout. Existing Brand Agents installations need no extra catalog upload under the announcement. For a new installation, confirm beta approval before proceeding with the platform’s setup: Shopify requires the Clarity script to be enabled and the agent status checked as Active; WooCommerce includes permissions, a live demo, and a traffic-percentage rollout. Preview before exposing the assistant to shoppers.

Ask Clarity remains a closed retailer beta; its planned AI Visibility and advertising inputs are future features. Clarity is advertised as free; confirm the terms for any feature you enable.

For shoppers in the EEA, UK, and Switzerland, pass their cookie-consent choice to Clarity through a supported integration. Without consent, journey continuity and recordings can be limited. Check both consent paths when diagnosing missing sessions.

Local-service bookings use a different acceptance case: confirm the selected date and party size in the connected scheduling system, using a test destination your team controls. For eligible WordPress businesses, the announced integrations include Square, Jobber, HubSpot, Calendly, Housecall Pro, and Salesforce. Confirm your connector and market access before planning a booking test.

When a test fails, rerun the purchase that failed

1. Save the expected basket, actual order state, and the first failed transition.
2. Fix the responsible form, cart, payment integration, or optional widget. If a newly enabled widget caused the regression, pause its rollout while repairing it.
3. After resolving any pending order, start a separately labeled run of the same basket and confirm one matching order and test payment.
4. Run the phone checkout again, including the error-recovery step. Release the repair when both journeys meet their acceptance conditions.