Shopify builds ShopGym to create repeatable sandboxes for shopping agents
Shopify built ShopGym, a platform that converts live storefronts into resettable sandbox shops and validates it with 224 tasks across six sandbox environments The system’s ShopArena component creates anonymized shop specifications from a store’s homepage, sitemap, search and cart endpoints, public catalog, and policy pages
For each exploration task, ShopArena launches a fresh specification agent equipped with browser tools to capture evidence and write an anonymized fragment of the shop. These fragments together outline the shop’s design spec, the structured attributes for features like filters and navigation, and the catalog statistics that show product counts and price distributions. While writing fragments, the agents omit brand names, product names, people’s names, and absolute URLs, ensuring the resulting sandbox stays anonymous without a later redaction step. A consolidation step merges the fragments into a single specification, and ShopArena can combine specs from multiple seed shops to increase structural and behavioral variety. Because the specification can be read, benchmark creators are able to inspect or modify the environment without going back to the original storefront. ShopGuru creates benchmark tasks across seven skill categories, each rooted in the shop’s catalog, navigation, policies, and interaction options. Validation involved graph-based structural analysis and agent-based behavioral evaluation across 224 tasks in six sandbox shops, three synthetic and three real. Synthetic stores retain the key structural traits of live storefronts, and agent performance on them predicts performance on the corresponding live shops.
A shopping agent might help you find a jacket within your budget, choose the right size, check the return policy, and add it to your cart. That part feels easy, but training and evaluating these agents is challenging because products, prices, and store layouts keep changing.
What we know
We validated the framework with graph-based structural analysis and agent-based behavioral evaluation over 224 tasks across six sandbox shops, three built from synthetic data and three from real data.
On top of these storefronts, ShopGuru synthesizes benchmark tasks across seven skill categories, grounding each task in the shop's catalog, navigation structure, policies, and interaction affordances.
ShopArena builds the environment, and ShopGuru builds the benchmark.
Exploration begins with the storefront's key surfaces: its homepage, sitemap, search and cart endpoints, public catalog, and policy pages.