Shopify has been actively developing AI products for developers, a topic previously explored in May 2025. At the core of these efforts is Sidekick, Shopify’s AI assistant designed to combine deep commerce knowledge with advanced reasoning capabilities. Further insights into how Shopify leverages AI agents for product taxonomy at scale are available on their blog.
The Journey into Software and Shopify
Vanessa Lee, VP of Product at Shopify, began her career as a robotics engineer, specializing in mechatronics. With a passion for building, she naturally gravitated towards software within robotics teams, mastering C++ and Java. After founding a startup where she was the sole coder, Lee joined Shopify nine years ago as a Senior Product Manager on the app platform. At that time, Shopify’s platform was nascent, lacking API versioning, extensions, and functions. Over nearly a decade, the platform has seen significant growth in capabilities for developers.
Shopify’s Expanding Scope and the AI Renaissance
Shopify’s scope has evolved from simply providing online store creation to becoming a comprehensive operating system for merchant businesses, encompassing point-of-sale systems and agentic surfaces. Lee has contributed to various parts of the platform, including the online store and Liquid, and has been involved in significant updates. This expansion aligns with a broader view of AI as a technological renaissance.
Despite some developer skepticism about AI, particularly regarding its non-deterministic and ‘hallucination’ aspects, Shopify sees immense potential. The ambition for Sidekick, initially shared in 2024, envisioned an AI agent capable of assisting with platform tasks like creating products and collections, allowing users to review drafts. The past couple of years have focused on building this AI agent at scale, a challenging endeavor from scratch. A new architecture launched earlier this year significantly improved Sidekick’s success in conversations, leading to increased merchant demand. This period of foundational work and subsequent rapid feature development is what defines AI as a renaissance for the company, moving beyond mere demo features to deliver consistent value.
Understanding Sidekick: An AI Assistant for Merchants
Sidekick functions as an AI assistant integrated into the Shopify UI. It answers questions, creates products and collections, and helps edit online stores, enabling users to navigate the entire platform more efficiently. Ensuring the value of every interaction was crucial during its development.
Preventing AI Hallucinations and Ensuring Reliability
A key challenge in AI development is preventing the system from generating incorrect or ‘off-the-rails’ responses. Shopify addresses this through robust evaluation sets. This involves a creative process where large language models (LLMs) are used to grade other LLMs and generate synthetic data for ground truth sets. A critical component is including negative cases in the judge’s training set to identify and correct undesirable outputs. This extensive, time-consuming work on evaluation foundations has accelerated the internal development of reliable AI features for Sidekick.
The Human Element in AI Evaluation
In AI development, evaluation sets are considered the new specification. Unlike traditional software development with rule-based systems and APIs built to explicit requirements, AI systems must handle diverse inputs. Therefore, the evaluation set embodies the desired behavior and opinions for an AI agent like Sidekick. While LLMs assist in scaling the generation of conversations, human intervention is crucial for editing and refining these interactions, ensuring alignment with human preferences. Although human review for every one of Sidekick’s millions of conversations is impractical, sampled conversations that deviate from desired outcomes are used to refine the ground truth set. This continuous human input helps raise the bar for AI performance and prevents model collapse, where LLMs recursively generate data without external guidance.
Sidekick, UI, and Custom Applications
Exploring how Sidekick can evolve beyond a text-based interface, Shopify is investigating how LLMs can transform UI interactions. The goal is to enable Sidekick, or other LLMs, to build highly personalized UIs tailored to individual merchant needs, a capability previously impossible. This vision extends to Sidekick generating custom applications for businesses. For instance, a merchant could request a merchandising application that prioritizes specific product metadata fields, creating a bespoke tool that would otherwise be costly or time-consuming to develop. This concept, akin to
