Building a Shopify Experimentation Practice

Problem

Shopify locks its checkout — the highest-stakes page in the funnel — into a sandbox where you can only touch specific widgets. As we took on more clients already on Shopify, the question wasn't whether to test but how: where you run experiments when the most important step is untouchable.

Approach

I picked up the platform fast — Liquid theme code, building apps, the checkout sandbox — and got our stack running: Convert for A/B tests, GA4 for measurement.

The checkout was the call I think about most. The instinct is to test it — it's closest to the money. But it's a flow shoppers already trust, the sandbox only allows one widget at a time, and no realistic lift justified degrading a trusted step. So I left checkout alone and concentrated testing upstream, where you can actually move the numbers.

I also wrote a script that generates a client's full set of Convert goals — minor time savings, but identical goal names across clients, so the team can read each other's results.

Results

Three to four clients run on this setup, and onboarding is consistent instead of ad-hoc. The lesson stuck: spend your testing budget where the platform lets you win, and don't burn trust chasing a rounding-error lift.