Case study
Agent Store: an e-commerce store that AI agents can shop, run and improve
I built a store designed for the next type of customer and operator: AI agents. Shopping agents buy through a discoverable JSON API, operator agents run the store via MCP and tool APIs, and every experiment is a Thompson-sampling bandit that optimises profit, with a full audit log of every action an agent takes.
At a glance
- Role
- Solo: product thesis, system design, experimentation engine, agent tooling, build
- Company
- Independent project
- Timeline
- Oct 2026
- Team
- Solo (AI-assisted coding)
- 21% → 26.8%
- simulated conversion, baseline vs. after the first concluded experiment
- +26%
- revenue per session after 5 optimizer cycles (simulated)
- 5 surfaces
- under live bandit tests: headline, CTA, sort, shipping threshold, price
- 100%
- of agent tool calls captured in an audit log
Summary
E-commerce has been built for humans clicking through pages. I built Agent Store for two shifts I think are coming: customers will send agents to shop, and small merchants will want an agent, not a team, running their growth experiments. The store exposes contracts instead of layouts, runs every experiment as a bandit that optimises profit, and logs every action any agent takes. Status: working demo with a deterministic optimizer; the Claude-planned optimizer path is built but awaits a live-key comparison. Source on GitHub.
The problem
- Customers will send agents to shop. An agent can't reliably use a store built from visual pages and pop-ups. It needs structured discovery and a purchase API.
- Small merchants can't run experiments. A/B testing needs traffic, analysts and patience. Most D2C stores leave their headline, pricing and shipping threshold unchanged for years.
Users
- Shopping agents, and the people who send them. They need discoverable, structured APIs.
- Merchants. They want growth without hiring a growth team, and they want to stay in control.
Options considered
Fixed 50/50 A/B tests
The standard approach, but a small store takes weeks to reach significance on one test, and loses half its traffic to the loser the whole time. Rejected.
Optimise conversion rate
The obvious metric, and the wrong one for price tests: optimising conversion alone teaches the store to cut prices. Rejected as the objective for pricing.
Let the LLM act directly on the database
Fastest to build, impossible to audit, and one hallucinated price change ends the merchant's trust. Rejected.
Thompson-sampling bandits on profit, behind a single audited tool registry
Chosen
Traffic shifts to winners while still learning. Price arms are weighted by unit profit. Every operator, human or agent, HTTP or MCP, goes through the same tools and the same log.
Key decisions
How I validated it without real traffic
I built a shopper simulator with hidden preferences: price elasticity by category, sensitivity to headlines and CTAs, and customer segments. The store and the optimizer never read that file, so they have to discover those preferences through experiments. If the conversion chart on the operator console goes up, the system learned something real about the simulated market.
Results
One run on a fresh database, deterministic optimizer, 400 simulated shoppers per cycle.
| Stage | Sessions | Orders | Conversion | Revenue / session |
|---|---|---|---|---|
| Baseline, no optimizer | 400 | 84 | 21.0% | $33.29 |
| Cycle 1: four experiments launched | 400 | 71 | 17.8% | $26.51 |
| Cycles 2–3: bandits collecting data | 800 | 204 | 25.5% | $38.07 |
| Cycle 4: headline winner applied, P(best) 95% | 400 | 108 | 27.0% | $42.15 |
| Cycle 5: restock, winner holding | 400 | 106 | 26.5% | $41.63 |
Conversion rate: baseline vs. the winning headline arm at conclusion
Revenue per session: baseline vs. the two cycles after the winner was applied
- The optimizer launched four experiments in cycle 1 (headline, CTA copy, free-shipping threshold, a price raise on one product), waited two cycles while the bandits collected data, then concluded the headline test: "Free shipping on orders over $75" won at a 26.8% rate with 95% probability of being best, and was applied to the store automatically.
- Cycle 1 dipped below baseline. That is the cost of exploration: new arms get traffic before they are proven. Bandits shrink that cost compared with a fixed split, but do not remove it.
- Every one of the 13 tool calls across the five cycles is in the action log with the agent's name, inputs and outcome.
Risks I designed for
- Runaway pricing. Price arms are bounded and weighted by unit profit, so the bandit cannot "win" by giving the product away.
- Agent mistakes. The audit log, an insights memory agents write to and read first, and a human-visible operator console.
- Model unavailability. The deterministic fallback runs through the same tools, so the store never depends on the model being up.
What I'd do next
Next
Live test of the Claude optimizer
Same simulator seeds, Claude-planned policy vs. the rule-based policy. That is the eval that says whether the LLM is worth paying for.
Then
Merchant approval rules
"Auto-apply copy tests, ask me before any price change." Control that matches how much trust the merchant has built.
Later
Pilot on a real Shopify store
Through an app wrapper, on one merchant's traffic, with a hard revenue guardrail.
What this shows
Product thinking for AI agents, experimentation design, choosing the right metric (profit, not clicks), and building trust and safety into the core rather than adding it later.